Given a binary install I’m wondering what should be the steps to essentially “reset” a worker. Currently these are the steps we’re following (running concourse as a systemd service):
- Retire the worker (and wait until it disappears from the listed workers from
fly -t <target> ws)
- Stop the systemd service
- Remove everything in the worker dir (this includes running
umounton all volumes before deleting the files)
- Remove all concourse generated network interfaces
- Remove all runc files under
- Start systemd service
This seems to work fairly well however there is one small issue. It seems the space occupied by the volumes isn’t being freed until the machine reboots.
df shows almost no difference before and after the volumes are unmounted and deleted. Running
du on the affected partition (where the worker dir is) shows correctly that the space is no longer being used). After a machine reboot
df shows the “correct” used space again (matching what
Tried with no success using
lsof to figure out what is keeping the file handles (which I assume must be the cause). Anyone have any insight into either the “reset” process or how I can find out why the space isn’t being freed until a reboot happens?
Running 3.10.0 binary install with btrfs using loopback, Debian 8.10 with