I do create pipelines programatically often and destroy as soon as execution is done. In that case, would like to know if workers clean up all the associated containers, volumes, logs and anything else immediately. We do face “no space left” issue at times. Need to know if that’s because of worker clean up delay and if disk needs to be increased to provide enough room until GC kicks in and cleans up. Or do I need to do anything in my code in addition to destroy-pipeline every time?
You cannot do anything more than calling “destroy-pipeline”, so I suggest to increase the worker disk size (and also to monitor it, so that you get an alarm before it is too late).
I forgot to mention: once you correctly size the disk and observe, you might sporadically notice the classic sawtooth shape. If the sawtooth keeps increasing (like a leak), one way for a temporary fix is to retire the worker and cleanup the disk.
For example, we have the workers in an autoscaling group and the quickest is to terminate the worker (we hook the termination signal to a clean concourse worker retire, so that no jobs will be aborted). The autoscaling will kick in and spawn a brand new worker.