Concourse Workers running out of disk space

#1

Hey folks (cross-post from https://github.com/concourse/concourse/issues/3543)

We have recently had a customer approach our team with an issue regarding concourse and tasks we publish.

Due to the nature of our backing up product, we often have large GBs of files being produced by our concourse tasks. Increasing the size of the workers to accommodate large files transferring between tasks doesn’t seem to be a solution that will scale for the customer. They also seem to encountering a problem where if a task fails, the files created are not ‘garbage-collected’ for x amount of time. These leftover files contribute to the previous problem.

We believe we are using Concourse idiomatically at the moment, but we would like to help the customer with this issue. Does the concourse team have any ideas for solutions/mitigations for the customer? The solution suggested was to create one big task but this doesn’t seem like a clean solution to us.

tldr; We have tasks that produce big files -> they are transferred between tasks -> runs out of disk space on workers -> any ideas?

Thanks!
Glen

0 Likes

#2

Hey @gmrodgers. I’d first point you to two articles that we wrote up on Concourse volume caching that might give you a better understanding of what may be happening:


The followup questions I’d have is:

  • What version of Concourse are they using?
  • Do you have metrics set up (similar to what we have here https://metrics.concourse-ci.org/) so that you can monitor disk usage over time?
0 Likes