What might help is changing the container placement strategy, the closest thing to scheduling in Concourse. By default Concourse tends to follow “container gravity”: if a worker already has a volume for a given task, it will tend to get tasks that need that container. Sometimes a worker tends to become hot for this reason.
If in your tests you triggered multiple builds at the same time, this is what you might have observed: not all workers are utilized fully. This is all speculation on my part, to really know you should have monitoring of the load of the workers all displayed together so that you can validate if worker load is uneven.
If you can confirm that the worker utilization is uneven, then you can try changing scheduling to “random” via an option to
concourse web. See https://github.com/concourse/concourse/issues/1741 for details. In our tests, switching to “random” makes workers load more even and is the setting we have enabled in production. But, we monitor worker load closely
The trade-off with “random” is that, the more workers you have, the more the task will land a worker that doesn’t have the container cached. If for example you have aggressive autoscaling (workers are disposed very fast after they become idle), you might always land a worker that needs to download the task image. This might or might not be a problem, only close monitoring will tell