Builds get stuck for ~15-20 minutes waiting for a docker image push


#1

My builds get stuck for ~15-20 minutes waiting for a docker image push. Additionally, I cannot intercept into the docker push/pull containers.

> fly -t tb containers
...
54adb380-2cde-43ae-7757-4ccaf7a86ce6  concourse-worker-0  textbrawlers  build-auth-server  6        45        put    docker-image-auth-server  n/a
...

> fly -t tb intercept -j textbrawlers/build-auth-server -s docker-image-auth-server
error: websocket: bad handshake

The GUI looks like this for 15-20 minutes:

After a while though, the build will succeed and the docker image is published. The docker images are rather small (25MB - 100MB), so transferring them between build steps shouldn’t be that slow.

I’m running concourse 4.0 using a slightly modified version of the helm chart. It’s behind 2 layers of nginx’s, could that be a factor?


#2

The helm chart defaults to the ‘naive’ baggageclaim volume driver, which will be incredibly slow and inefficient. I would recommend configuring it as ‘detect’ instead.


#3

Thanks! I had already experimented with changing the volume driver, but for some reason I thought that wasn’t the cause. Changing it from overlay to btrfs did solve it.