How to consolidate several "put" steps?

#1

I have a job that builds 20 small Docker images.

Building those images is very fast (several seconds per image), but when I run the pipeline, each step starts with “waiting for docker to come up…”, which takes about 30 seconds. Therefore the resulting pipeline takes 25 minutes.

How would you recommend making it faster?

The job looks like this:

resources:
    - name: service_1
      type: docker-image
      source: ...
    # ... more repos ...
    - name: service_20
      type: docker-image
      source: ...

jobs:
    - name: docker-images
      serial: true
      plan:
          - get: service_base_image

          - put: service_1
            params: {build: ..., dockerfile: ...}
          # ... more services ...
          - put: service_20
            params: {build: ..., dockerfile: ...}
#2

You have two paths to investigate (you could even put both together):

One is to parallelize the pipeline, creating a few jobs, each of them containing some of the puts. How much parallelism depends on how many workers / type of workers / other concurrent activity on your Concourse system, you have to try.

The other is to use a task and not the docker-image resource to build, see https://github.com/concourse/builder. You then push with https://github.com/concourse/registry-image-resource. Note that this is the new way of building and pushing docker images, but is not documented yet.