Building several Dockerfiles that depend on each other

Hi!

I am currently in the process of building a CI/CD for several docker images that depend on each other.
Let’s say we have a Dockerfile (Level A) that gets used by another Dockerfile (Level B). This Dockerfile is used by another Dockerfile (Level C) and so on. Last layer may be Level Z.

What i want is if a Dockerfile from any Layer is changed the rebuild of that layer and all succeeding layers will be triggered.
Example:
Layer B has changed -> Layer B, C, D, E, … will be rebuilt.
Layer Z has changed -> only Layer Z will be rebuilt.

What i also want is, that all docker images are tagged with a timestamp with the format YYYYmmdd-HHMM.

I managed to build Level A with the correct Tag information by using:

jobs:
- name: build-amd64-ubuntu-bionic-base
  public: true
  serial: true
  plan:
  - get: source
    trigger: true
  - task: build-cached-image-workspace
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: ubuntu
      outputs:
      - name: workspace
      run:
        path: /bin/bash
        args:
        - -c
        - |
          output_dir=workspace
          echo $(date +"%Y%m%d%H%M") > "$output_dir/timestamp"
          echo bionic-$(cat "$output_dir/timestamp") > "$output_dir/tag"
    
  - put: amd64-ubuntu-bionic-base
    params:
      build: source/amd64/ubuntu/bionic/base/
      dockerfile: source/amd64/ubuntu/bionic/base/Dockerfile
      tag_file: workspace/tag

This creates the timestamp in “$output_dir/timestamp” that can be used by the tag_file option for the docker build.

The problems that i have are now:

  1. How do i trigger the succeeding levels? I thought of using some gating resource (meshcloud/gate-resource). Is this the way to go? I could’t test because of issue number two.
  2. How to i extract a string from "$output_dir/timestamp" that can be used in the docker build as build_args? The Dockerfiles uses a ARG to reference the version the Dockerfile uses. I didn’t find a way to use the content of the timestamp file as an argument for the docker build.

This is a snippet from the level B job. Notice the ??? where i don’t know how to handle this the right way or least any way.

 - put: amd64-ubuntu-bionic-cpp
    params:
      build: source/amd64/ubuntu/bionic/cpp/
      dockerfile: source/amd64/ubuntu/bionic/cpp/Dockerfile
      tag_file: workspace/tag
      build_args:
        DOCKER_REGISTRY: {{docker_registry}}
        ARCH: amd64
        OS: ubuntu
        OS_RELEASE: bionic
        TIMESTAMP:  ??????

This is the header of my level B Dockerfile.

ARG DOCKER_REGISTRY
ARG ARCH
ARG OS
ARG OS_RELEASE
ARG TIMESTAMP

FROM ${DOCKER_REGISTRY:-none}/{ARCH:-amd64}/{OS:-ubuntu}/${RELEASE:-bionic}:${TIMESTAMP:-latest} 

Can anybody give me a hint?

You can trigger based on a new docker image.

Let’s say that docker-image-a:latest is always the last known good image of A. When your build and smoketest of A passes, it’s published to the repository as A. We also publish to the repository as :<branch>-<sha>; the docker image resource can tag as many ways as you want.

So you might have a pipeline like this:

jobs:
- name: build-b
  plan:
  - get: source-b
    trigger: true
  - get: docker-image-a
    trigger: true
  - put: docker-image-b 

As for how to satisfy the timestamp, I’d advise against it for Docker. Instead, create a tag with the meaning you desire. If you “promote” to latest at a later step, then make a tag called candidate instead and let the default var resolve to that.