How to pull docker image on a dynamic specific tag

Hi,

I’ll try to explain what I’m trying to do and maybe someone has an idea to make it work…

I have a really simple pipeline where in the 1st steps of my build job I build an image that I push into an Artifactory with a temporary tag and then, after all the tests are passed, in the last step I push it again with a final version tag.

Here is what it looks like (I’ve tried to make it as short as possible :grimacing:)

registries: &registries
  insecure_registries:
    - ((docker.registry))
  username: ((docker.utilisateur))
  password: ((docker.mot_de_passe))
  registry_mirror: http://((docker.registry))


resources:
- name: git-ci-pipeline
  type: git
  check_every: 24h
  webhook_token: webhook
  source:
    uri: ssh://git@git-scm.intra:2222/my-repo/separated-ti.git
    branch: solution-artifactory
    private_key: ((gitlab.cle_privee))
    paths:
    - pipeline.yaml

- name: git-ci
  type: git
  check_every: 24h
  webhook_token: webhook
  source:
    uri: ssh://git@git-scm.intra:2222/my-repo/separated-ti.git
    branch: solution-artifactory
    private_key: ((gitlab.cle_privee))


- name: git-myProject
  type: git
  check_every: 24h
  webhook_token: webhook
  source:
    uri: ssh://git@git-scm.intra:2222/my-project.git
    branch: master
    private_key: ((gitlab.cle_privee))

- name: version-myProject
  type: semver
  source:
    driver: git
    uri: ssh://git@git-scm.intra:2222/my-project.git
    file: version.txt
    branch: master
    private_key: ((gitlab.cle_privee))
  webhook_token: webhook

- name: image-myProject
  type: docker-image
  source:
    repository: ((docker.registry))/my-project
    <<: *registries

- name: git-tests
  type: git
  source:
    uri: ssh://git@git-scm.intra:2222/myTestProject.git
    branch: master
    private_key: ((gitlab.cle_privee))
  check_every: 24h
  webhook_token: webhook


jobs:
- name: Pipeline installation
  plan:
  - get: git-ci-pipeline
    trigger: true
  - set_pipeline: separated-ti-arti
    file: git-ci-pipeline/pipeline.yaml

###################
### IMAGE BUILD ###
###################
- name: Build & unit tests
  serial: true
  plan:
  - in_parallel:
    - get: git-ci
    - get: version-myProject
    - get: git-myProject
    - get: git-tests

  - task: Build myProject image
    privileged: true
    file: git-ci/tasks/build-image/task.yaml
    params:
      CONTEXT: source
      TARGET: image/myProject-image.tar
      IMAGE: my-project
      BUILD_ARG_VERSION: version-produit/version
    input_mapping:
      source: git-myProject
      version-produit: version-myProject

  - put: Publish myProject image with temporary tag
    resource: image-myProject
    params:
      load_file: image/myProject-image.tar
      load_repository: my-project
      tag_file: git-myProject/.git/short_ref
    get_params:
      skip_download: true

  - task: Deploy & test
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: ((docker.registry))/alpine
          insecure_registries:
            - ((docker.registry))
      inputs:
      - name: git-myProject
      - name: version-myProject
      - name: git-tests
      outputs:
      - name: final_image
      run:
        path: /bin/sh
        args:
        - -ec
        - |
          version=$(cat version-myProject/version)
          version_temporaire=$(cat git-myProject/.git/short_ref)
          echo "Prepare version $version ..."
          echo "Deploy my-project/$version_temporaire on test environment ..."
          echo "Execute tests ..."
          echo "$version" > final_image/tag
        
  - put: Publish myProject image with final tag
    resource: image-myProject
    params:
      load_file: image/myProject-image.tar
      load_repository: my-project
      tag_file: final_image/tag
      tag_as_latest: true
    get_params:
      skip_download: true

  - task: Delete temporary tag
    ...

This works perfectly well.

But the build is really long and sometimes, developers would like to only re-run the tests but they have to re-build the image too! Which is a useless waste a time.
So I try to separate the build from the tests and this is where I need advise.

To do that, I wanted to do the same for the build part (up to the temporary tag publish) and then, in a 2nd job, get the image on this published tag and re-publish it with the final tag if the tests are ok.

But the problem is that I cannot specify the tag I want to get in the docker image resource params! It’s only specified in the resource source configuration!

So here is what I tried (resources definitions are the same) :

jobs:
###################
### IMAGE BUILD ###
###################
- name: Build & unit tests
  serial: true
  plan:
  - in_parallel:
    - get: git-ci
    - get: version-myProject
    - get: git-myProject

  - in_parallel:
    - task: Build myProject image
      privileged: true
      file: git-ci/tasks/build-image/task.yaml
      params:
        CONTEXT: source
        TARGET: image/myProject-image.tar
        IMAGE: my-project
        BUILD_ARG_VERSION: version-produit/version
      input_mapping:
        source: git-myProject
        version-produit: version-myProject

    - task: Hot modify pipeline
      config:
        platform: linux
        image_resource:
          type: docker-image
          source:
            repository: ((docker.registry))/alpine
            insecure_registries:
              - ((docker.registry))
        inputs:
        - name: git-ci-pipeline
        - name: git-myProject
        outputs:
        - name: git-ci-pipeline
        run:
          path: /bin/sh
          args:
          - -ec
          - |
            temporary_image_tag=$(cat git-myProject/.git/short_ref)
            # Replace 1st occurence only
            sed -i '0,/HOT_DEFINITION/s//'${temporary_image_tag}'/g' git-ci-pipeline/pipeline.yaml

  - in_parallel:
    - put: Publish myProject image with temporary tag
      resource: image-myProject
      params:
        load_file: image/myProject-image.tar
        load_repository: my-project
        tag_file: git-myProject/.git/short_ref
      get_params:
        skip_download: true

    - set_pipeline: separated-ti-arti
      file: git-ci-pipeline/pipeline.yaml

###############
#### TESTS ####
###############
- name: Integration tests
  serial: true
  plan:
  - in_parallel:
    - get: git-ci
    - get: git-myProject
    - get: git-tests
    - get: version-myProject
      trigger: true
      passed: 
      - Build & unit tests

  - task: Deploy & test
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: ((docker.registry))/alpine
          insecure_registries:
            - ((docker.registry))
      inputs:
      - name: git-myProject
      - name: version-myProject
      - name: git-tests
      outputs:
      - name: final_image
      run:
        path: /bin/sh
        args:
        - -ec
        - |
          version=$(cat version-myProject/version)
          version_temporaire=$(cat git-myProject/.git/short_ref)
          echo "Prepare version $version ..."
          echo "Deploy my-project/$version_temporaire on test environment ..."
          echo "Execute tests ..."
          echo "$version" > final_image/tag

  - in_parallel:        
    - get: image-myProject
      params:
        save: true
        
    - put: Publish myProject image with final tag
      resource: image-myProject
      params:
        load: image-myProject/image
        load_repository: my-project
        tag_file: final_image/tag
        tag_as_latest: true
      get_params:
        skip_download: true

  - task: Delete temporary tag
    ...

So the only method I found is to change the pipeline after publishing the temporary tag in order to change the tag the docker-image-resource is targeting.

This seems to work but the problem is that I have several similar jobs with different projects that I need to have in the same pipeline and this can cause conflicts.

So is there another way to do this?

You may be able to accomplish what you want with: https://github.com/digitalocean/artifactory-docker-resource

Did you try adding passed to get: image-myProject in the Integration tests job?

    - get: image-myProject
      params:
        save: true
      passed: 
      - Build & unit tests

For what I saw, the docker image resource only pull the specified tag in the source config or latest by default.
cf. https://github.com/concourse/docker-image-resource

Could you help me with the artifactory-docker-resource @christophermancini?

I don’t see how to configure the get step in order to specify my custom tag…

Hi @OphyTe,

You can use asterisk for essentially glob wildcard searching of tags:

resources:
- name: myapplication
  type: artifactory
  icon: application-export
  source:
    endpoint: https://example.com/artifactory/
    user: ci
    password: ((artifactory.password))
    aql:
      repo: docker-local
      image: myapp/myimage
      tag: 'dev-*'

This tells Concourse to watch for new versions for any tag that starts with dev-.

Regardless of what tag you are tracking, the version that comes out of the docker image resource is the sha256 hash. If you are tracking e.g. latest, there will still be a unique version produced each time latest is built and tagged. If you use passed, you will only get those versions that successfully passed the job specified, not necessarily the most recent.

If I understand well (and that’s what I ended up doing), I don’t have to push my temporary image with a dynamic tag (.git/short_ref in my exemple) but prefer to push it with a fixed tag (e.g. latest or temporary-tag) that I will overwrite each time I build my project.

This could be something like that

resources:
- name: image-myProject
  type: docker-image
  source:
    repository: ((docker.registry))/my-project
    tag: temporary-tag
    <<: *registries

jobs:
###################
### IMAGE BUILD ###
###################
- name: Build & unit tests
  serial: true
  plan:
  - in_parallel:
    - get: git-ci
    - get: version-myProject
    - get: git-myProject

  - task: Build myProject image
    privileged: true
    file: git-ci/tasks/build-image/task.yaml
    params:
      CONTEXT: source
      TARGET: image/myProject-image.tar
      IMAGE: my-project
      BUILD_ARG_VERSION: version-produit/version
    input_mapping:
      source: git-myProject
      version-produit: version-myProject

  - put: Publish myProject image with temporary tag
    resource: image-myProject
    params:
      load_file: image/myProject-image.tar
      load_repository: my-project
    get_params:
      skip_download: true

###############
#### TESTS ####
###############
- name: Integration tests
  serial: true
  plan:
  - in_parallel:
    - get: git-ci
    - get: git-myProject
    - get: git-tests
    - get: version-myProject
      trigger: true
      passed: 
      - Build & unit tests
    - get: image-myProject
      passed: 
      - Build & unit tests

  - task: Deploy & test
      config:
      platform: linux
      image_resource:
        type: docker-image
        source:
        repository: ((docker.registry))/alpine
        insecure_registries:
            - ((docker.registry))
      inputs:
      - name: git-myProject
      - name: version-myProject
      - name: git-tests
      outputs:
      - name: final_image
      run:
        path: /bin/sh
        args:
        - -ec
        - |
        version=$(cat version-myProject/version)
        echo "Prepare version $version ..."
        echo "Deploy my-project/temporary-tag on test environment ..."
        echo "Execute tests ..."
        echo "$version" > final_image/tag
        
  - put: Publish myProject image with final tag
    resource: image-myProject
    params:
      load: image-myProject/image
      load_repository: my-project
      tag_file: final_image/tag
      tag_as_latest: true
    get_params:
      skip_download: true

  - task: Delete temporary tag
    ...

I may still try @christophermancini solution.

Thanks to both of you for your help.

You are correct, that will work, however, the downside of this approach is that you won’t have artifact history allowing you to quickly roll back should you need to. With the Artifactory approach, you have that flexibility. Also, static tags are easy to mixup / accidentally overwrite via copy paste errors, which could result in the wrong artifact getting deployed to production, I have seen it many times.

I still have final tags history with this solution.

And for the static tags mixup, I don’t see the case where it could happen because I don’t manipulate them manually. When the release is ok, the final tag is set and the temporary one is deleted …

This is only true if you never change your resource definition for the pipeline. If you change the pipeline or lose your Concourse database, that version history is lost.

For tag mixups, my teams don’t manipulate them manually either, however, one team still managed to deploy an image to prod that was meant for our staging environment leading to an incident because of a pipeline misconfiguration.