Little help for a new concourse user creating a pipeline


#1

I’m just getting started with Concourse as I’ve inherited it from a previous employee. I re-worked a staging pipeline that was broken. It’s fixed now and deploys a release candidate to our staging cluster. I’ve spent a few days trying to add production deployment job to the pipeline but haven’t had much luck.

Once the stage-deploy job is complete and passes, I’d like to kick off a prod-deploy job that would then deploy the passed release candidate image to the prod cluster. I’m not sure if Concourse has the ability to create a PR for a stage to prod merge and upon merge > deploy to the prod cluster.

Here is a portion from my pipeline.yml config that takes inputs and deploys to the staging cluster. Can somebody help me out here and help me create the production job that would create a PR, upon merge > deploy to production?

- name: stage-deploy
  serial: true
  serial_groups: [rc-test]
  plan:
  - aggregate:
    - get: staging-code
      trigger: true
      passed: [integration-tests]
    - get: rc-image
      trigger: true
      passed: [integration-tests]
      params:
        save: true
      passed: [integration-tests]
      params:
        save: true
    - get: service-version
      trigger: true
      passed: [integration-tests]
      params:
        bump: final
  - aggregate:
    - put: rc-image
      params:
        load: rc-image
        tag: service-version/version
        tag_as_latest: true
  - get: k8s-version
  - task: bump-service-git
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          aws_access_key_id: ((aws-access-key-id))
          aws_secret_access_key: ((aws-secret-access-key))
          repository: 123456789.dkr.ecr.us-blah-blah.amazonaws.com/yq
          tag: latest
      inputs:
      - name: service-version
      - name: k8s-version
      outputs:
      - name: bumped-k8s-final
      params:
        APP_NAME: the-staging-service
        VERSION_FILE: service-version/version
        DOCKER_REPO: 123456789.dkr.ecr.us-blah-blah.amazonaws.com/the-staging-service
        DEPLOYMENTS_YAML: the-staging-service/deployment.yml
        SOURCE_DIR: k8s-version
        DESTINATION_DIR: bumped-k8s-final
      run:
        user: root
        path: k8s-version/blah/blah/bump-deployment-git.sh
  - task: apply
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          aws_access_key_id: ((aws-access-key-id))
          aws_secret_access_key: ((aws-secret-access-key))
          repository: 123456789.dkr.ecr.us-blah-blah.amazonaws.com/kubectl-tools
          tag: '1.2.3'
      inputs:
      - name: bumped-k8s-final
      params:
        KUBECTL_CONFIG_FILE: ((kube-config))
      run:
        user: root
        path: sh
        args:
        - -ec
        - |
          set -e
          echo "$KUBECTL_CONFIG_FILE" > kubectl-config
          kubectl --kubeconfig kubectl-config apply -f bumped-k8s-final/stage/the-staging-service/deployment.yml -f bumped-k8s-final/stage/the-staging-service/service.yml
          kubectl --kubeconfig kubectl-config stage-deploy status -f bumped-k8s-final/stage/the-staging-service/deployment.yml

#2

Hi and welcome to Concourse,

For most purposes exist dedicated Concourse resources that implement the trigger behaviour you desire.

In your case i suggest the Concourse resource written by jtarchie called github-pullrequest-resource.

Instead of giving you a finished pipeline, ill provide you with the missing pieces.

To create a PR resource:

resources:
- name: staging-code
  type: pull-request
  source:
    base: master
    branch_base: develop
    private_key: ((my_gh_private_key))
    uri: git@github.com:myorg/mycode.git

Trigger job if new commit on “develop” branch appears:

(using normal Git resource)
- get: staging-code-pr
  trigger: true

Create PR from “develop” to “master” with status pending (PR cannot be merged in Github UI):

- put: staging-code-pr
  params:
     path: staging-code-pr
     context: release-cycle
     status: pending

Release candidate approved, set PR status to successful (PR can be merged in Github UI):

- put: staging-code-pr
  params:
     path: staging-code-pr
     context: release-cycle
     status: success

The resource will not merge the PR automatically in this case, but it assigns a state to the PR which will indicate in the Github UI if this PR is ready to merged.

I would suggest the following flow of your pipeline:

  • Check if new commits are merged to develop branch -> indicates new release candidate
  • Trigger release-candidate job -> Create PR with status pending
  • Migrate release-candidate to staging env and test
    -> if successful set state success in PR
    -> if failed set state failed in PR
  • Merge the PR to “master” manually in the Github UI
  • Trigger prod update Job using Git resource as soon as a new commit appears on “master”
  • Migrate prod env and run testsuite

If you have further questions take a look at the PR resource and play a bit with the options it provides. Don’t hesitate to ask, i am sure we will work this out. :slight_smile:

best, D


#3

Thank you for the information. I’ve been re-working the pipeline and I’ve gotten most of it to work. It’s the delivery to prod I’m still having trouble with. I’ve tried using the pr resource you suggested but it’s not going so well, sorry. Here’s what I have so far. Any help/suggestions are greatly appreciated, thanks.

resource_types:
- name: git-multibranch
  type: docker-image
  source:
    aws_access_key_id: ((aws-access-key-id))
    aws_secret_access_key: ((aws-secret-access-key))
    repository: 123456789.dkr.ecr.us-west-2.amazonaws.com/git-branch-heads-resource
    tag: latest
- name: pull-request
  type: docker-image
  source:
    repository: jtarchie/pr
resources:
- name: staging-code
  type: pull-request
  source:
    base: master
    branch_base: staging
    private_key: ((github-service-key))
    uri: git@github.com:myOrg/service.git
- name: service-code-staging
  type: git
  source:
    uri: 'git@github.com:myOrg/service.git'
    branch: staging
    private_key: ((github-service-key))
- name: service-code-prod
  type: git
  source:
    uri: 'git@github.com:myOrg/service.git'
    branch: master
    private_key: ((github-service-key))
- name: service-code-multibranch
  type: git-multibranch
  source:
    exclude: version
    uri: 'git@github.com:myOrg/service.git'
    private_key: ((github-service-key))
- name: k8s-rc-template
  type: git
  source:
    branch: master
    uri: 'git@github.com:MyOrg/k8s.git'
    private_key: ((github-k8s-key))
    paths:
    - 'prod/service-rc/*'
    - 'tools/concourse/*'
- name: k8s-version
  type: git
  source:
    branch: master
    uri: 'git@github.com:MyOrg/k8s.git'
    private_key: ((github-k8s-key))
- name: concourse
  type: git
  source:
    uri: 'git@github.com:myOrg/concourse.git'
    branch: master
    private_key: ((github-concourse-key))
- name: service-version
  type: semver
  source:
    driver: git
    uri: 'git@github.com:myOrg/service.git'
    branch: version
    file: service-version
    private_key: ((github-service-key))
- name: node-rc-image
  type: docker-image
  source:
    aws_access_key_id: ((aws-access-key-id))
    aws_secret_access_key: ((aws-secret-access-key))
    repository: 123456789.dkr.ecr.us-west-2.amazonaws.com/service-staging
- name: kubectl-tools-image
  type: docker-image
  source:
    aws_access_key_id: ((aws-access-key-id))
    aws_secret_access_key: ((aws-secret-access-key))
    repository: 123456789.dkr.ecr.us-west-2.amazonaws.com/kubectl-tools
    tag: '1.2.3'

groups:
- name: master
  jobs:
  - npm-unit-test-build
  - e2e-tests-deploy-rc
  - rollout
- name: commits
  jobs:
  - commit-unit-test
jobs:
- name: commit-unit-test
  plan:
  - aggregate:
    - get: service-code-multibranch
      trigger: true
    - get: node-rc-image
    - get: concourse
  - task: unit-test-run
    config:
      platform: linux
      inputs:
      - name: service-code-multibranch
      run:
        path: sh
        args:
        - -exc
        - |
          cd service-code-multibranch
          npm install dependencies
          npm run unit
    image: node-rc-image
- name: npm-unit-test-build
  serial_groups: [npm-unit-test-build]
  plan:
  - aggregate:
    - get: service-code-staging
      trigger: true
    - get: concourse
    - get: node-rc-image
    - get: service-version
      params:
        bump: patch
        pre: rc
  - task: unit-test-run
    config:
      platform: linux
      inputs:
      - name: service-code
      run:
        path: sh
        args:
        - -exc
        - |
          cd service-code
          npm install dependencies
          npm run unit
    image: node-rc-image
  - put: node-rc-image
    params:
      build: service-code
      dockerfile: service-code/Dockerfile
      tag: service-version/version
      tag_as_latest: true
      get_params:
        skip_download: true
  - put: service-version
    params:
      file: service-version/version
- name: e2e-tests-deploy-rc
  serial: true
  serial_groups: [npm-unit-test-build, e2e-rc-test]
  plan:
  - aggregate:
    - get: node-rc-image
      trigger: true
      passed: [npm-unit-test-build]
    - get: concourse
    - get: service-code
      passed: [npm-unit-test-build]
    - get: service-version
      passed: [npm-unit-test-build]
    - get: k8s-rc-template
    - get: kubectl-tools-image
  - task: parse-rc-template
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          aws_access_key_id: ((aws-access-key-id))
          aws_secret_access_key: ((aws-secret-access-key))
          repository: 123456789.dkr.ecr.us-west-2.amazonaws.com/yq
          tag: latest
      inputs:
      - name: service-version
      - name: k8s-rc-template
      outputs:
      - name: output
      params:
        APP_NAME: service
        VERSION_FILE: service-version/version
        DOCKER_REPO: 123456789.dkr.ecr.us-west-2.amazonaws.com/service-staging
        DEPLOYMENTS_YAML: k8s-rc-template/prod/service-rc/deployment.yml.template
        DEPLOYMENTS_METADATA_NAME: 'true'
        DEPLOYMENTS_METADATA_LABELS_APP: 'true'
        DEPLOYMENTS_SPEC_TEMPLATE_METADATA_LABELS_APP: 'true'
        DEPLOYMENTS_SPEC_TEMPLATE_SPEC_CONTAINERS_IMAGE: 'true'
        SERVICES_YAML: k8s-rc-template/prod/service-rc/services.yml.template
        SERVICES_METADATA_NAME: 'true'
        SERVICES_METADATA_LABELS_APP: 'true'
        SERVICES_SPEC_SELECTOR_APP: 'true'
      run:
        user: root
        path: k8s-rc-template/tools/concourse/parse-templates.sh
  - task: apply
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          aws_access_key_id: ((aws-access-key-id))
          aws_secret_access_key: ((aws-secret-access-key))
          repository: 123456789.dkr.ecr.us-west-2.amazonaws.com/kubectl-tools
          tag: '1.2.3'
      inputs:
      - name: output
      params:
        KUBECTL_CONFIG_FILE: ((kube-config))
      run:
        user: root
        path: sh
        args:
        - -ec
        - |
          set -e
          cat output/deployment.yml
          cat output/services.yml
          echo "$KUBECTL_CONFIG_FILE" > kubectl-config
          kubectl --kubeconfig kubectl-config apply -f output/deployment.yml
          kubectl --kubeconfig kubectl-config apply -f output/services.yml
          kubectl --kubeconfig kubectl-config rollout status -f output/deployment.yml
    on_failure:
      task: delete
      file: service-code/delete_deployment.yml
      image: kubectl-tools-image
      params:
        KUBECTL_CONFIG_FILE: ((kube-config))
  - task: test-health-metrics
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          aws_access_key_id: ((aws-access-key-id))
          aws_secret_access_key: ((aws-secret-access-key))
          repository: 123456789.dkr.ecr.us-west-2.amazonaws.com/kubectl-tools
          tag: '1.2.3'
      inputs:
      - name: service-version
      - name: output
      params:
        KUBECTL_CONFIG_FILE: ((kube-config))
      run:
        path: sh
        args:
        - -ec
        - |
          export NAME="service-$(cat service-version/version | sed 's/\./-/g')"
          echo "$KUBECTL_CONFIG_FILE" > kubectl-config
          echo "Checking: 'curl -sS --fail -L \"$NAME-internal.default.svc.cluster.local:5000/healthz/liveness\"'"
          kubectl --kubeconfig kubectl-config run -i --rm "$NAME-e2e-test" \
            --image=appropriate/curl:latest \
            --generator=run-pod/v1 \
            --restart=Never \
            --attach=true \
            --labels="app=$NAME,delete-me=true" \
            --command -- curl -sS --fail -L "$NAME-internal.default.svc.cluster.local:5000/healthz/liveness"
          echo "Checking: 'curl -sS --fail -L \"$NAME-internal.default.svc.cluster.local:5000/metrics\"'"
          kubectl --kubeconfig kubectl-config run -i --rm "$NAME-e2e-test" \
            --image=appropriate/curl:latest \
            --generator=run-pod/v1 \
            --restart=Never \
            --attach=true \
            --labels="app=$NAME,delete-me=true" \
            --command -- curl -sS --fail -L "$NAME-internal.default.svc.cluster.local:5000/metrics"
    on_failure:
      task: delete
      file: service-code/delete_deployment.yml
      image: kubectl-tools-image
      params:
        KUBECTL_CONFIG_FILE: ((kube-config))
  - task: test-rpcs
    config:
      platform: linux
      inputs:
      - name: service-code
      run:
        path: sh
        args:
        - -exc
        - |
          cd service-code
          ls -alh
          ls -d */
          export JWT_SECRET=((jwt-secret))
          export TEST_DB_HOST="mysql.default.svc.cluster.local"
          export TEST_DB_PORT="3306"
          export TEST_DB_DATABASE="test-service-auth"
          export TEST_DB_USERNAME="test-service-auth"
          export TEST_DB_PASSWORD="test-service-auth"
          npm install
          npm run api
    image: node-rc-image
    ensure:
      task: delete
      file: service-code/delete_deployment.yml
      image: kubectl-tools-image
      params:
        KUBECTL_CONFIG_FILE: ((kube-config))
- name: rollout
  serial: true
  serial_groups: [e2e-rc-test]
  plan:
  - aggregate:
    - get: service-code
      trigger: true
      passed: [e2e-tests-deploy-rc]
    - get: node-rc-image
      trigger: true
      passed: [e2e-tests-deploy-rc]
      params:
        save: true
      passed: [e2e-tests-deploy-rc]
      params:
        save: true
    - get: service-version
      trigger: true
      passed: [e2e-tests-deploy-rc]
      params:
        bump: final
  - aggregate:
    - put: node-rc-image
      params:
        load: node-rc-image
        tag: service-version/version
        tag_as_latest: true
  - get: k8s-version
  - task: bump-service-git
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          aws_access_key_id: ((aws-access-key-id))
          aws_secret_access_key: ((aws-secret-access-key))
          repository: 123456789.dkr.ecr.us-west-2.amazonaws.com/yq
          tag: latest
      inputs:
      - name: service-version
      - name: k8s-version
      outputs:
      - name: bumped-k8s-final
      params:
        APP_NAME: service-staging
        VERSION_FILE: service-version/version
        DOCKER_REPO: 123456789.dkr.ecr.us-west-2.amazonaws.com/service-staging
        DEPLOYMENTS_YAML: prod/service-staging/deployment.yml
        SOURCE_DIR: k8s-version
        DESTINATION_DIR: bumped-k8s-final
      run:
        user: root
        path: k8s-version/tools/concourse/bump-deployment-git.sh
  - task: apply
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          aws_access_key_id: ((aws-access-key-id))
          aws_secret_access_key: ((aws-secret-access-key))
          repository: 123456789.dkr.ecr.us-west-2.amazonaws.com/kubectl-tools
          tag: '1.2.3'
      inputs:
      - name: bumped-k8s-final
      params:
        KUBECTL_CONFIG_FILE: ((kube-config))
      run:
        user: root
        path: sh
        args:
        - -ec
        - |
          set -e
          echo "$KUBECTL_CONFIG_FILE" > kubectl-config
          kubectl --kubeconfig kubectl-config apply -f bumped-k8s-final/prod/service-staging/deployment.yml -f bumped-k8s-final/prod/service-staging/service.yml
          kubectl --kubeconfig kubectl-config rollout status -f bumped-k8s-final/prod/service-staging/deployment.yml
  - put: k8s-version
    params:
      repository: bumped-k8s-final
      rebase: true
  - put: service-version
    params:
      file: service-version/version
- name: rollout-prod
  serial: true
  serial_groups: [e2e-rc-test]
  plan:
  - aggregate:
    - get: service-code-prod
    - get: staging-code-pr
      trigger: true
  - aggregate:
    - put: node-rc-image
      params:
        load: node-rc-image
        tag: service-version/version
        tag_as_latest: true
    - put: staging-code-pr
      params:
        path: staging-code-pr
        context: release-cycle
        status: pending
  - get: k8s-version
  - task: bump-service-git
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          aws_access_key_id: ((aws-access-key-id))
          aws_secret_access_key: ((aws-secret-access-key))
          repository: 123456789.dkr.ecr.us-west-2.amazonaws.com/yq
          tag: latest
      inputs:
      - name: service-version
      - name: k8s-version
      outputs:
      - name: bumped-k8s-final
      params:
        APP_NAME: service-staging
        VERSION_FILE: service-version/version
        DOCKER_REPO: 123456789.dkr.ecr.us-west-2.amazonaws.com/service-staging
        DEPLOYMENTS_YAML: prod/service-staging/deployment.yml
        SOURCE_DIR: k8s-version
        DESTINATION_DIR: bumped-k8s-final
      run:
        user: root
        path: k8s-version/tools/concourse/bump-deployment-git.sh
  - task: apply
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          aws_access_key_id: ((aws-access-key-id))
          aws_secret_access_key: ((aws-secret-access-key))
          repository: 123456789.dkr.ecr.us-west-2.amazonaws.com/kubectl-tools
          tag: '1.2.3'
      inputs:
      - name: bumped-k8s-final
      params:
        KUBECTL_CONFIG_FILE: ((kube-config))
      run:
        user: root
        path: sh
        args:
        - -ec
        - |
          set -e
          echo "$KUBECTL_CONFIG_FILE" > kubectl-config
          kubectl --kubeconfig kubectl-config apply -f bumped-k8s-final/prod/service-staging/deployment.yml -f bumped-k8s-final/prod/service-staging/service.yml
          kubectl --kubeconfig kubectl-config rollout status -f bumped-k8s-final/prod/service-staging/deployment.yml
  - put: k8s-version
    params:
      repository: bumped-k8s-final
      rebase: true
  - put: service-version
    params:
      file: service-version/version