How to create multi branch pipeline

I was trying to create a multi-branch pipeline. When commit merge to feature branch only job related to feature wants to trigger. From feature, branch merge to master the only prod job have a trigger in a single pipeline.

What have you tried? Did it not work?

When doing something like this I would configure multiple git resources - one for each branch - then get/put with them to accomplish the triggering and promotion.

I have developed two pipelines for two env. I want a single pipeline for both.I was unable to post the concourse pipeline.yml file here.

Use the git resource (https://github.com/concourse/git-resource). You can specify the branch and use that as a trigger to your jobs.

One “instance” of that resource for master and another for your feature branch.

When pipeline triggers jobs both prod and dev starts. I want only dev job should run when merge happens to develop branch and I want only prod job triggers when merge happens to master branch

groups:

  • name: dev
    jobs:
    • Artifacts
    • build-nexus-dev-docker-image
  • name: prod
    jobs:
    • Artifacts-master
    • build-nexus-docker-image-master

resource_types:

  • name: pull-request
    type: docker-image
    source:
    repository: teliaoss/github-pr-resource

  • name: slack-notification
    type: docker-image
    source:
    repository: arbourd/concourse-slack-alert-resource

resources:

  • name: NeXus
    type: git
    icon: git
    webhook_token: ((github-enterprise.webhook-token))
    check_every: 1h
    source:
    branch: concourse
    uri: ((github-enterprise.NeXus-uri))
    username: ((svc_nexus.username))
    password: ((github-enterprise.access-token))

  • name: NeXus-master
    type: git
    icon: git
    webhook_token: ((github-enterprise.webhook-token))
    check_every: 1h
    source:
    branch: master
    uri: ((github-enterprise.NeXus-uri))
    username: ((svc_nexus.username))
    password: ((github-enterprise.access-token))

  • name: NeXus-pull-request
    type: pull-request
    check_every: 1h
    webhook_token: ((github-enterprise.webhook-token))
    source:
    repository: ebi-aws/nexus-devops
    access_token: ((github-enterprise.access-token))
    v3_endpoint: https://github.comcast.com/api/v3/
    v4_endpoint: https://github.comcast.com/api/graphql

  • name: ecr-docker-reg-dev
    type: docker-image
    icon: aws
    source:
    aws_access_key_id: ((aws_dev.aws_access_key_id))
    aws_secret_access_key: ((aws_dev.aws_secret_access_key))
    repository: ((aws_dev.aws-ecr-repo))

  • name: slack-alert
    type: slack-notification
    source:
    url: ((slack-webhook.nexus-concourse))

  • name: recipes
    type: git
    icon: chef-hat
    check_every: 1m
    source:
    branch: 0.0.69
    uri: https://github.comcast.com/concourse/recipes.git
    username: ((svc_nexus.username))
    password: ((github-enterprise.access-token))

  • name: ecr-docker-reg-prod
    type: docker-image
    source:
    aws_access_key_id: ((aws_prod.aws_access_key_id))
    aws_secret_access_key: ((aws_prod.aws_secret_access_key))
    repository: ((aws_prod.aws-ecr-repo))

  • name: build-artifacts
    type: s3
    source:
    versioned_file: build/build-artifacts.tgz
    bucket: com.ebi.nx.code.auto.cf.dev
    access_key_id: ((aws_dev.aws_access_key_id))
    secret_access_key: ((aws_dev.aws_secret_access_key))

jobs:

  • name: Artifacts
    serial: true
    serial_groups: [version]
    plan:

    • get: NeXus
      trigger: true
    • put: build-artifacts
      params:
      file: NeXus/*
      on_failure:
      put: slack-alert
      params:
      alert_type: failed
      channel: nexus-concourse
      icon_emoji: ‘:concourse:’
      silent: true
      text: |
      $BUILD_PIPELINE_NAME/$BUILD_JOB_NAME ($BUILD_NAME) FAILED to build image
      https://ci.comcast.net/teams/ebi-aws/pipelines/nexus-devops
  • name: build-nexus-dev-docker-image
    serial: true
    serial_groups: [version]
    plan:

    • get: NeXus
      trigger: true
      passed: [Artifacts]
    • get: recipes
    • task: get-version
      file: recipes/tasks/misc/get_npm_package_json_version.yml
      input_mapping:
      code: NeXus
    • put: ecr-docker-reg-dev
      params:
      build: NeXus
      tag_file: package-json-version/version
      tag_as_latest: true
      on_failure:
      put: slack-alert
      params:
      alert_type: failed
      channel: nexus-concourse
      icon_emoji: ‘:concourse:’
      silent: true
      text: |
      $BUILD_PIPELINE_NAME/$BUILD_JOB_NAME ($BUILD_NAME) FAILED to build image
      https://ci.comcast.net/teams/ebi-aws/pipelines/nexus-devops
      on_success:
      put: slack-alert
      params:
      alert_type: success
      channel: nexus-concourse
      icon_emoji: ‘:concourse:’
      silent: true
      text: |
      $BUILD_PIPELINE_NAME/$BUILD_JOB_NAME ($BUILD_NAME) SUCCESS - Image has been published
      https://ci.comcast.net/teams/ebi-aws/pipelines/nexus-devops
  • name: Artifacts-master
    serial: true
    serial_groups: [version]
    plan:

    • get: NeXus-master
      trigger: true
    • put: build-artifacts
      params:
      file: NeXus/*
      on_failure:
      put: slack-alert
      params:
      alert_type: failed
      channel: nexus-concourse
      icon_emoji: ‘:concourse:’
      silent: true
      text: |
      $BUILD_PIPELINE_NAME/$BUILD_JOB_NAME ($BUILD_NAME) FAILED to build image
      https://ci.comcast.net/teams/ebi-aws/pipelines/nexus-devops
  • name: build-nexus-docker-image-master
    serial: true
    serial_groups: [version]
    plan:

    • get: NeXus-master
      trigger:
      passed: [Artifacts-master]
    • get: recipes
    • task: get-version
      file: recipes/tasks/misc/get_npm_package_json_version.yml
      input_mapping:
      code: NeXus-master
    • put: ecr-docker-reg-prod
      params:
      build: NeXus-master
      tag_file: package-json-version/version
      tag_as_latest: true
      on_failure:
      put: slack-alert
      params:
      alert_type: failed
      channel: nexus-concourse
      icon_emoji: ‘:concourse:’
      silent: true
      text: |
      $BUILD_PIPELINE_NAME/$BUILD_JOB_NAME ($BUILD_NAME) FAILED to build image
      https://ci.comcast.net/teams/ebi-aws/pipelines/nexus-devops
      on_success:
      put: slack-alert
      params:
      alert_type: success
      channel: nexus-concourse
      icon_emoji: ‘:concourse:’
      silent: true
      text: |
      $BUILD_PIPELINE_NAME/$BUILD_JOB_NAME ($BUILD_NAME) SUCCESS - Image has been published
      https://ci.comcast.net/teams/ebi-aws/pipelines/nexus-devops

Which one is your develop branch? I’m going to assume it’s your NeXus resource that maps to the concourse branch.

If Artifacts is your initial develop job, the trigger should be your develop branch, which it looks like you have. Then when Artifacts passed, it triggers, build-nexus-dev-docker-image and passes NeXus develop branch to it.

- name: Artifacts
  serial: true
  serial_groups: [version]
  plan:
    - get: NeXus
      trigger: true
  ....

- name: build-nexus-dev-docker-image
  serial: true
  serial_groups: [version]
  plan:
    - get: NeXus
      trigger: true
      passed: [Artifacts]
    ....

Similar for your master branch. Your setup looks right to me.

The only thing that looks off is your resource spacing for source params. The branch, uri, etc params should be a nested under source, not at the same level.

- name: NeXus
  type: git
  icon: git
  webhook_token: ((github-enterprise.webhook-token))
  check_every: 1h
  source:
      branch: concourse
      uri: ((github-enterprise.NeXus-uri))
      username: ((svc_nexus.username))
      password: ((github-enterprise.access-token))

What is it that you end up seeing?