Is it possible to trigger the same pipeline multiple times in parallel at the same time?


#1

I’m looking into using concourse as a simple task manager by triggering various resources and reading different inputs. Is it possible to launch the same pipeline 1000’s of times? I need to execute lots of long-term work across multiple resources at the same time. I cannot wait for the pipeline to complete before repeating the same job on a different target.


#2

Launching the same pipeline, no. On the other hand, in the same pipeline, you can put N jobs in fan-out, so if something is triggering, all the N jobs will run in parallel. It is difficult to be more precise since it is not clear what you want to do.


#3

Thank you for the reply. I thought about putting all of my N jobs into a single pipeline. That makes sense to me. I will investigate the fan-out function further.

Thank you.


#4

I realized that you might not be familiar with Concourse and that “fan-out” is not mentioned in the documentation, so I wrote a minimal example.

The following pipeline configuration

# We use YAML anchors to reduce duplication, with the simplifying assumption
# that all the jobs are really identical.
job-template: &job-template
  plan:
    - get: project-x.git
      trigger: true
    - task: t1
      config:
        platform: linux
        image_resource:
          type: docker-image
          source: {repository: alpine}
        run:
          path: echo
          args: ["Hello, world!"]


###############################################################################

resources:

- name: project-x.git
  type: git
  source:
    uri: https://github.com/marco-m/concourse-pipelines.git
    branch: master

###############################################################################

jobs:

- name: j1
  <<: *job-template
- name: j2
  <<: *job-template
- name: j3
  <<: *job-template
- name: j4
  <<: *job-template
- name: j5
  <<: *job-template

will generate this pipeline, where jobs j1 … j5 are in fan-out and run in parallel:

in this case, the trigger is a commit in the git repo. Any suitable Concourse resource can be used as trigger.