Executing jobs or sub-jobs sequentially

#1

I am having some problems getting my concourse jobs to execute sequentially. The reason I am trying to do this is because our current set up results in all jobs being executed immediately. This overwhelms our testing resources and causes false failures. I have tried every permutation that I can think of around serial: true and none of it has worked. Attached in a picture of the pipeline in question.

What I would like is anytime the tests are executed, either manually via the start-pipeline job or the twice per day timer, that jobs 1, 2, 3, and 4 are executed sequentially (or in pairs). Is this something that can be done easily?

image

#2

You have two options:

  1. serial_groups where you could define some number of “groups” for your jobs then only one job per group is allowed to run at any given time. For example, adding serial_groups: [jobs] to all 4 jobs would only allow one job to run at a time while the others queued. It’s worth noting that this doesn’t guarantee any kind of order so if you were to manually trigger several times in a row you wouldn’t be able to guarantee that all four jobs run with the first trigger before job_1 runs with the next trigger.

  2. passed constraints if you changed your pipeline jobs to look something like

jobs:
- name: job_1
  plan:
  - get: manual_trigger
    trigger: true
  - get: source
  - get: twice-a-day
    trigger: true
- name: job_2
  plan:
  - get: manual_trigger
    trigger: true
    passed: [job_1]
  - get: source
  - get: twice-a-day
    trigger: true
- name: job_3
  plan:
  - get: manual_trigger
    trigger: true
    passed: [job_2]
  - get: source
  - get: twice-a-day
    trigger: true
- name: job_1
  plan:
  - get: manual_trigger
    trigger: true
    passed: [job_3]
  - get: source
  - get: twice-a-day
    trigger: true

Then each job would follow the previous one which guarantees they run sequentially and in the correct order. If you manually trigger a second time before the first trigger makes it through all the jobs then there is nothing stopping job_1 from running while job_4 is still running though.

Depending on your needs it might make sense to do both approaches. The reason serial: true didn’t do what you needed is that it only impacts parallel runs of a particular job. serial: false means multiple runs of the job will execute at the same time while serial: true means multiple runs of the job will queue and execute one at a time. Personally I put serial: true on almost every job I write.

1 Like
#3

Thank you so much! Serial groups are exactly what I needed.