Merge request workflow and "passed" constraint


I’m fairly new to Concourse and trying to wrap my head around the ways to use it with Merge Requests/Pull Requests. Our team is using GitLab to host a mono-repository of several micro-services and we currently use Jenkins to trigger our whole pipeline (linting & unittests per micro-service, e2e tests, etc.) on every MR. We want to migrate to Concourse.
Now, I see that there will be some improvements coming soon with v10 of the roadmap on the “handling of MRs” front, but until then I want to ask for a possible workflow with running the whole pipeline on every MR and to not get everything mixed up.
As our project is meant to be run/deployed in docker containers I see the rough outline of our pipeline like this:

MR_Resource |--triggers--> Create_Images_Job |-- --> UT_Service1_Job --| --> E2E_Tests_Job
                                             |-- --> UT_Service2_Job  /

After the E2E_Tests_Job I have to post a message in GitLab’s MR that CI has passed or not.

Let’s suppose that there are 2 MRs issued almost at the same time.

My questions are:

  1. Since for passing inputs/outputs between jobs one needs to put them in an external resource and later get them in another what would happen if: the first MR triggers Create_Images_Job, but shortly after that the second MR comes in and triggers another build of images. If second MR removed some code its Create_Images_Job job will finish first so the dependent jobs (the unittests jobs) will be executed 2 times with the images corresponding to the second MR (and respectively, not executed with images corresponding to MR 1), is this correct? If it is, how can I fight against it?
    My understanding is that even if the unittests (and subsequent jobs) have “passed” constraint on the Images_Resource (what is put by Create_Images_Job), they again will be run with the last version of the resource that also has successful Create_Images_Job build. In other words, Resources with “passed” constraint don’t run with the Resource version produced by the previous job, but the last version of that resource that happened to be at the time the dependent job starts that also has any successful build in the previous job?

  2. If the above is correct, it means that there is no reliable way to update the GitLab MR with the status of the CI? Will I be able to access the version of the MR1 at the end of the pipeline if MR2 triggered another round through the pipeline before MR1 finished?

We use Concourse version 5.8.

I’m not sure I’ve fully groked what you’re trying to do but I’ll take a stab at it.

First off, I assume in your example pipeline structure that the output of the Create_Images_Job is some kind of image that is then put to some external registry via a resource put then is pulled into each of the service jobs via a get. All Concourse resources must be versioned and each version must be immutable. So every time the create images job succeeds it must create a new version of the image that is differently versioned (i.e. different hash) from any other image produced by the job.

In Concourse the way you get one job to follow another is by putting passed constraints on the input resources specifying jobs that must have passed using a particular resource version before this job can start. So each time the create image job succeeds, it will produce a new version of the image that will then trigger the two service jobs. In your example of two MRs triggering two different builds of the create images job at roughly the same time you are correct that which ever run of the job finishes first will create the image that will trigger the two service jobs first. The easy way around this is to put serial: true on the create images job which prevents it from running builds in parallel. This guarantees that the MR that triggers the job first will produce an image before any subsequent MR.

Something else to keep in mind is that by default Concourse only checks for new versions of resources once per minute so even if two MRs are made close together in Gitlab that doesn’t necessarily mean they will both enter the pipeline close together. If one catches a check cycle and the other just misses it the second MR will trigger the first job about a minute after the first MR. On the other hand if both MRs happen somewhere in between checks then by default only the most recent one will trigger the pipeline. This is because the version param on get defaults to latest. If you want every MR to trigger a run of the pipeline you will need to set version: every on the get.

Other useful features for locking down the execution order of jobs are:

  • serial_groups which allow you to apply arbitrary labels to multiple jobs thereby preventing jobs with common groups from running in parallel.
  • the pool resource which allows you to define a pool of locks then use this to force multiple consecutive jobs to run in sequence before the next input can run through those jobs.

Does this help answer your questions?

Hey crsimmons,

Thank you for your explanation! You’ve groked everything right.
Sorry for my late response, I was 2 weeks off.

I decided to go with single job for all the building and testing in our pipeline. Both serial_groups and pool resource seem to hurdle the parallelism of the pipeline. In fact, I can’t wrap my head around how the pool resource can help with executing several consecutive jobs before next input is read. Is there an example somewhere?

Anyway, I have one more question now as I’m using parallel tasks in my single job. Is it possible, in Concourse, to pass single directory among tasks and every task to add to this directory? For example, the linting task will place it’s report there, then (or in parallel) the test coverage will be added there, and so on. Ultimately, at the end of the job this result will be put in some resource.

  1. I suspect that specifying same output in parallel tasks will result in content of the first finished task to be overwritten by the second task. Is this true?
  2. If the tasks are serial – I guess I can pass the output of the previous task to the input of next task, then copy contents of the input to the output directory and add the new content. With the help of output_mapping I can specify the same name for the output in the job test plan. But, is this efficient? I guess inputs and outputs are volumes under the hood in Concourse. So, after every task the output volume will be recreated. I think I saw somewhere that one can specify the same directory as input and output. Is this possible? I think one can’t mount two volumes on the same mountpoint in Docker, for example.
  3. Third variant will be to use input volume for passing across several tasks. Of course, this would work only if input volumes are mutable. However, If one can mount input and output on the same directory, something tells me that the input volume will be read-only. Is this true?


With the pool resource you can require the first job in a pipeline to request a lock before starting then have the last job relinquish it. This would prevent the first job from triggering off of other resources until the lock is returned. Unfortunately all the places I’ve seen this implemented recently are closed source.

For the other questions, originally Concourse required inputs and outputs to have different names. Maybe a year ago (I can’t remember specifically) something was changed that allows you to specify the same name for an input and an output in a task. This effectively makes it RW. If you opt for a different output name then Concourse will create the directory for you when the task container starts and it will be your responsibility to make sure relevant files are copied from the input. Either way the only values that will be available to subsequent job steps are ones written to the task outputs.

I have no idea how Concourse would handle parallel steps with the same inputs and outputs. Especially if they’re named the same. Maybe @vito or someone else from the Concourse team might have more insight into that.