Merge request workflow and "passed" constraint

Hello,

I’m fairly new to Concourse and trying to wrap my head around the ways to use it with Merge Requests/Pull Requests. Our team is using GitLab to host a mono-repository of several micro-services and we currently use Jenkins to trigger our whole pipeline (linting & unittests per micro-service, e2e tests, etc.) on every MR. We want to migrate to Concourse.
Now, I see that there will be some improvements coming soon with v10 of the roadmap on the “handling of MRs” front, but until then I want to ask for a possible workflow with running the whole pipeline on every MR and to not get everything mixed up.
As our project is meant to be run/deployed in docker containers I see the rough outline of our pipeline like this:

MR_Resource |--triggers--> Create_Images_Job |-- --> UT_Service1_Job --| --> E2E_Tests_Job
                                             |-- --> UT_Service2_Job  /

After the E2E_Tests_Job I have to post a message in GitLab’s MR that CI has passed or not.

Let’s suppose that there are 2 MRs issued almost at the same time.

My questions are:

  1. Since for passing inputs/outputs between jobs one needs to put them in an external resource and later get them in another what would happen if: the first MR triggers Create_Images_Job, but shortly after that the second MR comes in and triggers another build of images. If second MR removed some code its Create_Images_Job job will finish first so the dependent jobs (the unittests jobs) will be executed 2 times with the images corresponding to the second MR (and respectively, not executed with images corresponding to MR 1), is this correct? If it is, how can I fight against it?
    My understanding is that even if the unittests (and subsequent jobs) have “passed” constraint on the Images_Resource (what is put by Create_Images_Job), they again will be run with the last version of the resource that also has successful Create_Images_Job build. In other words, Resources with “passed” constraint don’t run with the Resource version produced by the previous job, but the last version of that resource that happened to be at the time the dependent job starts that also has any successful build in the previous job?

  2. If the above is correct, it means that there is no reliable way to update the GitLab MR with the status of the CI? Will I be able to access the version of the MR1 at the end of the pipeline if MR2 triggered another round through the pipeline before MR1 finished?

We use Concourse version 5.8.

I’m not sure I’ve fully groked what you’re trying to do but I’ll take a stab at it.

First off, I assume in your example pipeline structure that the output of the Create_Images_Job is some kind of image that is then put to some external registry via a resource put then is pulled into each of the service jobs via a get. All Concourse resources must be versioned and each version must be immutable. So every time the create images job succeeds it must create a new version of the image that is differently versioned (i.e. different hash) from any other image produced by the job.

In Concourse the way you get one job to follow another is by putting passed constraints on the input resources specifying jobs that must have passed using a particular resource version before this job can start. So each time the create image job succeeds, it will produce a new version of the image that will then trigger the two service jobs. In your example of two MRs triggering two different builds of the create images job at roughly the same time you are correct that which ever run of the job finishes first will create the image that will trigger the two service jobs first. The easy way around this is to put serial: true on the create images job which prevents it from running builds in parallel. This guarantees that the MR that triggers the job first will produce an image before any subsequent MR.

Something else to keep in mind is that by default Concourse only checks for new versions of resources once per minute so even if two MRs are made close together in Gitlab that doesn’t necessarily mean they will both enter the pipeline close together. If one catches a check cycle and the other just misses it the second MR will trigger the first job about a minute after the first MR. On the other hand if both MRs happen somewhere in between checks then by default only the most recent one will trigger the pipeline. This is because the version param on get defaults to latest. If you want every MR to trigger a run of the pipeline you will need to set version: every on the get.

Other useful features for locking down the execution order of jobs are:

  • serial_groups which allow you to apply arbitrary labels to multiple jobs thereby preventing jobs with common groups from running in parallel.
  • the pool resource which allows you to define a pool of locks then use this to force multiple consecutive jobs to run in sequence before the next input can run through those jobs.

Does this help answer your questions?