Conditional pipeline: run only if another pipeline is green

Hi,
I currently have no clue how to solve following problem/usecase. Do you have an idea/approach I could try out?

I am using concourse to deploy systems via ansible - so I use git and ansible as main ressources. Currently I am only in the state of testing and I want to go the step further applying jobs to production.

My current use case would be:

  • set up 2 pipelines: A) 1 for a Deployment on Test environment and B) 1 for a deployment on Production site.
  • As we currently only have a semi-automated quality gate I have 2 conditions I’d realize to allow B) to be run: xa) pipeline A with same version Tag/commit is successfully run. xb) A Jira-Ticket with a special Trigger (in that ticket an engineer documents the successful manual tests) or a manual trigger in concourse (for certain users) is pressed.

For xb) I have a clue. Ignore this.
But for xa) I currently have no idea yet. Or, I have a little but I am not satisfied and before I manipulate you with my ideas I’d like you to simply ask: How would you solve it? How would you only allow a pipeline-job to start deploying only if A with a certain commit was deployed successful?

Side-scenarios:

  • B must not be triggered by A! Because:
  • A is in continuous development. So B also must not only build on successful latest commit of infra-code A used. I need a factor (like version tag - meh now I manipulated you :P) that matches the constraint to B
  • as of the second point and condition xb) B must not deployed on every successful build of A. Most Systems need a downtime - and we have to schedule downtimes and communicate to customers.

Thank you. :slight_smile:
Links, Ideas etc. are welcome.

Hi @snoopotic, welcome to the forum!

I do a lot of work with clients who are using Concourse to deploy Cloud Foundry. In these cases we often want to have the pipeline in the test environment “trigger” the one in the next environment with the same input versions.

Generally the approach I’ve taken in the past is to have pipeline A output some kind of bill of materials file as its last job. The contents of this file varies but its usually yaml indicating the versions of all the inputs that went into A. Then B is configured to trigger on this file and use the contents to determine the versions to run with. A common way to consume versions is to use the versions file as a vars file when setting pipeline B (--load-vars-from=). Having a pipeline that sets B is a common approach as well. We made stopover to make creation of these version files easier.

If you make A put its version file in S3 (or similar) then have B trigger on it that’s the simplest way to link pipelines. But you don’t want this as you want more control over when B starts. A pattern I’ve seen before is for the generation of the versions file to also send an email to some nominated person/list saying something to the effect of “a release is ready with these versions…”. This leaves it open for a human to do something (put file in S3/trigger manually/etc) to actually trigger B.

I realise I rambled a bit. Hopefully that was useful.

It sounds like you may need to restructure your pipeline.

[Build Job(s)]  -> [Testing Job(s)]  -> [Manual Trigger] -> [Production Deploy Job(s)]

A manual trigger can be any job where all “get” steps have the option trigger: false (or omit the trigger option); see Concourse CI Docs: Steps and Concourse CI Docs: Manually triggered job example
Additionally, you can even add a job that will start on a manual trigger that will send downtime notifications followed by another trigger to actually do the production deployment.

To get your jobs to depend on eachother in order, use the passed: [earlier-job-name] settings.

In our environment, we set it up kinda like this…

[Get Git Stuff] -> [Run Tests] -> [Build Images] -> [Deploy to DevEnv]...
-> [Quality Scan (Sonarqube)] -> [Check Major/Minor Version Num] ...
-> [Deploy Staging] -> [Manual Gate] -> [Send PagerDuty Notifs]...
-> [Deploy to Production (Rolling)]
       :On Failure -> [Rollback Production]

It’s all one massive pipeline with many more steps but this is the gist of it. Designing it this way allows you to more easily get a picture of what’s really going on and what all your teams are doing.

Almost forgot! You can use your git source multiple times for multiple branches; this allows you to use a push to your master branch as a trigger for the production builds.