Fan in with optional resources


#1

Hi All,
I’m looking to optimise my pipeline by avoiding unnecessary builds if the source hasn’t changed. Essentially we have a monorepo with a frontend and a backend. When either of these, or both change we want to do a single deployment to our integration env and run a bunch of tests. This is a simplified example:

fe_source -> build_fe -> docker_repo-----> do_deploy -> run_tests
be_source -> build_be -> s3 ---------------/

So, if just the frontend has changed, then I want to just wait until that resource is ready. Alternatively if both the front and backends have changed I want to wait for both.

As a version number we’re using the short_ref from git.

Cheers in advance!


#2

Could you create a mock, simplified pipeline and paste a screenshot of the Concourse pipeline rendered in the web interface ?

I don’t see where are all the resources (git, s3, …) and how they are routed.


#3

Sure (apologies my ASCII art skills failed me):

Both the fe_source and the be_source are actually the same repo, but with paths set to only look at the relevant bit.

The semantics I want are for a given git commit, get the deploy to wait for either the frontend image, or the backend image or both depending on which directory has changed.


#4

You can use https://www.concourse-ci.org/jobs.html#job-serial-groups just use 2 different serial groups (one for be, another for fe) then make the deploy job part of those two groups.

That should make sure that deploy only runs when neither of be or fe is building.

Note though that it’s not foolproof and there are some caveats like:

  • Neither be-build or fe-build will run if deploy is running
  • If either be-build or fe-build get triggered (due to the same commit) far apart in relation to each other that it allows for either job to complete before the other one starts deploy could still possibly run before the “later-to-trigger” build job starts.

Unless you have a specific reason why you wouldn’t want to just run the deploy twice or have two independent deploy jobs (one for fe and another for be) I don’t see why you’d want to have the deploy job wait for both to be done… Why do they have to be deployed together? the pipeline seems to indicate that they are separate artifacts.


#5

Thanks for the suggestion.

To answer your question - they are indeed separate artefacts. The process of deploying and running the tests is relatively time-consuming. We only have one integration environment to run our end to end tests in, so this is generally the bottleneck and we’d obviously like to minimise the dependencies on it. This is the reason that I’d rather have all the build stuff done in one go and then grab hold of the environment and do all the extra bits.