Testing concourse ci setup or pipelines


#1

Hello,

Since I have started building/using CI CD infrastructure, I eventually always bump in the same issue.
As the CI CD improves, it becomes more critical as the teams rely on it more and more.
Evolving the CI CD becomes a pain because breaking it has a huge cost.

Are there best practices to setup and test Concourse infrastructure and pipelines that enable painless (or less painful) evolution ?

I am looking forward to hearing from the Consourse community.


#2

Hello,

there is some discussion here Unit test pipelines code.

Going back to your question, I think each CI or devops team went through your pain. Here is my story:

  1. We started with mutable infrastructure with ad-hoc installs. Everybody was frightened at the idea of touching Jenkins configuration. Random build failures.
  2. Then switched to mutable infrastructure with SaltStack/Ansible provisioning. Situation got better but still a super pain to maintain. Everybody was still frightened at the idea of touching Jenkins configuration. Still random build failures. This was pre Jenkins 2.0 pipelines, so only way to configure Jenkins was with the web UI and the mouse. Horror.
  3. Then switched to virtualized immutable infrastructure. Immutable VM images with Packer, deployment with Terraform, CI/CD with Concourse. Everything immutable and configuration as code. Separate prod and staging environments. We prepare upgrades to Concourse in staging. Situation is way better, but there is no silver bullet. You need to spend time and effort in mastering the tools and the concepts, and do multiple interations.

The way we evolve pipelines, which is also explained in Unit test pipelines code, is that we have one pipeline per feature branch. Each pipeline puts/gets artifacts to a directory in a S3 bucket named after the branch, so we can iterate on the branch with full CI feedback before merging.

It also depends on the OS you have to build for. If only Linux, then Concourse works well with Docker and there is going work to make it work flawlessly with Kubernetes. If on the other hand you need also non-Linux builds, then there is no container support, this is why we use VMs.

You also have to decide how to deploy. You can use BOSH or you can roll your-own, maybe with Terraform.


#3

Thanks for a long and wise answer.


#4

Two questions

  • Do you see any value to run tests against your concourse setup to help you support Concourse upgrades ?
  • When you speak of feature branch, do you mean feature branch of the pipeline itself ?

#5

More and more teams (or CI/CD providers) are moving to a per team strategy. See Jenkins X or Drone IO for example.

With IaaC and some best practices, it starts to be less painful and you can let teams choose when they want to move (except for critical changes).


#6

I actually wrote a post recently on how we run Concourse:

Setting config parameters aside, the part which might look interesting to you is staging/prod envs for Concourse. The way we approach Concourse upgrades is that we upgrade staging first, then run some pipelines in staging and let it brew there for a week or two before upgrading Concourse in prod. This approach works pretty well for us.

What exact pipelines you should run in staging and how to ensure these pipelines won’t conflict with prod pipelines… it’s use case specific I think.


#7

We made our pipelines sufficiently parameterized so that we can test them.

For instance, the official CI is deployed from the official branch to the official CI server in the official concourse team with official credentials and produces artifacts to the official namespace in the artifact store.

However, when I’m working on a pipeline I deploy from my fork to my CI sandbox in my sandbox team and the pipeline produces artifacts to my sandbox namespace in the artifact store.

When my CI updates are ready, we can see them working in the source branch. I can make a PR, have it reviewed, and when it’s accepted my updated pipeline is automatically deployed as the official pipeline.

It’s not for free, but I certainly do not want to build a process that depends on manual, unreviewed changes to our production CI.