Unit test pipelines code


#1

Is it possible to test Concourse pipelines with unit tests?

Jenkins has something (see below) and I was wondering if something similar exists for Concourse.


#2

I don’t think so. On the other hand, I never felt the need. The way we develop / modify Concourse pipelines is as follows:

Given a project / repository foo, we have two types of pipelines:

  • foo-master, the pipeline that observes the master branch. It is the “real” one, the one that delivers/deploys.
  • One pipeline per feature branch, for example foo-branchX, foo-branchY, and so on.

We use only one pipeline configuration file and we use ((params)) to differentiate between the master and the feature branch.

We use the same S3 bucket, and again ((params)) to differentiate a subdirectory per branch.

All this together means that if I want to change the pipeline, I do all my experiments in foo-branchZ and once happy I merge to master. That’s it!

We have an helper script that invokes fly set-pipeline for us, giving us the correct ((params)) depending on the branch.


#3

Us pivots are a bit passionate about TDD and so its been a topic of discussion for us internally for a while. Specifically, @jchesterpivotal has a lot to say about this. I’ve gone ahead and reproduced some of the salient / scrubbed conversation points from our own internal threads:

We can easily TDD at the task level. Our current pattern is a driver script that creates a relevant object from a library. We can test that object in isolation.

But we haven’t worked out how to test from the outside . How to test-drive a pipeline, then use that to test-drive a job, and only then test-driving tasks.

We’re interested in seeing if anyone else has done work in this direction, so we can steal it and take credit.

to which someone replied with ideas around doing it:

  • general yaml linting, perhaps with pre-checkin hooks?
  • Concourse-Specific yaml linting
    • Fly + Concourse do some validation at fly set-pipeline time
    • Maybe you have team-specific style guidelines? e.g. Assert that every pipeline job specifies an image.
  • If your build scripts have modular components that are re-used, then test those so that you have protection if one sub-set of the project wants to change the shared componentry in a breaking way
  • We write specs for the underlying Docker Images. We assert that the gpdb-dev-image is able to do the gymnastics required to run a cluster. For example, groupadd and usermod have special behavior in CentOS 5, which we need to have.
  • “nanny” specs that check: “Does my pipeline match what’s in source control?” Alternative: Never manually set-pipeline – use another pipeline to update it (although there are bootstrapping problems & race conditions)
  • Test-drive the access control to multi-team Concourse set-ups? For complicated multi tenancy situations, maybe that’s a thing – for indeed, right now, it’s all just hand-crafted artisinal Team organization through the CLI…

BUT, to @marco-m point, there was also some controversy around whether or not you even NEEDED to do it.

Anyways, that thread is over a year old now and i’ll let @jchesterpivotal chime in with any insights he may have learned since then


#4

Do you guys think that a project like Spring Cloud Pipelines could help with testing pipelines code?


#5

hi secoviana,

i think you brought up an important point.

We write all of our task scripts in Ruby and have Unit tests in RSpec for them. But we still haven’t figured out how to do e2e with Concourse. Sometimes one of us messes up something in the pipeline YAML file which already caused us some problems (e.g. not performing a stemcell update on one of our BOSH envs).

I just took a quick look into Cloud Pipelines and it makes a heavy weight impression on me. We prefer to keep the code simple so we can introduce new devops to our automation rather quick.

best,
D


#6

So, as far as I understand, it’s not easy to test Concourse pipelines because they are not code, rather configuration files written in YAML.

Did I got it right?


#7

For me, testing is all about having clean boundaries.

For any artifacts I build, I generally aim to eliminate environment binding, that means not putting in any explicit hostnames, credentials, or anything else to the artifact, instead injecting it at runtime.

Thus if I want to test any configuration, I can generally just substitute different values (point it to a test db instead of the real one, etc).

I don’t see why the same can’t apply to pipelines. You could even model a pipeline in concourse that results in a set-pipeline of that revision against the ‘master’ pipeline after running it through a test job that loads and runs jobs from the pipeline using an alternate set of parameters (the ‘test’ environment).

However, looking at that jenkins unit test library, it feels like it’s overstepping its bounds a bit. I don’t think concourse does any complex logic, and any it seems to have (e.g. inline scripts) you can either move to a separate script which you test in your TDD method of choice, or test the results of the pipeline’s output and assert on that.

Thus as others said, I don’t see the real value of unit testing concourse pipelines. IMO, all of that should be black boxed away in task scripts and resources, which themselves should have their own tests. I don’t want to re-test their contracts, that’s the whole point of black boxes. e.g. the same reason I don’t test that a directory exists after calling ‘mkdir’, I trust that if mkdir succeeds, it made the directory.


#8

Thank you eedwards-sk for the light you shed on this subject. I think you did a good job describing when it’s and when it’s not a good idea to test.


#9

Your approach might work for small pipelines but take a pipeline that updates your complete Cloud Foundry & BOSH environment. A simple substitution of different values won’t do the Job here.

We thought into a direction of mock resources and a framework that lets you run a single task (in a Container like Concourse would do) and expect certain outputs on the resource given a set of input.

best,
D


#10

A simple substitution of different values won’t do the Job here.

Can you clarify and be more specific?


#11

Hi eedwards-sk,

yes sure,

our pipelines are all about deploying our BOSH CloudFoundry environments and take, depending on which part of the update the pipeline performs, hours or even days to finish.

We would need some way to switch the bosh-deployment-resource, we use heavily in this pipelines, in a “test mode”. In test mode you should be able to tell the Concourse resources that they shall expect some params on a put step (in the case of the bosh-deployment-resource a BOSH manifest , deployment name, releases) and raise an error in the Concourse UI when this expectation is not met after the execution of the job (to which this put step belongs) finished.

This would have these advantages for us:

  • We don’t need to wait hours - days to verify that a new versions of our pipeline works
  • We would not modify our environment (we typically test our new pipeline version for every environment starting with our internal staging -> customer staging -> customer prod) in case the pipeline does not perform correctly
  • We would have explicit expect conditions on each resource and by failing the job whose put step were not meeting the expectation a precise feedback what was not working (according to the specs).

#12

Ah okay, using BOSH resources is outside the standard pattern of most pipelines, since it does a significant amount of orchestration within itself.

That’s a problem with BOSH though, not concourse.