Feature: Value stream map


#1

Doing CD right means acknowledging the dependencies between your projects (libs, apps, base images, Dockerfiles, …) and lifting them — making them explicit — onto the CD tool so that it can compose them and run jobs when resources change, transitively.

This means then that a commit in e.g. a library repo can cascade down to a few deployments of different services that use it. Conversely, it also means that any given deployment has built into it specific upstream artefacts. E.g. the base docker image, the JVM, jars from libs, and your app jar.

GoCD, as much as you might hate it for other reasons, has a very nice feature: a page showing a graph of pipeline instances (a particular run) connected to other pipelines that caused it to run, transitively all the way up and down from a selected pipeline.

Because GoCD conflates pipelines and their artefacts, this may sound a bit disjointed. Let me put it in Concourse parlance. Given a build (a job run), you construct a graph collecting all the input resources and the job they’re outputs of, and for each said job their input resources, and so on transitively until you reach resources that aren’t outputs of any job. Take a second to visualise how that could look like.

It’s worth pointing out this isn’t the same as the pipeline view where you see jobs and resources tied together. In the value stream map you have the actual, particular, versioned instances of those jobs and resources.

The data for this is available, is just a matter of visualisation.

As a quick win that might give us a flavour of this feature, we could add a way to navigate from each resource line in a build (job instance) page (/teams/main/pipelines/example/jobs/deploy-stage/builds/31) to the respective resource page with that particular version expanded (/teams/main/pipelines/example/resources/artefact). That would give us a way to traverse that graph manually.

With this in place, having a way to show the differences between two artefacts in terms of their components becomes almost trivial. Diffing the two graphs gives you which upstream resources are different. You now just need a way to compare them.

We could then easily and precisely answer the question “what am I going to deploy if I click here”.


#2

I remember value stream maps from GoCD and I agree they are useful. To make the concept more evident for people not familiar with GoCD, could you please add some links / images to the GoCD doc where they show this UI feature ? As often, an image is worth 1000 words :slight_smile:


#3

I agree. I’ll put something together as soon as I have some time.

In the meanwhile, curious users can have a look at the description of the feature in GoCD.


#4

This sounds exactly like what we explored with “causality”: https://github.com/concourse/concourse/issues/1538

We’re really excited to work on it but other big things like spaces and RBAC are taking priority at the moment. We have code for it checked in already (the magical method that gets you the full sequence of jobs and resources from a given version), it’s just not worked in to the UI at all. Feel free to play with it!

The API endpoint is /api/v1/teams/<team>/pipelines/<pipeline>/resources/<resource-id>/causality. Here’s an example: http://ci.concourse-ci.org/api/v1/teams/main/pipelines/main/resources/concourse/versions/582150/causality


#5

Great! is #1538 the place to follow it?


#6

We’ll probably make a new issue for this once we start working on it again. The linked issue was just a spike, so it was just exploratory work with no user-visible features delivered, and we closed it once we felt happy with what we’d learned. The API endpoints are still there, though.

If/when we do make a new issue, I’ll post an update on the spike issue, so you can just follow that for now!