Managing versions of task resources


#1

I’m wondering if there are any best practices for managing task resources stored in Git.

What we’re coming across fairly quickly is that as the task implementation changes via commits to Git, existing pipelines will fail (due to changes such as requiring a different parameter).

Presumably, part of the management is to use Git tagging. However, since our GIt repository has multiple tasks, using a simple “1.0.0” version tag is insufficient.

I’m curious what strategy people are using to manage their Concourse tasks to control these types of conflicts.

Thanks!


#2

You mean that you have multiple pipelines that share the same concourse task, and that when you update that task you validate it with one (or a subset) of these pipelines, and so when you merge the change to master the other pipelines will sometimes fail ?

How many repositories are at play ?


#3

In this particular case, it’s just one repository. The first version of the pipeline (as it were) implemented the task as simply as possible. As new features get integrated, existing pipelines break due to newly required variables until they get updated.

What I’ve decided to do, at least for the short term, is to pull in the task files via git and a tagging strategy of ‘/’ and then use the git resource patterns to identify compatible versions. This will hopefully limit collisions.

As a more concrete example, we’re rolling an internal managed Cloud Foundry buildpack service. First round simply got it functional and validated that it functions. As we are a business that uses change management discipline, we also need to create tickets for our production roll of buildpacks to document the change. As that new capability gets introduced and tested, the ‘task.sh’ file that rolls the buildpacks also gets updated… but it requires new settings and breaks the production environments until the development is tested.

What’s best practice to manage change to existing pipelines? At this point, we’re using a tagging strategy against the git repository.


#4

What’s needed is redistributable tasks, which I hope will become a thing before too long.

A tactic I have explored is to build a task image and then have my task YAML point to that.

The task image can then be lockstepped, as can the task YAML. This is not a complete solution: you need to pull in the task YAML from the controlling repo. But it means you can separate the construction and testing of your tasks from their consumption.

OK, that’s all words. Here’s an example. I wrote a fairly complex resource which fetches a whole bunch of data. For convenience, I wrote tasks that can pretty-print the data that the resource fetches.

So in the repo, as well as in and check commands in cmd/, I also added show-build, show-resources etc as regular CLI commands. They get a little bit of acceptance testing but not much as they’re not very complex.

Next, I have the resource’s Dockerfile, which looks a bit like this:

FROM ubuntu:18.04

COPY binaries/check           /opt/resource/check
COPY binaries/in              /opt/resource/in

# ...

COPY binaries/show-build      /opt/tasks/show-build

That is: I ship the task binaries with the resource image. This has the nice side-effect of taking advantage of the cached resource image (since they’re used together).

Then, in the same repository, I provide a toplevel tasks/ directory. Someone who wants to access the task binaries I built can check out my repo and use those directly. Here’s what the task YAML for show-build looks like:

platform: linux

image_resource:
  type: docker-image
  source:
    repository: jchesterpivotal/concourse-build-resource
    tag: v0.10.1

inputs:
- name: build

run:
  path: /opt/tasks/show-build

I can use the same resource image, since I added my helper tasks to it as binaries.

As you can see, I also use a tag in the YAML. This is intended for cases exactly like yours. In an ideal world folks are careful to keep track of dependencies. In the actual world everyone pulls from master and only compares versions when there’s an explosion.

My pipeline for this resource is set up to keep the versioning consistent. Upon release, the task YAML is updated to the release version. The release version is tagged on the image and on the github release. Bing bang boom you can pull from master without too many surprises (though really, you ought to be using the tag on the git repo).

I think this approach ameliorates some of your pain. It’s a halfway house to “true” redistributable tasks, but it’s a fairly nice halfway house.