What’s needed is redistributable tasks, which I hope will become a thing before too long.
A tactic I have explored is to build a task image and then have my task YAML point to that.
The task image can then be lockstepped, as can the task YAML. This is not a complete solution: you need to pull in the task YAML from the controlling repo. But it means you can separate the construction and testing of your tasks from their consumption.
OK, that’s all words. Here’s an example. I wrote a fairly complex resource which fetches a whole bunch of data. For convenience, I wrote tasks that can pretty-print the data that the resource fetches.
So in the repo, as well as
check commands in
cmd/, I also added
show-resources etc as regular CLI commands. They get a little bit of acceptance testing but not much as they’re not very complex.
Next, I have the resource’s Dockerfile, which looks a bit like this:
COPY binaries/check /opt/resource/check
COPY binaries/in /opt/resource/in
COPY binaries/show-build /opt/tasks/show-build
That is: I ship the task binaries with the resource image. This has the nice side-effect of taking advantage of the cached resource image (since they’re used together).
Then, in the same repository, I provide a toplevel
tasks/ directory. Someone who wants to access the task binaries I built can check out my repo and use those directly. Here’s what the task YAML for
show-build looks like:
- name: build
I can use the same resource image, since I added my helper tasks to it as binaries.
As you can see, I also use a tag in the YAML. This is intended for cases exactly like yours. In an ideal world folks are careful to keep track of dependencies. In the actual world everyone pulls from master and only compares versions when there’s an explosion.
My pipeline for this resource is set up to keep the versioning consistent. Upon release, the task YAML is updated to the release version. The release version is tagged on the image and on the github release. Bing bang boom you can pull from master without too many surprises (though really, you ought to be using the tag on the git repo).
I think this approach ameliorates some of your pain. It’s a halfway house to “true” redistributable tasks, but it’s a fairly nice halfway house.