Thanks for your responses! I’m very passionate about concourse so I’m glad y’all seem to perceive the same gap I do.
With the work on spaces and resources 2.0, it seems like pipelines becoming a first-class citizen would fit well into the upcoming ecosystem.
Right now concourse does make pipeline development very easy…
If I store the pipeline along with its source code, and then run a pipeline based on that commit (using that version of the pipeline), and then the pipeline fails halfway through, it’s often advantageous to re-run from where the pipeline failed if the only change needed was to the pipeline config.
(You can imagine scenarios where a build and test cycle takes 30 minutes but the packaging process failed due to a pipeline config issue, so you want to re-run the packaging step by continuing from where you left off after the build/test)
However, at that point, it means the earlier part of the build was done with a different pipeline config commit than the later part, so we can no longer say that the resulting artifact at the end was generated entirely with a specific pipeline config (since it changed part of the way through).
So this is great for pipeline development, but not necessarily for master builds where you want to ensure that an artifact can be built by a specific pipeline commit.
Ways to solve it? Well, circle / travis and others seem to force you to start over from the beginning, since every build comes from the pipeline file in that commit.
For concourse, to me I see two sort of pathways, like what vito was saying about experimentation:
- one case we want to be able to quickly iterate on the pipeline config and re-run steps of the pipeline from later stages
- another case where we want to ensure that the source commit for the pipeline config is the same version that builds it
However, with the latter, how do we ensure that the pipeline doesn’t start building a new artifact before it updates itself from the new version of the pipeline config?
For example, given:
- I have a repo
- it contains a pipeline and app source code
- the pipeline has its own repo as a resource, and triggers builds off it
With today’s behavior, if I commit and push that pipeline file change, then the old version of the pipeline will attempt to build the new commit, unless I remember to ‘set-pipeline’ before doing it.
However, if I ‘set-pipeline’ and then DON’T push the updated commit, there’s a chance that pipeline will run again (off a different trigger source or manually triggered) and build the old source using the new pipeline config.
So I wonder if the behavior we could use is the ability to say “this pipeline comes from this source control resource, and do NOT build any new commits of this source control resource inside the pipeline, until you’ve first checked the new commit for a new version of the pipeline, and then applied any pipeline changes”…
Does that make sense? I feel like this is obvious to anyone who has used concourse, but I’m struggling to describe it succinctly.