Splitting Jobs Into Individual Files


Is there a way to split jobs within a pipeline into individual files rather than defining them inline, like we can with tasks?


Hello, welcome to Concourse!

No, there is no built-in support. You could check out https://github.com/geofffranks/spruce, it goes in the same direction. There are trade-offs in readability and you would require some scripting to automate this.

What is your end goal ?


We have a microservices app. We keep the code for all services within the same repo. We’d like to keep all the details about each individual service within the directory where that service lives, instead of spread across various locations within the repo. It just makes life easier.

From my understanding, there are certain things you just can’t do in a task. I may be wrong here, so please correct me if that’s the case. But, for example, I don’t think it’s allowed to, as part of a task put on a docker-image resource. I think that is reserved at the job level.

I didn’t ask this, but I see a similarity with resources. I’m almost certain you’re not allowed to define resources at the job or task level, they belong at the pipeline level, correct? But, if you’re following with the design of the app, it makes a whole lot more sense to have either a job or a task be able to define its own resources, and do other things like put to a resource.

With our setup, it doesn’t make much sense otherwise anyway, because no other part of the system will ever concern itself with the details of building and deploying an image for any service other than itself. I think the design of Concourse is to facilitate complex scenarios where you have lots of inputs and outputs for interdependent stages of work, but we’re just not really in that position. Each service has a relatively few simple set of actions it needs the CI to do, and it really just makes a lot fo sense to be able to define them in isolation.


That’s correct. On the other hand, nobody forces you to put on a docker-image resource to build and push a Docker image, you can always invoke docker directly from a task, again, it is a trade-off because it becomes invisible from the pipeline. Specifically for Docker, you should also know that docker-image is deprecated. The new resource for Docker is https://github.com/concourse/registry-image-resource. The main difference is that registry-image only supports pulling and pushing. The build is delegated to a task, see as blueprint https://github.com/concourse/builder-task.

That’s correct. This, and the fact that you are using a monorepo, makes me think that you should “go with the flow” of Concourse, embrace that resources are at the pipeline level, docker images are to be manipulated as described and consider the following:

The fact that you have a monorepo doesn’t force you to represent it with only one git resource. You can use the path filtering feature of https://github.com/concourse/git-resource

`paths` :  *Optional.*  If specified (as a list of glob patterns), only changes
           to the specified files will yield new versions from  `check` .

and have multiple git resources in the same pipeline, each filtering on one directory representing a microservice.

Or you can have multiple pipelines.