Guidelines for developing and maintaining reusable concourse resources or tasks


#1

When creating components intended for reuse by multiple teams in their own Concourse pipelines, when should we use tasks vs. resources?

What are the tradeoffs of using one abstraction vs. the other for different use cases?

Notably, cf-deployment-concourse-tasks, has been extremely effective in enabling the open source CF community with a set of reusable, composable pieces. The existing implementation of cf-deployment-concourse-tasks uses tasks, but there advocates for moving that functionality to use resources instead.

Over the past year or so, I’ve seen this conversation come up in a number of places, but I wanted to start a topic here to see if we can converge on any common understanding.


I’m bringing it up now because recently the idea has surfaced a number of times independently.

One thing I feel like is not the answer is to share complete pipelines, and recently another team that was doing this seemed do reach the same conclusion:

We briefly discussed the possibility of a different, but related, offering such as making primitives (tasks), the product so that customers can choose which tasks to deploy, rather than the current offering which enforces their use of a prescribed set of tasks


In part, I think the opinion to use resources for things that interact with the outside world comes from the stated design goals of these two abstractions from the Concourse docs.

On tasks:

Going a bit further, ideally tasks are pure functions: given the same set of inputs, it should either always succeed with the same outputs or always fail. This is entirely up to your script’s level of discipline, however. Flaky tests or dependencies on the internet are the most common source of impurity.

I don’t see any explicit counterpoint in the docs on resources, but I’ve seen and heard multiple people indicate that when there are side effects, resources should be preferred. For example, Denise Yu gave a talk a while back and there is a slide (deck) in the presentation that shows this:

getting-started-with-concourse-resources-slide-7

I’ve heard the same when I was first learning Concourse. And I’ve here’s another Github issue where it came up not too long ago.

My two cents, Concourse is built to model stateful interactions via resources, not tasks. I think the cf-deployment-concourse-tasks are already feeling this friction, e.g. ops-files. If you want no ops files you need to pass a dummy input, if you want multiple ops file directory you have to write a separate task to merge them together into one dir.

This is getting off topic, but I think ideally tasks should be purely functional, no side effects outside of their inputs and outputs. A task that reaches out and messes with the world outside the container is much harder to test and reason about. Resources much more intuitively model stateful interacts. A get reads information from the outside world, a put mutates the outside world. I could go on…

But another way of interpreting the design goals of resources from the docs, is that it’s really about modeling things with have multiple versions that are created and be retrieved:

A resource is any entity that can be checked for new versions, pulled down at a specific version, and/or pushed up to idempotently create new versions. A common example would be a git repository, but it can also represent more abstract things like time itself.

Design purity aside, when do you think it makes sense to use one vs. the other? Should things like cf-deployment-concourse-tasks be converted to resources? Or should they continue to be tasks?