Pipeline Labels

I propose to introduce a feature whereby pipelines may be labeled. Labeling pipelines will allow a team to group and categorize pipelines for a number of purposes, but the one I am most interested in addressing is triggering a group of pipelines from one API call.

Currently Concourse supports triggering a pipeline from an API call like

POST /api/v1/teams/<team_name>/pipelines/<pipeline_name>/resources/<resource_name>/check/webhook?webhook_token=<concourse_webhook_token>​

This has the effect of checking the named resource as defined within the named pipeline and if the resource has changed, running the pipeline.

What I would like to do is something like

POST /api/v1/teams/<team_name>/pipeline_labels/<pipeline_label>/resources/<resource_name>/check/webhook?webhook_token=<concourse_webhook_token>​

This would have the effect of checking the named resource as defined within all pipelines associated with the named pipeline label and if the resource has changed, running those pipelines.

The impetus for this feature being used in this way is that GitHub has a hard limit of at most 20 webhooks per repository. If I have a Git resource defined in more than 20 pipelines all of which I need to run when that resource changes (and I do have this), then GitHub webhooks is not a viable solution. (And there seems to be no compunction on GitHub’s part to ease if not remove this arbitrary restriction).

This actually sounds a lot like the same problem discussed in this issue, which has an alternative proposal: https://github.com/concourse/concourse/issues/2230

It sounds like your proposal would span multiple pipelines, which is different from the proposed feature in #2230.

I’m a little hesitant to introduce something called ‘pipeline labels’ because we have an upcoming feature that may sound confusingly similar to new users: instanced pipelines.

Webhooks right now are definitely pretty crummy to deal with, and the GitHub hard cap of 20 webhooks feels like a good reason to re-think things. I’ve been wondering if we could just have a general webhook endpoint and have Concourse actually understand the payloads that are received and know which resources to check based on it.

It seems like it shouldn’t be too hard to just have Concourse know about these services. The hard part is then knowing what resources to trigger, since Concourse doesn’t know anything about how resources are configured and therefore has no way to identify them from the hook payload. :thinking:

Maybe we could have these webhook endpoints yield a string identifier like github:repo:foo/bar, and resources configured with the same “hook tag” would be checked? Maybe in the future prototypes could even support auto-detecting their supported hook tags in the Info call?

Sorry, I’m scope creeping quite a bit, but if any of this sounds interesting let me know. :sweat_smile: