Conditional Resource Checking?

Hi

Are there ways to limit the amount of resource checking that happens - other than setting the “check_every” parameter (which TBH seems a bit too simple for many cases) or using webhooks.
What I was thinking of was…

  • The current method (check periodically) is fine for resources that have triggers associated - or webhooks.
  • For any resources that don’t have triggers or webhooks, automatically check only when the pipeline starts. If the pipeline isn’t running and the resource hasn’t got a trigger, don’t keep checking it.
    and possibly:
  • for resources that are never ‘got’ only ‘put’, perhaps don’t check them at all (but if they were checked once as per the second point above, that would be OK too).

What do you think? Is that feasible?

@evanchaoli just PR’d this, it’ll be in v6. :slight_smile: (#5149)

There’s also a PR to have an higher operator-configurable default check interval for resources which configure check_every. (#5091) I think this is pretty straightforward and makes sense to merge, maybe even with a default higher than 1m. I don’t know if it’ll make it in for v6 though since we’re pretty close to shipping.

Re: webhooks overall, I put some thoughts in Pipeline Labels and hope to turn those thoughts into a proper item on the roadmap.

This approach raises a lot more questions than you’d think, given how pipeline scheduling currently works, which is to compute the appropriate versions for a given build prior to scheduling it. If checking is deferred until some later time (i.e. when running a build), we’d have to change this, as it’d be a circular dependency.

I think the bigger fish to fry will probably be the general webhooks implementation. :+1:

Hi Vito,
Thanks for the quick reply - yes, these look like some good options.

evanchaoli just PR’d this, it’ll be in v6. #5149

Yes, that looks like it would help a lot. It would lighten the unnecessary load we place on our Docker registry (and Git repos) - and hopefully Concourse in general too.

Also

…PR to have an higher operator-configurable default check interval for resources…

and
Add a default resource check interval for webhooked resources #5088
Yes, this also looks very interesting and useful - providing the webhook trigger will always pull the latest change (that caused the webhook). Which I guess it would do regardless of a resource check? I.e. you wouldn’t want a triggered pipeline to pull the resource state from before the commit that caused the trigger… if I’m making sense there.

This approach raises a lot more questions than you’d think

Yes, I suspected it could be more complicated that I thought! :thinking:

I guess I just like to avoid hard-coded times (pauses, sleeps, etc.) generally. Having to estimate check_every values on a per-resource basis can be useful at times - but a default “you take care of it, Concourse” kind of option would be great - if you know what I mean… :grin:

Your reply with:

…have a general webhook endpoint and have Concourse actually understand the payloads that are received and know which resources to check based on it…

sounds like it would help a lot.

Thanks for the info and links to other related PRs and discussions. :+1:

Hi @vito,
Thanks for the reply and info. Re your comment…

“This approach raises a lot more questions than you’d think, given how pipeline scheduling currently works, which is to compute the appropriate versions for a given build prior to scheduling it.”

You think if I created a “refresh” job at the start of my pipeline that started a container with fly installed - and then ran a fly resources command to get a list of resources for the pipeline, followed by fly check-resource for each item in the resource list to refresh all resources in the pipeline before it continues…

Would that be an option, do you think?
Or will I be causing myself problems in future by doing something like this?

:thinking:

Sounds a bit convoluted but it’d do what you want, the only problem would be authenticating the inner fly command.

I don’t really think it’s worth the babysitting to be honest. The default interval of 1 minute isn’t really intended to be changed all the time, it’s meant to be just slow enough to not kill servers and just fast enough that you can just leave Concourse alone and your pipeline will run on its own without having to kick it too often.

Hi @vito,
Yes, I know what you mean re. ‘convoluted’. It does seem like a bit of a hack perhaps. :thinking:
I think the surprise for us was when we started measuring just how much load Concourse was placing on our Docker registry (though admittedly that was with default 1 min intervals everywhere).
I guess I was trying to see a way around setting a static interval - I’m not too keen on it myself.
If I try the pre-build check method, I’ll add a post here and let folks know it goes.
Thanks,
G