I’m writing this up because it seems to me that, for idempotent pipelines, there is absolutely no benefit to having put resources compared to writing a script to the same end; that putting a resource provides value only for simpler pipelines where the put parameters can encapsulate enough of the complexity by themselves.
Where am I coming from on this?
- I desire for my pipelines to be idempotent, because in the real world people hit the trigger-build button in the UI. Yes, I know I can disable the button, but I won’t disable it, because stuff happens - maybe webhooks don’t get delivered, maybe a resource doesn’t have a trigger on it because in general it shouldn’t trigger a build but here I want to check the effects of a new version of a trigger-less resource, maybe I’m fiddling with the pipeline definition and something broke and I need to rebuild, for whatever reason it’s an important escape hatch and so it’s important that the builds be idempotent.
- Say I have a resource that already exists, and I try to put to it. How should I handle this? I can’t fail upon putting the resource, because then for the same input, I have a green build the first time I run it and a red build the second time, so the build isn’t idempotent. But if I wrap the put in a try, then if it is the first time I run the build, and the put fails for a “real” reason, then the put failure gets swallowed by the try and I incorrectly get a green build!
- This ultimately means that the decision of whether or not to put to a resource is dynamically made during the pipeline itself, in order to preserve idempotence.
- Static pipelines, as they exist today, fundamentally cannot handle this kind of dynamic decision making regarding whether or not to run steps.
- Hence, instead of using a put resource - I need to always handle the put myself within a task, which can make these decisions on the spot. If downstream depends on waiting for a new version, then it’ll get picked up anyway when the resource is checked and a new version is produced during the check.
Maybe it would be best for the resource to explicitly handle idempotent cases, by reporting success when it tries to put a version which already exists, but this requires changing the underlying contract that a resource implements, because in some cases, a quick failure on the put may actually be desired behavior, and so the contract would need to change to define a standard way of expressing these preferences, which seems like a violation of the
Does somebody in the community have an insight on how to keep the value of putting a resource without resorting to various pipeline hacks (passing “real” status between tasks with files so that puts can be wrapped with try but then downstream tasks can trigger failure if needed)?