Feature Suggestion: Task Sidecars

#1

Hi,

The idea here is to run a Task with one ore more additional Tasks in parallel, where the Primary Task uses resources from those additional Tasks - somewhat similar to the Sidecar Design Pattern.

Motivation is to find a better solution to the existing DinD approach when trying to build/test Microservices. And perhaps a method which brings in more possibilities - for people to do whatever they need on the side as they build and test Containers. I can imagine “measurement” sidecars which perform some associated functions which could be useful - as a way of pushing certain kinds of Tests to a separate Container (log analysis for instance, using a log forwarder/collector).

The YAML might look a little like:

plan:
- task: foo
  ...
  sidecars:
  - task: redis
      ....
  - task: zookeeper
      ....
 run:
  path: /bin/bash
  args:
    - |
      do_something_with_redis.sh redis=redis_REDIS_URL
      do_something_with_zookeeper.sh zookeeper=red_ZOOKEEPER_URL

Perhaps the life-cycle of such a Task would be

  1. When the Task is started/entered
  2. Any Sidecar Tasks are created and started (waiting until each startup is completed).
  3. Dynamic-parameters/environment-variables from the Sidecar Tasks are collected. Resources might have been created on remote systems (i.e. Kubernetes cluster), therefore a method is needed to extract the parameters - probably the existing Task already supports that mechanism.
  4. The Primary Task is started with the dynamic-parameters/environment-variables from the Sidecar Tasks.
  5. The Primary Task runs and completes.
  6. The Sidecar Tasks are stopped - this is done in a controlled way so that each Sidecar can release any resources it might have created (i.e. a Kubernetes deployment).
  7. Perhaps a Sidecar can influence the result of the Task, that is to say, a Measurement Sidecar Task might decide that the Primary Task was not performant and that the Primary Task should be failed.

It would probably be possible to visualize the Sidecars in the UI too.

Again, the motivation is to find better ways to build and test Microservices.

0 Likes

#2

I like the idea. Just a reminder: Concourse is not only Linux, it is also Windows and Mac, where containers do not exist (more precisely: do not exist on Mac; exists but are not Concourse integrated in Windows). The “task sidecar” should be done in a way that keeps Concourse behave in a sane way for all platforms :slight_smile:

0 Likes

#3

I’ve worked around sidecars by creating a docker compose resource that runs DIND + Docker Compose.

I can then stand up an entire stack of dockerized services for UTs/ITs.

0 Likes

#4

Yes, DinD is the workaround … that this Feature Suggestion is trying to address :wink:

0 Likes