Load all parameters that are available in Vault


#1

Hi there all!

I have started using concourse recently and we basically have a lot of applications that we deploy in kubernetes. We also use vault to store secrets and configuration values.
What this means is that with proper parameterization we can re use our pipeline definiton and tasks across applications.

Where the problem arises is that since different applications need to load different parameters we need to duplicate our task files just to update the parameters mapping.

My thinking is that it would be nice if concourse could automatically load all available keys present in Vault under the specific pipeline directory. This way we wouldnt have to replicate our tasks files and we would basically have a form of rudimentary templating

Cheers


#2

Could you post a small example that shows the problem you are describing ?


#3

Hey Sammy, you might consider cross-posting to an issue that @vito opened recently on the RFCs repo: https://github.com/concourse/rfcs/issues/5


#4

Hey all,

Thanks for your responses.

To answer to marco-m, consider this sort of pipeline to deploy to kubernetes:

> resources:
> - name: source-code
>   type: git
>   source:
>     uri: git@github.com:organization/((repo-name)).git
>     branch: ((branch))
>     private_key: ((github-private-key))
>     disable_ci_skip: true
>     tag_filter: ((git-tag-filter))
> 
> - name: deployment-scripts
>   type: git
>   source:
>     uri: git@github.com:organisation/deployment-scripts.git
>     branch: master
>     private_key: ((github-private-key))
> 
> jobs:
> - name: Deploy.Application
>   serial: true
>   plan:
>   - get: source-code
>     trigger: true
>   - get: deployment-scripts
> 
>   - task: deploy-k8s-manifest
>     file: source-code/deployment/deploy-to-k8s.yaml

and below see the task configuration:

platform: linux

image_resource:
  type: docker-image
  source:
    repository: myrepo/my-image

inputs:
  - name: source-code
  - name: deployment-scripts

run:
  path: sh
  args:
  - -exc
  - |
    ./deployment-scripts/myscript.sh

params:
  NAMESPACE: ((namespace))
  APP_NAME: ((app-name))
  CPU_REQUESTS: ((cpu-requests))
  CPU_LIMITS: ((cpu-limits))
  MEMORY_REQUESTS: ((memory-requests))
  MEMORY_LIMITS: ((memory-limits))
  SOME_ENV_VAR: ((some-env-var))
  SOME_OTHER_VAR: ((some-other-var))

So my point is that the task could be fully reusable if we didnt have to do the mapping of env vars to vault keys, and concourse automatically load all keys present in the pipeline path in vault
Or if we could specify a file from where to load the vars.
Then i could centralize the pipeline and task configuration for all apps we deploy on kubernetes instead of replicating it because i need to load different environment variables per app.

Hope i make sense :slight_smile:
Thanks


#5

I am still confused :slight_smile: Let’s see if I understand:

You have N apps and 1 repo per app, what in the example is the source-code resource, so each pipeline with names:

  • app1-pipeline
  • appN-pipeline

has a resource with the same name source-code that points to a different repo, keyed by the ((repo-name)) param. So the ((repo-name)) comes from the Vault paths:

  • /concourse/myteam/app1-pipeline/repo-name
  • /concourse/myteam/appN-pipeline/repo-name

Then you have a common repo with resource name deployment-scripts, that contains the script deployment-scripts/myscript.sh

And each of the N apps repos contains the task file that you would like not to duplicate, the source-code/deployment/deploy-to-k8s.yaml which would be at git@github.com:organization/((repo-name)).git/deployment/deploy-to-k8s.yaml

Then the environment variables in the task params:

NAMESPACE: ((namespace))
APP_NAME: ((app-name))
CPU_REQUESTS: ((cpu-requests))
CPU_LIMITS: ((cpu-limits))
MEMORY_REQUESTS: ((memory-requests))
MEMORY_LIMITS: ((memory-limits))
SOME_ENV_VAR: ((some-env-var))
SOME_OTHER_VAR: ((some-other-var))

And some of the ((params)) are under /concourse/myteam and some others under /concourse/myteam/app3-pipeline, exploiting the Concourse param resolution rules.

Here I am confused. They look generic to me, why not removing all the source-code/deployment/deploy-to-k8s.yaml files, have only one copy at deployment-scripts/deployment/deploy-to-k8s.yaml, update the pipelines from

- task: deploy-k8s-manifest
  file: source-code/deployment/deploy-to-k8s.yaml

to

- task: deploy-k8s-manifest
  file: deployment-scripts/deployment/deploy-to-k8s.yaml

and when you need a new env variable simply append it to the list ?


#6

You got it correct marco-m with one catch.

We have multiple teams and pipelines per team, and each pipeline can have a completely different set of task params/env vars.

So the problem arises that if i centralize deploy-to-k8s.yaml then:

  1. I need to have all possible env vars/params in there
  2. I need to set all those env var/params for each team as Concourse wont skip parameters that are not set*
  3. the deploy-to-k8s.yaml file will become unmanageable (hundrends of params), which will need to be set for each team

(*) To expand on #2 imagine a team that develops in java and needs to set JVM options and another team develops in Go and JVM options are irrelevant. If I centralize deploy-to-k8s every team will need to define JVM_OPTS which is confusing.

So my solution would be to either be able to load params from a file, that way the deploy-to-k8s task can be centralized and each repo (source-code) can define its own parameters in a file present always at the same path, or that concourse automatically loads whatever is available under : /concourse/TEAM/PIPELINE && /concourse/TEAM