Concourse + credhub and ((variable) substitution


#1

I’m told that our concourse is integrated with credhub. My pipeline has ((username)), and when it runs, return error “Expected to find variables: username”.
I went to concourse web VM, /var/vcap/sys/log/uaa/uaa_events.log and other logs, there is no log for "concourse_to_credhub entry which indicates concourse is looking for username.
I then add “username: abc” to my variable file and upload pipeline, the pipeline runs w/o error.
Assuming that concourse+credhub integration is done correctly, concourse should be able to look for ((username)) in credhub (Yes, I do have entry username in credhub). And I assume it will look for the vars file only when it could not find the variable in credhub.
Thanks for your inputs


#2

You’ve actually got the precedence around the wrong way. When looking to fill in a variable the order of precedence (from first to last considered) is:

  1. --var flags passed to fly set-pipeline
  2. --load-vars-from variable files passed to fly set-pipeline
  3. credential managers (like credhub) at execution of the pipeline

The first two fill in the variables at time of setting so running fly get-pipeline returns the pipeline yaml with all the secrets filled in. With something like credhub it fills in variables only when they are requested by the pipeline.

What path did you put your username on in credhub? Concourse only looks in /concourse/<team name>/<variable> or /concourse/<team name>/<pipeline name>/<variable. I assume you have cli access to credhub since you already set a variable. You can use credhub find to figure out where you’ve put your username.

You can read more in the docs.


#3
  1. Thank you for the message, I appreciate your correction and inputs.
  2. I have a pipeline with few variables: api, ssl-flag, username, etc.
  3. I only put api, ssl-flag in the vars file, username is set in credhub
  4. After setting the pipeline, I “fly get-pipeline”, and see that the ((username)) is still there; others was resolved by the --load-vars-from file.
  5. I then execute the pipeline, and certainly I got the error: Expected to find variables: username
  6. What do I need to fix so execution of pipeline will look for ((username)) in credhub?

#4

btw, my variables are for the resource
-name: cf-env
type: cf-cli-resource
source:
api: ((api))
ssl-flag: ((flag))
username: ((username))
etc.

vars.yml
api: https://sys.abc.com
ssl-flag: true

credhub: /concourse/my-team/username

setting: fly -t my-target configure -c pipeline.yml -p testp -l vars.yml
this sets testp pipeline in my-team
fly -t my-target get-pipeline -p testp shows ((username); other variables were resolved


#5

Its possible that the uaa token you gave concourse doesn’t have access to that path in credhub. If you are on credhub version 2.0+ you are going to need to explicitly give concourse permission to read credentials on the /concourse/* path. You can do this with the following manifest snippet in your credhub job:

    authorization:
      permissions:
      - path: /concourse/*
        actors:
        - uaa-client:concourse_to_credhub
        operations:
        - read
        - write
        - delete

#6

Thank you the for input, joshzarrabi. Sorry for the late reply. The concourse-credhub integration had problem as I had suspected. It is fixed now. Again, thank you for the tip.