Creating workers on-demand

Hi folks,
I need some assumptions I’m making validated, based upon reading https://concourse-ci.org/concourse-worker.html

  1. Every worker needs it’s own unique tsa_worker key
  2. The public key needs to be put in the authorized_workers file, readable by all the web instances
  3. The web nodes will read any updates to the authorized_workers WITHOUT needing to be restarted after the file changes.
  4. I don’t need to run any extra commands to register the worker

If all of the above is correct, then why does the worker emit the following until the web is restarted?

failed to establish SSH connection with gateway: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

It certainly seems like assumption 3 is incorrect, because restarting the web nodes makes the error go away.

For assumption 4, I’ve tried looking for any commands on either fly or concourse, but while fly has land/retire worker, there’s no “register worker”. The concourse worker command’s help indicates that it does the registration, but it doesn’t seem to attempt to connect to the web interface to do the registration.

1 Like

I’m a Concourse newb but I’m sharing keys among workers. I’m not sure why they’d need to be unique (though I can see why that might be an ideal setup). In my current situation I have azure vm images built for concourse-web and coucourse-worker, all pre-loaded with the necessary configs. I am not doing any kind of auto-scaling but if I was I’d just spin up a new instance of a concourse-worker image.

I’m not sure why they’d need to be unique

I’m not sure either, but it could be because the key is used for identification. I’m going to give it a go not changing the key for each new worker.

Again, I’m a newb to this, but I thought it was just to encrypt the connection.

The worker key doesn’t need to be unique. You can re-use them on all of your workers. The only exception to that is team workers. They need their own keys per-team. The same rule doesn’t apply to tags because you don’t need to tell the web node ahead of time which keys belong to which tags. For team workers you do need to tell the web: this team = this key.

I don’t believe this is true, as you figured out on your own. I’d need to look at the code to figure out if Concourse watches this file. My gut feeling is that it doesn’t.

We have deployed additional workers (via Bosh) and the worker seems to register itself without us needing to restart the web node. We pass worker’s private key (same for all workers), TSA public key and TSA address.

Okay, so to summarise for the next struggling person that comes along and finds this…

When creating workers on-demand:

  1. Generate and use a SINGLE tsa_worker key for all workers that are not team-specific.
  2. Team-specific workers must have a unique tsa_worker key for each team.
  3. All tsa_worker public keys must be registered with the authorized_workers held by the web nodes BEFORE the web nodes are started.
  4. If you create new tsa_worker keys (eg: new teams, or key rotation), then you must add the new key to authorized_workers and restart all web_nodes, as there is no API to have the web nodes recognise the new keys, and the keys are not stored in the database.
  5. Similarly, retiring/unauthorizing keys requires restarting all web nodes.
1 Like