Concourse CredHub TLS Certificate Expired - urgent help needed

Team,

We desperately need your help.

We had recently successfully updated NAT certificate in our Concourse env using “control-tower” ( maintain option ) and it was successful change for us. There were no errors during the cert upgrade.

But right after last day of old cert expiry we started getting error if trying to login to credhub.

credhub login

Error connecting to the targeted API: “Get https://:8844/info: x509: certificate has expired or is not yet valid”. Please validate your target and retry your request.

but if we run:

control-tower info --iaas AWS --cert-expiry

Sep 28 08:18:40 2021 GMT

so NAT cert upgrade was successful and we were assuming that it will take care of certificate in credhub too but unfortunately it did not.

We checked online references and we found following:

We tried as per above link ( although it was for “concourse-up” but we used “control-tower” ) and we see that self signed certificate was updated too but we still see same

“x509: certificate has expired or is not yet valid”.

.

I even logged in to web instance and there I see new certificate @ /var/vcap/jobs/credhub/config

Thanks in advance. Our env is unusable so any help or suggestion will really be appreciated

We are using control-tower 0.9.0

Hi @Vaid. The Control Tower GitHub repo is the best place to ask for help, as the team get an alert whenever an issue is raised.

I’ve pinged @crsimmons who may be able to take a look, but if you could raise a GitHub issue that would be great.

Hey @Vaid. I’m glad you’re finding Control Tower useful. The cert for credhub is a created by BOSH when Concourse is deployed using a CA cert that BOSH also creates. We currently don’t have a command for rotating this for you but it can be done manually quite simply.

When you deploy Control Tower creates a config S3 bucket called something like control-tower-<your deployment name>-<your region>-config. To rotate the internal certificate (used by credhub and a few other components):

  1. Download the file called director-creds.yml from your deployment’s config S3 bucket
  2. In a text editor delete the entries for ca and internal_tls. Note that both of these are of type certificate so they will have sub-keys called ca, certificate, and private_key.
  3. Upload this file to the S3 bucket making sure it is named the same thus replacing the version that was there before.
  4. Run control-tower deploy again.

This will force Control Tower to generate a new cert. I just tested it and verified that a new cert was generated for credhub and that I can interact with credhub using it.

As a user of Control Tower you should consider joining our Slack where we try our best to answer user questions about the tool.

1 Like

Thanks a ton @crsimmons.

I applied fix suggested by you and error is gone. I am able to login to credhub successfully.

I was trying something similar earlier but I was just removing “internal_tls” section and not the “ca” ( after reading one of old post where some user were facing similar issue with concourse-up ) but this time I removed both and ran deploy. It worked. :slight_smile:
I will also join Slack soon.

Thanks again.

Yeah I saw that when I tested my fix yesterday. The certificate in ca is used as the ca cert for generating the internal_tls cert so if ca is expired and you only regenerate internal_tls it will still be an invalid cert.

I’m glad you got your deployment working again.