Concourse DB hit 100% CPU

Hey guys, our team just raised a issue about DB 100% CPU usage and the jobs are failing or stalling. https://github.com/concourse/concourse/issues/3948
Do you have any advice or workaround?

Are you running Postgres? If so you might want to upgrade. The version we installed when upgrading to concourse 5.1 was paired with a Postgres that had a deadlock issue. Upgrading to a new Postgres release solved this for us.

Yes we are running Postgres. What version are you running now? Did you create a github issue about it?

We upgraded from Postgres release v36 to release v37 in our bosh deployment of concourse. For us this changed Postgres from 11.2 to 11.3.
We performed the upgrade after finding https://github.com/cloudfoundry/postgres-release/issues/50 . It is not something we raised ourselves.

Thanks!
But we are using 10.x so I guess it’s unrelated.

I would ensure that your DB is properly tuned and large enough to handle your volume. Enable slow queries and see if you have a bunch piling up, causing CPU load to climb as more requests come in. This month we processed over 140,000 builds across almost 400 pipelines. Our DB runs on a 20cpu / 64GB ram Droplet VM. :slight_smile: