Step-by-step guideline to deploy Concourse CI in production


#1

Hi all,

I’m looking for step-by-step instructions on how to deploy Concourse CI in production. I read through the setup & operations section, but it has limited information on best practices and how to configure a production ready Concourse CI server.

Thanks, Chris


#2

Hi r-chris,

that depends on where you want to deploy it. If you deploy Concourse with BOSH
i suggest you use the official repo concourse-bosh-deployment.

We have several clustered Concourse Bosh deployments in production and had no trouble with the concours-bosh-deployment so far.

best,
D


#3

Thanks gdenn,

I am using a CloudFoundry service provider, to which we typically deploy applications via cf push. We do not have our own cluster and I have never worked with bosh. There are a number of restrictions on that cloud provider (for example each application can only expose a single port). I was looking for a very simple setup that ideally would only involve the concourse web node and a (single) worker.

Thanks, Chris


#4

@gdenn - it looks like our cloud provider has a very similar setup to anynines - what would you recommend in that setup for a client, who is mostly concerned with keeping things simple and reducing overhead - we don’t exactly have a DevOps team to do this sort of thing :slight_smile:

On the link you provided - should there be some example configuration for CloudFoundry focused deployments (I see GCP, AWS, etc, but no mention of CloudFoundry) or am I missing something here?


#5

Hi r-chris,

BOSH is basically the glue between your IaaS provider aka GCP, AWS, vSphere… and the Cloud Foundry deployment of your provider.

Unfortunately as an end customer you most likely have only access to your own Cloud Foundry Org/Space and not to a BOSH director.

I couldn’t get it running on Cloud Foundry now, but ill check on Friday with our technical platform leader. I am sure we will figure it out.

@vito are you aware of a way to deploy Concourse using cf push as e.g. a Docker app?

Can you ask your provider if their Cloud Foundry setup allows Docker apps and has container-to-container networking enabled?

best,
D


#6

Great - thanks for the clarification - I think you are completely correct. I can see many restrictions and have mostly been told by support that those are operator level features, which are not for me to use.

It would be great to have concourse easily deployable on CloudFoundry - I was hoping it would be a natural thing to do.

In regards to Docker apps: yes, we deploy almost everything via the Docker-build-pack. Container-to-container networking has been mentioned multiple times in the past and I assume it’s been activated by now (I can see a network tap in the console and I can connect apps there).

If going the docker-way, I was wondering two things:

  • would we probably run into similar issues regarding stalling workers on restart?
  • how should I manage the keys, since I don’t want to put them into the docker container (maybe I could provide them through an ENV variable?)

#7

The things I already tried:

  • Deploy the existing docker container with the quickstart command. The initial problem with that was the port 2222 already being in use (I assume to allow cf ssh). The second problem is described here: Error trying to deploy concourse quickstart

  • Deploy a single web-node and worker-node independently. The web-node actually worked, but I couldn’t get the worker to run: How to deploy a worker node on cloud foundry

  • “Manual” deployment on AWS Lightsail: it works, but our client in this case does not want us to run things on any other provider than whom we are already using


#8

Hi r-chris,

we couldn’t deploy it on our Cloud Foundry since we have no c2c networking enabled. Furthermore i am not sure if it makes sense to deploy worker instances on Cloud Foundry as applications since they are not stateless.

If you get access to a BOSH director, i can support you with a cluster installation on the Director which is at the moment my recommended setup for you (worked good for us in the past).

You can create BOSH directors on most of the major IaaS like vSphere, Openstack, AWS, Azure…

If you have questions, contact me here or via pm.

best,
Dennis


#9

Hi gdenn,

Thanks for picking up on this again - I really appreciate that. Unfortunately, it looks like we will have to abandon Concourse CI under these circumstances, which is a shame as I have come to love this project.

From a cloud foundry customer perspective I would like to argue that it should definitely be possible to deploy concourse ci setup on top of a managed cloud foundry service pause. I agree that workers are not stateless apps in the cloud foundry sense, but it would be great if there was a way to get this running.

Now I am in a position where I would have to setup my own bosh director on another, supported cloud service (which I can not do due to restrictions set by our clients) just to get concourse working. When deciding on concourse ci - one natural thought was of course - oh look it’s Cloud Foundry related, so it must be really easy to get this to run on Cloud Foundry. I guess I always ignored the role that the bosh deployment played at the time.

Well, lesson learned and time to move on.


#10

Hi r-chris,

i can understand your problem and also share the disappointment in the fact that there is no (at least official) support for Cloud Foundry.

Maybe your company has access to a kubernetes installation? I found a Helm chart for Concourse which would make it very easy and convenient for you to provision it on kubernetes.

@checkout https://github.com/helm/charts/tree/master/stable/concourse

I can’t say if anynines will provide a Concourse as a Service as part of our public offer in the future, but i will let you know in case of.

If you decide to go the BOSH route still, you can pm me for questions.

best,
D