Connect to task container from regular kubernetes pod

Hello!

tl;dr: Is there a way to connect to container of a concourse task from a different container running on kubernetes?

During the integrationtesting / end-to-end-testing of our application the application send out emails. The tests should be able to receive those emails in order to validate if they are correct.
Our application consists of docker-based microservices, running in kubernetes. During the pipeline execution I would like to deploy our application via helm to kubernetes, which also sets up kubernetes services for the components. I can connect to the application from the test-task-container through such a kubernetes service. But the application must also be able to connect to the test-task-container, in order to send emails to the test.

Is this possible in Concourse? If so, can someone provide me a hint on how to make that work?

Previously we used a different tool for our pipeline in which each step of the pipeline is a docker container running in k8s (not like Concourse, which creates a runc container for each task of the pipeline in one of the concourse workers (as far as I understood the internals correct)). With this approach we were able to create a regular k8s service for the container that runs our tests (and were able to connect to that container from our application through that service).

I was able to realize this with:

  • Provide a valid kubernetes config (e.g. via params)
  • Create a Pod using kubectl in a ‘create test pod’ task (in my case with a maven docker container)
  • Upload your tests to the pod via kubectl in a ‘upload tests to test pod’ task
  • Execute the tests through kubectl and heredoc in a ‘run tests’ task and copy the results to your task-container via kubectl afterwards

Example (from my PoC):

- task: deploy test pod
  config:
    platform: linux
    image_resource:
      type: docker-image
      source:
        repository: maven
        tag: 3.6.2-slim
    params:
      KUBE_CONFIG: ((kubernetes-config.config))
    run:
      path: sh
      args:
        - -c
        - |
          mkdir -p /root/.kube/
          printf %s "${KUBE_CONFIG}" > /root/.kube/config
          curl -L https://storage.googleapis.com/kubernetes-release/release/v1.16.2/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl
          chmod +x /usr/local/bin/kubectl
          cat <<EOF > deployment.yaml
          ---
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: maven-deployment
            labels:
              app: maven
          spec:
            replicas: 1
            selector:
              matchLabels:
                app: maven
            template:
              metadata:
                name: maven
                labels:
                  app: maven
              spec:
                containers:
                - name: maven
                  command: ["sh"]
                  args: ["-c", "sleep 36000"] # overwrite command so that the container does not exit
                  image: maven:3.6.2-slim
                  ports: # define all ports that are needed for your tests
                  - containerPort: 2336
                    name: sms
                    protocol: TCP
                  - containerPort: 6000
                    name: email
                    protocol: TCP
                  - containerPort: 2514
                    name: syslog-tcp
                    protocol: TCP
          ---
          apiVersion: v1
          kind: Service
          metadata:
            name: maven
          spec:
            selector:
              app: maven
            ports: # define all ports that are needed for your tests
            - protocol: TCP
              name: sms
              port: 2336
              targetPort: 2336
            - protocol: TCP
              name: email
              port: 6000
              targetPort: 6000
            - protocol: TCP
              name: syslog-tcp
              port: 2514
              targetPort: 2514
          EOF
          kubectl apply -f deployment.yaml -n concourse-integrationtesting
- task: copy tests to test pod
  config:
    platform: linux
    image_resource:
      type: docker-image
      source:
        repository: maven
        tag: 3.6.2-slim
    params:
      KUBE_CONFIG: ((kubernetes-config.config))
    inputs:
    - name: test-git-repo
    run:
      path: sh
      args:
        - -c
        - |
          mkdir -p /root/.kube/
          printf %s "${KUBE_CONFIG}" > /root/.kube/config
          curl -L https://storage.googleapis.com/kubernetes-release/release/v1.16.2/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl
          chmod +x /usr/local/bin/kubectl
          POD_NAME=$(kubectl get pods | grep maven | awk '{print $1}')
          tar -cvf tests.tar test-git-repo/
          kubectl cp tests.tar concourse-integrationtesting/${POD_NAME}:/root/tests.tar

- task: run tests
  config:
    platform: linux
    image_resource:
      type: docker-image
      source:
        repository: maven
        tag: 3.6.2-slim
    params:
      MAVEN_SETTINGS: ((maven-settings.settings))
      KUBE_CONFIG: ((kubernetes-config.config))
    outputs:
      - name: output
    run:
      path: sh
      args:
        - -c
        - |
          mkdir -p /root/.kube/
          printf %s "${KUBE_CONFIG}" > /root/.kube/config
          curl -L https://storage.googleapis.com/kubernetes-release/release/v1.16.2/bin/linux/amd64/kubectl -o /usr/local/bin/kubectl
          chmod +x /usr/local/bin/kubectl
          POD_NAME=$(kubectl get pods -n concourse-integrationtesting | grep maven | awk '{print $1}')
          ENCODED_MAVEN_SETTINGS=$(echo ${MAVEN_SETTINGS} | base64 -w 0)


          # execute the tests on the pod. If you use heredoc, all env variables from the task can be used on the remote side :)

          kubectl exec -it ${POD_NAME} bash -n concourse-integrationtesting << EOF
          mkdir -p /root/.m2/
          echo ${ENCODED_MAVEN_SETTINGS} | base64 -d > /root/.m2/settings.xml
          cd /root
          tar -xvf tests.tar -C .
          cd test-git-repo
          mvn clean verify
          tar -cvf target.tar target/
          EOF
          kubectl cp concourse-integrationtesting/${POD_NAME}:/root/test-git-repo/target.tar output/target.tar

Note: For simplicity I always used a maven image for each task. Of course things will be more readable if you use a container that already has kubectl installed. Also you do not have to use three different tasks for that.

Important Note: Since the test-pod gets created manually via kubectl, you have to make sure that you also delete the pod / namespace afterwards. In my full PoC I used ‘ensure’ to make sure that I leave a clean state afterwards.

1 Like