Tips re K8s deployment with Istio

Hi guys,

We’ve been successfully using Concourse on Kubernetes, deployed via the helm chart. Now, we’ve bravely/foolishly tried to apply the Istio service mesh on top, and are running into difficulties…

Istio works by adding a sidecar proxy to each pod, and using the sidecars to intercept inter-pod traffic, and apply mutual TLS on top.

By default, all pod-to-pod traffic is proxied, although this can be selectively changed.

When I deploy concourse into this environment, my workers can’t seem to register against TSA. Or more accurately, they seem to try to register, but TSA then complains that it can’t connect back to the workers, since (I believe) that the proxying process makes the worker’s source IP appear to be 127.0.0.1:

{"timestamp":"2019-11-01T05:09:26.612769835Z","level":"error","source":"tsa","message":"tsa.connection.channel.command.register.failed-to-list-volumes","data":{"command":"forward-worker","error":"Get http://10.121.97.144:44379/volumes: EOF","remote":"127.0.0.1:45856","session":"2.4.1.4805"}}
{"timestamp":"2019-11-01T05:09:26.612856155Z","level":"info","source":"tsa","message":"tsa.connection.channel.command.register.failed-to-reach-worker","data":{"baggageclaim-took":"823.272µs","command":"forward-worker","garden-took":"977.423µs","remote":"127.0.0.1:45856","session":"2.4.1.4805"}}

So I used some pod annotations to tell concourse-web and concourse-workers to bypass the proxying process, and I’ve confirmed that the traffic between them no longer passes through the sidecars.

However, concourse-web is now logging the following:

{"timestamp":"2019-11-01T19:48:44.415527250Z","level":"error","source":"tsa","message":"tsa.connection.channel.command.register.failed-to-fetch-containers","data":{"command":"forward-worker","error":"Get http://api/containers: read tcp 10.121.97.179:38786-\u003e10.121.97.179:38053: read: connection reset by peer","remote":"10.121.97.177:41344","session":"5.4.1.611"}}

Which I think is concourse-web trying to talk to itself, and being denied (for some reason).

Has anybody successfully deployed Concourse with Istio?

Thanks!
D