Workers containers no longer resolve internal DNS in v5.0

#1

I updated my Concourse deployment today from 4.2.2 to 5.0. Previously, in 4.2.2, I used the --garden-dns-server option on my concourse worker command to add a local DNS server to resolve. While this works in 4.2.2, 5.0 changed the behavior (https://github.com/concourse/concourse/issues/2574). According to the issue documentation, I should be able to apply the DNS server through an environment variable CONCOURSE_GARDEN_DNS_SERVER=10.206.159.20. However, Concourse doesn’t seem to be passing the variable to Garden, and containers running on the workers no longer resolve. Any ideas how to resolve this? The workers themselves can resolve internal addresses just fine – its just the garden containers that are barfing.

0 Likes

#2

how are you setting the env var? have you confirmed it’s set in the process space?

0 Likes

#3

Yes. It is in the process space as CONCOURSE_GARDEN_DNS_SERVER. I’ve also been trying to import it via garden.inf using a few different formats…a bit of documentation on the garden config format would be welcome. Some I’ve tried in the garden config:

unknown option: garden.additional_dns_servers
unknown option: garden.dns_servers
unknown option: dns_servers
unknown option: dns_server

0 Likes

#4

How are you setting it? What does your value look like?

0 Likes

#5

export CONCOURSE_GARDEN_DNS_SERVER=10.250.160.90

garden config I’ve tried a few different formats, since I can’t find an example. I’m completely willing to accept that the problem exists between the keyboard and chair.

dns_server=10.250.160.90

dns_server: 10.250.160.90

garden.dns_server=10.250.160.90

I’ve probably tried a few others.

0 Likes

#6

you also need to set CONCOURSE_GARDEN_ALLOW_HOST_ACCESS=1 if the dns sever is running on the worker’s host

0 Likes

#7

I added that, so my environment is:

CONCOURSE_GARDEN_DNS_SERVER=10.250.160.90
CONCOURSE_GARDEN_ALLOW_HOST_ACCESS=1

/etc/resolvd.conf

nameserver 127.0.0.53
options edns0
search reddog.microsoft.com

systemd-resolve --status output

– snip –
Link 2 (eth0)
Current Scopes: DNS
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNS Servers: 10.250.5.2
10.250.4.240
10.206.159.190

The host resolves the name correctly:

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>>
– snip –
;; ANSWER SECTION:
github.mktp.io. 2992 IN A 10.206.159.180

But still can’t get garden to resolve the name. As I said, this worked under 4.x of Concourse.

0 Likes

#8

Also, hijacking the container shows nothing useful in its resolv.conf. It looks like CONCOURSE_GARDEN_DNS_SERVER is not getting added to the containers running on the worker. Explicitly adding the nameservers (via hijack) to the resolv.conf in the container works. Retrying this under 4.2.3 shows that the DNS server is propagated to the garden container. This feels more like a bug in 5.0 than a misconfiguration on my part.

0 Likes

#9

I finally have this resolved – though I was unable to resolve why CONCOURSE_GARDEN_DNS_SERVER isn’t getting passed to the garden containers. This does feel like a bug in Concourse 5.0 related to the changes introduced in https://github.com/concourse/concourse/issues/2574. I was able to resolve the issue by symlinking resolv.conf to /run/systemd/resolve/resolv.conf as outlined in https://askubuntu.com/a/1057752

This results in the hosts resolv.conf populating with the upstream DNS servers, which do get passed to the containers.

0 Likes

#10

Did you confirm the env var present in the worker process’s environment?

0 Likes