Concourse 4.2.1 - Workers Not Registering


#1

I have a web and worker node in the same subnet. All local traffic is open between them. I can even see in netstat that the worker node is communicating to the web node.

tcp6 0 0 172.21.8.99:2222 172.21.10.33:49370 ESTABLISHED -

I can see the worker node in some of my logs on the web. 172.21.10.33 is my worker, 172.21.8.99 is my web.

Sep 24 19:40:09 ip-172-21-8-99 concourse[3625]: {"timestamp":"1537818009.834708214","source":"tsa","message":"tsa.connection.channel.channel-request","log_level":1,"data":{"remote":"172.21.10.33:52630","session":"1100.1","type":"exec"}}
Sep 24 19:40:09 ip-172-21-8-99 concourse[3625]: {"timestamp":"1537818009.838180304","source":"tsa","message":"tsa.connection.channel.sweep-volume-worker.successfully-sweep-volumes","log_level":1,"data":{"remote":"172.21.10.33:52630","session":"1100.1.1"}}
Sep 24 19:40:09 ip-172-21-8-99 concourse[3625]: {"timestamp":"1537818009.838199615","source":"tsa","message":"tsa.connection.channel.sweep-volume-worker.finished-getting-sweep-volumes","log_level":1,"data":{"handles":"[]\n","remote":"172.21.10.33:52630","session":"1100.1.1"}}
Sep 24 19:40:09 ip-172-21-8-99 concourse[3625]: {"timestamp":"1537818009.838225126","source":"tsa","message":"tsa.connection.channel.sweep-volume-worker.finished-writing-sweeper-volumes","log_level":1,"data":{"bytes-written":3,"err":null,"remote":"172.21.10.33:52630","session":"1100.1.1"}}
Sep 24 19:40:11 ip-172-21-8-99 concourse[3625]: {"timestamp":"1537818011.430258751","source":"tsa","message":"tsa.connection.channel.forward-worker.heartbeat.start","log_level":1,"data":{"remote":"172.21.10.33:49370","session":"488.1.1.156","worker-address":"172.21.8.99:38598","worker-platform":"linux","worker-tags"
Sep 24 19:40:11 ip-172-21-8-99 concourse[3625]: {"timestamp":"1537818011.440593719","source":"tsa","message":"tsa.connection.channel.forward-worker.heartbeat.done","log_level":1,"data":{"remote":"172.21.10.33:49370","session":"488.1.1.156","worker-address":"172.21.8.99:38598","worker-platform":"linux","worker-tags":
Sep 24 19:40:14 ip-172-21-8-99 concourse[3625]: {"timestamp":"1537818014.753271341","source":"tsa","message":"tsa.connection.keepalive","log_level":1,"data":{"remote":"172.21.10.33:49370","session":"488","type":"keepalive"}}
Sep 24 19:40:19 ip-172-21-8-99 concourse[3625]: {"timestamp":"1537818019.753230333","source":"tsa","message":"tsa.connection.keepalive","log_level":1,"data":{"remote":"172.21.10.33:49370","session":"488","type":"keepalive"}}

I can see my web node in some of my worker node logs.

ep 24 19:40:41 ip-172-21-10-33 concourse[3925]: {"timestamp":"1537818041.450775623","source":"tsa","message":"tsa.connection.channel.forward-worker.heartbeat.done","log_level":1,"data":{"remote":"172.21.10.33:49370","session":"488.1.1.157","worker-address":"172.21.8.99:38598","worker-platform":"linux","worker-tags"
Sep 24 19:41:09 ip-172-21-10-33 concourse[3925]: {"timestamp":"1537818069.748150587","source":"worker","message":"worker.reporting-containers","log_level":1,"data":{}}
Sep 24 19:41:09 ip-172-21-10-33 concourse[3925]: {"timestamp":"1537818069.748631954","source":"guardian","message":"guardian.list-containers.starting","log_level":1,"data":{"session":"343"}}
Sep 24 19:41:09 ip-172-21-10-33 concourse[3925]: {"timestamp":"1537818069.748681068","source":"guardian","message":"guardian.list-containers.finished","log_level":1,"data":{"session":"343"}}
Sep 24 19:41:09 ip-172-21-10-33 concourse[3925]: {"timestamp":"1537818069.803464413","source":"worker","message":"worker.sweep","log_level":1,"data":{"cmd":"sweep-containers"}}
Sep 24 19:41:09 ip-172-21-10-33 concourse[3925]: {"timestamp":"1537818069.820593119","source":"worker","message":"worker.sweep","log_level":1,"data":{"cmd":"sweep-volumes"}}
Sep 24 19:41:09 ip-172-21-10-33 concourse[3925]: {"timestamp":"1537818069.837743521","source":"baggageclaim","message":"baggageclaim.api.volume-server.destroy-volumes.waiting-for-volumes-to-be-destroyed","log_level":1,"data":{"session":"5.1.465"}}
Sep 24 19:41:09 ip-172-21-10-33 concourse[3925]: {"timestamp":"1537818069.837772369","source":"baggageclaim","message":"baggageclaim.api.volume-server.destroy-volumes.destroyed","log_level":1,"data":{"session":"5.1.465"}}
Sep 24 19:41:11 ip-172-21-10-33 concourse[3925]: {"timestamp":"1537818071.450946093","source":"tsa","message":"tsa.connection.channel.forward-worker.heartbeat.start","log_level":1,"data":{"remote":"172.21.10.33:49370","session":"488.1.1.158","worker-address":"172.21.8.99:38598","worker-platform":"linux","worker-tags
Sep 24 19:41:11 ip-172-21-10-33 concourse[3925]: {"timestamp":"1537818071.461780071","source":"guardian","message":"guardian.list-containers.starting","log_level":1,"data":{"session":"344"}}
Sep 24 19:41:11 ip-172-21-10-33 concourse[3925]: {"timestamp":"1537818071.461841345","source":"guardian","message":"guardian.list-containers.finished","log_level":1,"data":{"session":"344"}}

I have started my worker node like the following:

/usr/local/bin/concourse worker --tag=cicd --team=cicd --work-dir=/var/lib/concourse --tsa-worker-private-key=/etc/concourse/worker_key --tsa-public-key=/etc/concourse/tsa_host_key.pub --tsa-host=fqdn.of.web:2222

Logging into fly, as the cicd team works.

targets:
  cicd:
    api: http://fqdn.of.web:8080
    team: cicd
    token:
      type: Bearer
      value: long_tok_value


[mgmt] fly -t cicd  status                                                                                                                                                                                                                                                                       15:42:15  ☁  cicd_account ☀
logged in successfully

Yet, no workers.

[mgmt] fly -t cicd workers                                                                                                                                                                                                                                                                       15:42:18  ☁  cicd_account ☀
name  containers  platform  tags  team  state  version

Is it possible to Group Workers so they can only run certain pipelines?
#2

ya, i have the same question. it hit me not workers when i run the demo of concourse guide