Concourse picking old commit randomly


#1

Hello Colleagues,

We have seen an issue in the concourse, where concourse is picking very old commits(3 months old) randomly instead of latest commit.

Why does this happen? any idea what could have gone wrong?

we are using version: v4.2.1

BRs, Gowrisankar


#2

@eedwards-sk Can you help me here ?


#3

No clue! As always, I suggest by starting with capturing logs when it happens.


#4

@vito

After login to the container which took the wrong commit, i can see it from looks it changed from latest to old commit.

b987aa211fcf3208d5439efb962323b3fcdc9a39 - lastest commit
119c9ee662c8997c55f2fd2bbc3e2cbb69838831 - old commit

13a10fe0d60f:/tmp/build/ee14fb3b/maindir/.git/logs# cat HEAD
0000000000000000000000000000000000000000 b987aa211fcf3208d5439efb962323b3fcdc9a39 git git@localhost 1552603741 +0000 clone: from git@github.com/maindir.git
b987aa211fcf3208d5439efb962323b3fcdc9a39 119c9ee662c8997c55f2fd2bbc3e2cbb69838831 git git@localhost 1552603742 +0000 checkout: moving from master to 119c9ee662c8997c55f2fd2bbc3e2cbb69838831

Found lot of failures from web and worker process.
timestamp":“1552597205.596938133”,“source”:“tsa”,“message”:“tsa.connection.channel.register-worker.register.failed-to-fetch-containers”,“log_level”:2,“data”:{“error”:“Get http://api/containers: dial tcp 10.2.6.12:7777: connect: connection refused”,“remote”:“10.2.6.12:54488”,“session”:“423.1.1.4”}}
“message”

{“timestamp”:“1552596505.638084412”,“source”:“tsa”,“message”:“tsa.connection.channel.register-worker.register.failed-to-list-volumes”,“log_level”:2,“data”:{“error”:“Get http://10.2.6.11:7788/volumes: dial tcp 10.2.6.11:7788: connect: connection refused”,“remote”:“10.2.6.11:44576”,“session”:“250.1.1.4”}}

{“timestamp”:“1552595168.791915894”,“source”:“tsa”,“message”:“tsa.connection.tcpip-forward.failed-to-accept”,“log_level”:2,“data”:{“error”:“accept tcp 0.0.0.0:34937: use of closed network connection”,“remote”:“10.1.6.14:43348”,“session”:“8.3”}}

{“timestamp”:“1552595168.791915894”,“source”:“tsa”,“message”:“tsa.connection.tcpip-forward.failed-to-accept”,“log_level”:2,“data”:{“error”:“accept tcp 0.0.0.0:34937: use of closed network connection”,“remote”:“10.1.6.14:43348”,“session”:“8.3”}}

{“timestamp”:"1{“timestamp”:“1552595171.494493961”,“source”:“tsa”,“message”:“tsa.connection.channel.register-worker.register.failed-to-register”,“log_level”:2,“data”:{“error”:“Post http://10.2.6.0:8080/api/v1/workers?ttl=1m0s: dial tcp 10.2.6.0:8080: connect: connection refused”,“remote”:“10.1.6.12:55814”,“session”:“15.1.1.5”}}

552595168.792088032",“source”:“tsa”,“message”:“tsa.connection.channel.register-worker.wait-for-process.failed-to-close-channel”,“log_level”:2,“data”:{“error”:“EOF”,“remote”:“10.1.6.14:43348”,“session”:“8.1.1.2”}}

{“timestamp”:“1552604220.301949739”,“source”:“atc”,“message”:“atc.build-tracker.track.lock.release.failed-to-release”,“log_level”:2,“data”:{“build”:3486478,“build_id”:3486478,“error”:“lock was lost while held, possibly due to connection breakage”,“id”:[1,3486478]}}

could you help me find what is the problem ?