Cannot use Concourse on Windows

Hi, currently we use concourse to run builds of applications from Github repositories, until now using a single web-node and multiple worker nodes, both using Linux (standard Ubuntu AMIs on AWS).

Recently I’ve but have recently tried to expand this to include Windows applications that would require Windows worker nodes to build, however I can’t seem to run anything on my Windows worker nodes, as I keep getting error create process: Acess is denied.

I’ve tried to create a minimally reproducible scenario:

$ fly --target test workers --details
name                        containers  platform  tags  team  state    version  garden address   baggageclaim url        resource types                                                                                                                     
75263884C4D7-1567536809     1           windows   none  none  running  2.2      127.0.0.1:36359  http://127.0.0.1:37643  none                                                                                                                               
ip-172-31-18-30-1567537579  0           linux     none  none  running  2.2      127.0.0.1:45063  http://127.0.0.1:45293  bosh-io-release, bosh-io-stemcell, cf, docker-image, git, github-release, hg, mock, pool, registry-image, s3, semver, time, tracker

I’m using a sample-pipeline (i’ve tested with a few, my “real” pipeline, a few hello-world types, etc.) from the concourse/pipelines repository, here. Setup as follows:

$ fly --target test set-pipeline --config /Users/siraj/Library/Preferences/IntelliJIdea2019.1/scratches/scratch_4.yml --pipeline minimally-reproducable-test
resources:
  resource pipelines has been added:
+ name: pipelines
+ type: git
+ source:
+   uri: https://github.com/concourse/pipelines.git
+ check_every: 10000h
  
jobs:
  job do-something-on-windows-worker has been added:
+ name: do-something-on-windows-worker
+ serial: true
+ plan:
+ - get: pipelines
+   trigger: true
+ - task: do-the-windows-thing
+   privileged: true
+   config:
+     platform: windows
+     run:
+       path: pipelines/airbus.concourse.ci/josh-is-cool/scripts/windows.bat
+     inputs:
+     - name: pipelines
  
apply configuration? [yN]: y
pipeline created!
you can view your pipeline here: https://a5aa243b.ngrok.io/teams/main/pipelines/minimally-reproducable-test

the pipeline is currently paused. to unpause, either:
  - run the unpause-pipeline command:
    fly -t test unpause-pipeline -p minimally-reproducable-test
  - click play next to the pipeline in the web ui

$ fly --target test unpause-pipeline --pipeline minimally-reproducable-test
unpaused 'minimally-reproducable-test'

And finally I try to run it:

$ fly --target test trigger-job --watch --job minimally-reproducable-test/do-something-on-windows-worker
started minimally-reproducable-test/do-something-on-windows-worker #2

initializing
running pipelines/airbus.concourse.ci/josh-is-cool/scripts/windows.bat
Backend error: Exit status: 500, message: {"Type":"","Message":"create process: Access is denied.","Handle":"","ProcessID":"","Binary":""}

errored

Logs on the web node:

{"timestamp":"2019-09-03T19:43:06.861362800Z","level":"info","source":"atc","message":"atc.build-tracker.track.running","data":{"build":5,"job":"do-something-on-windows-worker","pipeline":"minimally-reproducable-test","session":"17.696"}}
{"timestamp":"2019-09-03T19:43:06.865757500Z","level":"info","source":"atc","message":"atc.build-tracker.track.get-step.initializing","data":{"build":5,"job":"do-something-on-windows-worker","job-id":2,"pipeline":"minimally-reproducable-test","session":"17.696.3","step-name":"pipelines"}}
{"timestamp":"2019-09-03T19:43:06.877874300Z","level":"info","source":"atc","message":"atc.build-tracker.track.get-step.starting","data":{"build":5,"job":"do-something-on-windows-worker","job-id":2,"pipeline":"minimally-reproducable-test","session":"17.696.3","step-name":"pipelines"}}
{"timestamp":"2019-09-03T19:43:06.967208000Z","level":"info","source":"atc","message":"atc.build-tracker.track.get-step.finished","data":{"build":5,"exit-status":0,"job":"do-something-on-windows-worker","job-id":2,"pipeline":"minimally-reproducable-test","session":"17.696.3","step-name":"pipelines"}}
{"timestamp":"2019-09-03T19:43:06.972058900Z","level":"info","source":"atc","message":"atc.build-tracker.track.task-step.initializing","data":{"build":5,"job":"do-something-on-windows-worker","job-id":2,"pipeline":"minimally-reproducable-test","session":"17.696.4","step-name":"do-the-windows-thing"}}
{"timestamp":"2019-09-03T19:43:09.612823500Z","level":"info","source":"atc","message":"atc.build-tracker.track.task-step.spawning","data":{"build":5,"job":"do-something-on-windows-worker","job-id":2,"pipeline":"minimally-reproducable-test","session":"17.696.4","step-name":"do-the-windows-thing"}}
{"timestamp":"2019-09-03T19:43:09.766504900Z","level":"info","source":"atc","message":"atc.build-tracker.track.errored","data":{"build":5,"error":"Backend error: Exit status: 500, message: {\"Type\":\"\",\"Message\":\"create process: Access is denied.\",\"Handle\":\"\",\"ProcessID\":\"\",\"Binary\":\"\"}\n","job":"do-something-on-windows-worker","pipeline":"minimally-reproducable-test","session":"17.696"}}
{"timestamp":"2019-09-03T19:43:09.784380300Z","level":"info","source":"atc","message":"atc.build-tracker.track.finish.errored","data":{"build":5,"error":"Backend error: Exit status: 500, message: {\"Type\":\"\",\"Message\":\"create process: Access is denied.\",\"Handle\":\"\",\"ProcessID\":\"\",\"Binary\":\"\"}\n","job":"do-something-on-windows-worker","pipeline":"minimally-reproducable-test","session":"17.696.5"}}

Logs on the linux worker:

{"timestamp":"2019-09-03T19:42:47.342957773Z","level":"info","source":"guardian","message":"guardian.list-containers.starting","data":{"session":"195"}}
{"timestamp":"2019-09-03T19:42:47.343334543Z","level":"info","source":"guardian","message":"guardian.list-containers.finished","data":{"session":"195"}}
{"timestamp":"2019-09-03T19:42:47.571732294Z","level":"info","source":"guardian","message":"guardian.api.garden-server.get-properties.got-properties","data":{"handle":"c98b47c7-5962-442a-4deb-bb82e98d04d2","session":"3.1.170"}}
{"timestamp":"2019-09-03T19:42:47.648056242Z","level":"info","source":"guardian","message":"guardian.run.started","data":{"handle":"c98b47c7-5962-442a-4deb-bb82e98d04d2","path":"/opt/resource/check","session":"196"}}
{"timestamp":"2019-09-03T19:42:47.648109039Z","level":"info","source":"guardian","message":"guardian.run.exec.start","data":{"handle":"c98b47c7-5962-442a-4deb-bb82e98d04d2","path":"/opt/resource/check","session":"196.1"}}
{"timestamp":"2019-09-03T19:42:47.649166578Z","level":"info","source":"guardian","message":"guardian.run.exec.exec-with-bndl.start","data":{"handle":"c98b47c7-5962-442a-4deb-bb82e98d04d2","path":"/opt/resource/check","session":"196.1.2"}}
{"timestamp":"2019-09-03T19:42:47.656445835Z","level":"info","source":"guardian","message":"guardian.run.exec.exec-with-bndl.execrunner.start","data":{"handle":"c98b47c7-5962-442a-4deb-bb82e98d04d2","id":"a4f09833-a2e8-4a0c-6af9-8be9957b87ee","path":"/opt/resource/check","session":"196.1.2.2"}}
{"timestamp":"2019-09-03T19:42:47.658971805Z","level":"info","source":"guardian","message":"guardian.run.exec.exec-with-bndl.execrunner.read-exit-fd","data":{"handle":"c98b47c7-5962-442a-4deb-bb82e98d04d2","id":"a4f09833-a2e8-4a0c-6af9-8be9957b87ee","path":"/opt/resource/check","session":"196.1.2.2"}}
{"timestamp":"2019-09-03T19:42:47.716436788Z","level":"info","source":"guardian","message":"guardian.run.exec.exec-with-bndl.execrunner.runc-exit-status","data":{"handle":"c98b47c7-5962-442a-4deb-bb82e98d04d2","id":"a4f09833-a2e8-4a0c-6af9-8be9957b87ee","path":"/opt/resource/check","session":"196.1.2.2","status":0}}
{"timestamp":"2019-09-03T19:42:47.716537430Z","level":"info","source":"guardian","message":"guardian.run.exec.exec-with-bndl.execrunner.done","data":{"handle":"c98b47c7-5962-442a-4deb-bb82e98d04d2","id":"a4f09833-a2e8-4a0c-6af9-8be9957b87ee","path":"/opt/resource/check","session":"196.1.2.2"}}
{"timestamp":"2019-09-03T19:42:47.716870864Z","level":"info","source":"guardian","message":"guardian.run.exec.exec-with-bndl.finished","data":{"handle":"c98b47c7-5962-442a-4deb-bb82e98d04d2","path":"/opt/resource/check","session":"196.1.2"}}
{"timestamp":"2019-09-03T19:42:47.717086386Z","level":"info","source":"guardian","message":"guardian.run.exec.finished","data":{"handle":"c98b47c7-5962-442a-4deb-bb82e98d04d2","path":"/opt/resource/check","session":"196.1"}}
{"timestamp":"2019-09-03T19:42:47.717261389Z","level":"info","source":"guardian","message":"guardian.run.finished","data":{"handle":"c98b47c7-5962-442a-4deb-bb82e98d04d2","path":"/opt/resource/check","session":"196"}}
{"timestamp":"2019-09-03T19:42:47.717436185Z","level":"info","source":"guardian","message":"guardian.api.garden-server.run.spawned","data":{"handle":"c98b47c7-5962-442a-4deb-bb82e98d04d2","id":"a4f09833-a2e8-4a0c-6af9-8be9957b87ee","session":"3.1.171","spec":{"Path":"/opt/resource/check","Dir":"","User":"root","Limits":{},"TTY":null}}}
{"timestamp":"2019-09-03T19:42:48.512385748Z","level":"info","source":"guardian","message":"guardian.api.garden-server.run.exited","data":{"handle":"c98b47c7-5962-442a-4deb-bb82e98d04d2","id":"a4f09833-a2e8-4a0c-6af9-8be9957b87ee","session":"3.1.171","status":0}}

Windows Worker Logs:

{"timestamp":"2019-09-03T19:43:07.131474100Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.get-volume.volume-not-found","data":{"session":"1.1.269.1","volume":"32eec12b-55f4-4dfb-4f7d-2f09d49f12c9"}}
{"timestamp":"2019-09-03T19:43:07.131474100Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.volume-not-found","data":{"session":"1.1.269","volume":"32eec12b-55f4-4dfb-4f7d-2f09d49f12c9"}}
{"timestamp":"2019-09-03T19:43:07.556225800Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.get-volume.volume-not-found","data":{"session":"1.1.273.1","volume":"8b1c9bf4-9dab-4190-56dc-ee7bd34daed0"}}
{"timestamp":"2019-09-03T19:43:07.556225800Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.volume-not-found","data":{"session":"1.1.273","volume":"8b1c9bf4-9dab-4190-56dc-ee7bd34daed0"}}
{"timestamp":"2019-09-03T19:43:08.032935000Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.get-volume.volume-not-found","data":{"session":"1.1.277.1","volume":"8c76f2e1-6684-437f-7475-fcdb3da59e81"}}
{"timestamp":"2019-09-03T19:43:08.032935000Z","level":"info","source":"baggageclaim","message":"baggageclaim.api.volume-server.get-volume.volume-not-found","data":{"session":"1.1.277","volume":"8c76f2e1-6684-437f-7475-fcdb3da59e81"}}
{"timestamp":"2019-09-03T19:43:09.013029200Z","level":"info","source":"worker","message":"worker.garden.garden-server.create.created","data":{"request":{"Handle":"ceabe03f-9183-4a68-63c7-522349573688","GraceTime":0,"RootFSPath":"","BindMounts":[{"src_path":"C:\\opt\\concourse-worker\\volumes\\live\\32eec12b-55f4-4dfb-4f7d-2f09d49f12c9\\volume","dst_path":"/scratch","mode":1},{"src_path":"C:\\opt\\concourse-worker\\volumes\\live\\8b1c9bf4-9dab-4190-56dc-ee7bd34daed0\\volume","dst_path":"/tmp/build/6cc15add","mode":1},{"src_path":"C:\\opt\\concourse-worker\\volumes\\live\\8c76f2e1-6684-437f-7475-fcdb3da59e81\\volume","dst_path":"/tmp/build/6cc15add/pipelines","mode":1}],"Network":"","Privileged":false,"Limits":{"bandwidth_limits":{},"cpu_limits":{},"disk_limits":{},"memory_limits":{},"pid_limits":{}}},"session":"1.1.226"}}
{"timestamp":"2019-09-03T19:43:09.407233900Z","level":"info","source":"worker","message":"worker.garden.garden-server.get-properties.got-properties","data":{"handle":"ceabe03f-9183-4a68-63c7-522349573688","session":"1.1.227"}}
ssage":"baggageclaim.api.volume-server.get-volume.volume-not-found","data":{"session":"1.1.277","volume":"8c76f2e1-6684-437f-7475-fcdb3da59e81"}}
{"timestamp":"2019-09-03T19:43:09.013029200Z","level":"info","source":"worker","message":"worker.garden.garden-server.create.cre
ated","data":{"request":{"Handle":"ceabe03f-9183-4a68-63c7-522349573688","GraceTime":0,"RootFSPath":"","BindMounts":[{"src_path":"C:\\opt\\concourse-worker\\
volumes\\live\\32eec12b-55f4-4dfb-4f7d-2f09d49f12c9\\volume","dst_path":"/scratch","mode":1},{"src_path":"C:\\opt\\concourse-worker\\volumes\\live\\8b1c9bf4-
9dab-4190-56dc-ee7bd34daed0\\volume","dst_path":"/tmp/build/6cc15add","mode":1},{"src_path":"C:\\opt\\concourse-worker\\volumes\\live\\8c76f2e1-6684-437f-747
5-fcdb3da59e81\\volume","dst_path":"/tmp/build/6cc15add/pipelines","mode":1}],"Network":"","Privileged":false,"Limits":{"bandwidth_limits":{},"cpu_limits":{}
,"disk_limits":{},"memory_limits":{},"pid_limits":{}}},"session":"1.1.226"}}
{"timestamp":"2019-09-03T19:43:09.407233900Z","level":"info","source":"worker","message":"worker.garden.garden-server.get-proper
ties.got-properties","data":{"handle":"ceabe03f-9183-4a68-63c7-522349573688","session":"1.1.227"}}
{"timestamp":"2019-09-03T19:43:09.500866400Z","level":"error","source":"worker","message":"worker.garden.garden-server.get-prope
rty.failed","data":{"error":"property does not exist: concourse:exit-status","handle":"ceabe03f-9183-4a68-63c7-522349573688","session":"1.1.228"}}
{"timestamp":"2019-09-03T19:43:09.613707700Z","level":"error","source":"worker","message":"worker.garden.garden-server.attach.fa
iled","data":{"error":"unknown process: task","handle":"ceabe03f-9183-4a68-63c7-522349573688","session":"1.1.229"}}
{"timestamp":"2019-09-03T19:43:09.729402200Z","level":"error","source":"worker","message":"worker.garden.garden-server.run.faile
d","data":{"error":"create process: Access is denied.","handle":"ceabe03f-9183-4a68-63c7-522349573688","session":"1.1.230"}}

I’ve confirmed that the concourse worker process is running as SYSTEM and i’ve tried also to run it as Administrator. Note that I’ve tried releases 5.2.0 and 5.4.1 and the following systems:

  • Vagrant, Server 2019 Core
  • AWS, Server 2019 DataCenter w/ Containers
  • AWS, Server 2016 DataCenter w/ Containers
  • AWS, Server 2019 Core w/ Containers
  • AWS, Server 2019 Core Standard

all with the same error. Nothing is written to any of the Event logs (Application, System, Windows Powershell, etc.).

I should probably note that a hello-world example works when assigned to Linux nodes:

$ fly --target test set-pipeline --config <path-to-config>.yml --pipeline minimally-reproducable-test-linux
jobs:
  job job has been added:
+ name: job
+ public: true
+ plan:
+ - task: simple-task
+   config:
+     platform: linux
+     image_resource:
+       type: registry-image
+       source:
+         repository: busybox
+     run:
+       path: echo
+       args:
+       - Hello, world!
  
apply configuration? [yN]: y
pipeline created!
you can view your pipeline here: <concourse-test-url>/teams/main/pipelines/minimally-reproducable-test-linux

the pipeline is currently paused. to unpause, either:
  - run the unpause-pipeline command:
    fly -t test unpause-pipeline -p minimally-reproducable-test-linux
  - click play next to the pipeline in the web ui

$ fly --target test unpause-pipeline --pipeline minimally-reproducable-test-linux
unpaused 'minimally-reproducable-test-linux'

$ fly --target test trigger-job --watch --job minimally-reproducable-test-linux/job
started minimally-reproducable-test-linux/job #1

initializing
fetching busybox@sha256:895ab622e92e18d6b461d671081757af7dbaa3b00e3e28e12505af7817f73649
ee153a04d683 [======================================] 744.9KiB/744.9KiB
running echo Hello, world!
Hello, world!
succeeded

Want to give an update - I figured it out. Seems that I must’ve botched up some of the tests, as this is what I’ve determined so far:

  • Running as NT Authority/SYSTEM will always cause this error
  • Running as Administrator and using Start-Process, i.e. Start-Process concourse -ArgumentList "worker", will cause this error
  • Running as Administrator and calling the binary directly, i.e. concourse worker will not cause the error