Concourse Worker permissions denied while preparing rootfs

Concourse-CI Binary Worker 4.2.2 Gives this error:

{
  "timestamp":"1550257062.072880745",
  "source":"guardian",
  "message":"guardian.api.garden-server.create.failed",
  "log_level":2,
  "data":{
    "error":"runc run: exit status 1: container_linux.go:348: starting container process caused \"process_linux.go:402: container init caused \\\"rootfs_linux.go:46: preparing rootfs caused \\\\\\\"permission denied\\\\\\\"\\\"\"\n",
    "request":{
      "Handle":"ab1c8bbe-ec8f-4c41-7534-f10e081b5351",
      "GraceTime":0,
      "RootFSPath":"raw:///opt/concourse-ci/volumes/live/f3f53232-bd2a-428b-7cf6-925ea9e080c8/volume/rootfs",
      "BindMounts":[
        {
          "src_path":"/opt/concourse-ci/volumes/live/450737c4-a993-4d97-517a-19e65ee902c1/volume",
          "dst_path":"/scratch",
          "mode":1
        },
        {
          "src_path":"/opt/concourse-ci/volumes/live/00d9a1b3-8ff6-42cb-533f-4f35e70beab1/volume",
          "dst_path":"/tmp/build/dadbfeaa",
          "mode":1
        }
      ],
      "Network":"",
      "Privileged":false,
      "Limits":{
        "bandwidth_limits":{},
        "cpu_limits":{},
        "disk_limits":{},
        "memory_limits":{},
        "pid_limits":{}
      }
    },
    "session":"3.1.47"
  }
}

However, the Docker Container of the same version gives this:

{
  "timestamp":"1550260076.080206871",
  "source":"guardian",
  "message":"guardian.api.garden-server.create.created",
  "log_level":1,
  "data":{
    "request":{
      "Handle":"d32731db-a6fb-49fe-784f-4e5eeb4561fa",
      "GraceTime":0,
      "RootFSPath":"raw:///opt/concourse-ci/volumes/live/c7cf4d6f-376b-4248-69a0-8560621ff678/volume",
      "BindMounts":[
        {
          "src_path":"/opt/concourse-ci/volumes/live/24f7269c-23e2-45d4-498e-faf537641d58/volume",
          "dst_path":"/scratch",
          "mode":1
        },
        {
          "src_path":"/opt/concourse-ci/volumes/live/cde7fbcd-cdde-4136-68be-bff88291da8b/volume",
          "dst_path":"/tmp/build/get",
          "mode":1
        }
      ],
      "Network":"",
      "Privileged":true,
      "Limits":{
        "bandwidth_limits":{},
        "cpu_limits":{},
        "disk_limits":{},
        "memory_limits":{},
        "pid_limits":{}
      }
    },
    "session":"3.1.7"
  }
}

but the error of:

$ fly --target=poc trigger-job --job test_01/hello-world --watch
started test_01/hello-world #29

initializing
mount: can't find /sys/fs/cgroup in /proc/mounts
resource script '/opt/resource/in [/tmp/build/get]' failed: exit status 1
errored

Some other Issues I saw on GitHub said that maybe there was an issue with BTRFS, or file permissions, however none of those seemed to have any effect on the jobs being ran. I want to believe that the key lies in the “Privileged”: true statement that is presnt in the Docker output, and not in the Binary output. How would one set that to be ran with the Binary version of the workers?

This also seems to only be the case when I am running the Worker on a separate host from the Web host. If I run them both on the same Host using Docker-Compose, I am able to run jobs and do things. I do not want to run the jobs on the same node because that could prove to be disastrous for production setups. I’ll do it for a Proof of Concept setup, but not production.

I’ve also read about people downgrading and it worked, but no mention of the version they downgraded to was had.

root@concourse-ci-worker.novalocal:~ ( concourse-ci-worker.novalocal )
14:09:15 # cat /etc/oracle-release
Oracle Linux Server release 7.6
root@concourse-ci-worker.novalocal:~ ( concourse-ci-worker.novalocal )
14:09:22 # uname -a
Linux concourse-ci-worker.novalocal 4.1.12-124.25.1.el7uek.x86_64 #2 SMP Tue Feb 5 12:38:44 PST 2019 x86_64 x86_64 x86_64 GNU/Linux
root@concourse-ci-worker.novalocal:~ ( concourse-ci-worker.novalocal )
14:09:24 # ./concourse_linux_amd64 --version
4.2.2