Running ARM image with concourse on amd64

Hi!

I am currently building a CI/CD Infrastructure for some docker base images that depend on each other. I’ve come very far. Images are built depending on each other, given them build versions, and so on.
On requirement from our security people is: Give me a list of all installed packages (apt, npn, node, …) in the image.

So i wrote a script that gets that information.
I set up a pipeline for amd64 and armv7hf.
The amd64 image all depend on one base image ubuntu:bionic
The armv7hf images all depend on this base image: balenalib/armv7hf-ubuntu:bionic

If you know balena, you’ll know that they made some magic to be able to make native arm containers runnable on amd64 with qemu (https://www.balena.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/)

So my pipeline builds all images amd64 and armv7hf fine. The requirement from security tell me:
Run a container with one previously built image, run your script and put the results on git.

This process works fine for amd64. However for armv7hf it seems, that concourse is not able to run such a “foreign” arch image. Even if it is runnable locally on my machine and working fine.

I am using many variables to keep the pipeline arch independent, so don’t wonder…

My image resource:

- name: image-((pipeline-arch))-((os))-((os_release))-base
  type: docker-image
  source:
    repository: ((docker_registry))/elbb/((pipeline-arch))/((os))/((os_release))/base
    ca_certs: {{docker_registry_ca_cert}}
    tag: latest

My git resources:

- name: source-version
  type: git
  source:
    uri: {{git_source}}
    private_key: {{privatekey}}
    branch: version-((pipeline-arch))-((os))-((os_release))

- name: source-packages
  type: git
  source:
    uri: {{git_source}}
    private_key: {{privatekey}}
    branch: packages-((pipeline-arch))-((os))-((os_release))

- name: source
  type: git
  source:
    uri: {{git_source}}
    private_key: {{privatekey}}
    branch: master
    paths:
      - ((pipeline-arch))

- name: source-pipeline
  type: git
  source:
    uri: {{git_source}}
    private_key: {{privatekey}}
    branch: master

My image build job:

- name: build-((pipeline-arch))-((os))-((os_release))-base
  public: true
  serial: true
  serial_groups: [serial]
  plan:
  - get: source
    trigger: true
  - get: source-pipeline
  - get: source-version
  - task: commit-and-push-version
    file: source-pipeline/ci/tasks/increment-version.yml
    vars: 
      os: ((os))
      os_release: ((os_release))
      arch: ((pipeline-arch))
      git_user: {{git_user}}          
      git_email: {{git_email}}
  - put: source-version  
    params: 
      repository: source-version-modified
  - put: image-((pipeline-arch))-((os))-((os_release))-base
    params:
      build: source/((pipeline-arch))/((os))/((os_release))/base/
      dockerfile: source/((pipeline-arch))/((os))/((os_release))/base/Dockerfile
      tag_file: source-version-modified/((pipeline-arch))/((os))/((os_release))/version
      tag_as_latest: true

I have this job in my pipeline that does the package-list for the base image (and any other image):

- name: packages-((pipeline-arch))-((os))-((os_release))-base
  public: true
  serial: true
  serial_groups: [serial]
  plan:
  - get: image-((pipeline-arch))-((os))-((os_release))-base
    trigger: true
  - get: source-pipeline
  - get: source-packages
  - get: source-version
  - task: packages-list
    file: source-pipeline/ci/tasks/packages.yml
    vars: 
      docker_image: ((docker_registry))/elbb/((pipeline-arch))/((os))/((os_release))/base
      path: ((pipeline-arch))/((os))/((os_release))/base
      packages_args: "--system"
      docker_registry_ca_cert: ((docker_registry_ca_cert))
      os: ((os))
      os_release: ((os_release))
      arch: ((pipeline-arch))
      git_user: {{git_user}}          
      git_email: {{git_email}}
  - put: source-packages  
    params: 
      repository: source-packages-modified

with the packages.yml

---
inputs: 
- name: source-version
- name: source-packages
outputs:
- name: source-packages-modified
platform: linux
image_resource:
  type: docker-image
  source:
    repository: ((docker_image))
    tag: latest
    ca_certs: ((docker_registry_ca_cert))
run:
  path: /bin/bash
  args:
  - -exc
  - |
    VERSION=$(cat source-version/((arch))/((os))/((os_release))/version)
    apt update && apt -yqq --no-install-recommends install git
    git clone source-packages source-packages-modified
    git clone https://github.com/siredmar/packages-list.git packages-list
    TOOL_PATH=$(pwd)
    cd source-packages-modified
    mkdir -p packages/((path))
    cd packages/((path))
    $TOOL_PATH/packages-list/packages.sh -v $VERSION -p ((path)) ((packages_args))
    git config --global user.name "((git_user))"
    git config --global user.email "((git_email))}"
    git add packages
    git commit -m "Packages list for ((path)) version '$VERSION'"

The error i am facing in the armv7hf packages-list task is:

Backend error: Exit status: 500, message: {"Type":"","Message":"runc exec: exit status 1: exec failed: container_linux.go:345: starting container process caused \"exec format error\"\n","Handle":"","ProcessID":"","Binary":""}

Here is an example of running exactly that image locally on my machine:

$ docker run -it --rm $REGISTRY/elbb/armv7hf/ubuntu/bionic/base /bin/bash
Starting with UID : 9001
user@9ab2026141c7:/$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
user@9ab2026141c7:/$ 

I know, it’s much information but can anyone give me a hint if i am doing something horribly wrong here?

I made a minimal example:

This pipeline works:

resources:
- name: base-image
  type: docker-image
  source:
    repository: ubuntu
    tag: bionic

jobs:
- name: amd64_test
  plan:
  - get: base-image
    trigger: true
  - task: run
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: ubuntu
          tag: bionic
      run:
        path: /bin/bash
        args:
          - -c
          - |
            uname -a

This pipeline does not:

resources:
- name: base-image
  type: docker-image
  source:
    repository: balenalib/armv7hf-ubuntu
    tag: bionic

jobs:
- name: arm_test
  plan:
  - get: base-image
    trigger: true
  - task: run
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: balenalib/armv7hf-ubuntu
          tag: bionic
      run:
        path: /bin/bash
        args:
          - -c
          - |
            uname -a

Running the arm container locally using docker works just fine:

docker run -it --rm balenalib/armv7hf-ubuntu:bionic /bin/bash
root@aefaf3feeece:/# uname -a
Linux aefaf3feeece 4.15.0-62-generic #69-Ubuntu SMP Wed Sep 4 20:55:53 UTC 2019 armv7l armv7l armv7l GNU/Linux

FYI: I managed running the armv7hf image on amd64 using concourse by simply installing apt-get install binfmtc qemu qemu-user-static on the host that runs the concourse containers. Those were the same requirements needed on my local host. I just forgot to install them.