Hardware in the loop testing


#1

I am working in the field of embedded software and I’d like to introduce hardware-in-the-loop tests. There are two things for which I could not find an answer:

  1. How do I get access to the hardware (e.g. ttyUSB-devices)?

Docker run command does have the --device-Option (see https://stackoverflow.com/a/24231872/5495666 and https://docs.docker.com/engine/reference/commandline/run/#add-host-device-to-container---device). However, I did not find out, how to set such options on a container created using image_resource.

  1. How do I serialize tasks from different pipelines, so that only one task is run at a time?

Multiple tasks may use the same hardware, however, there is only exactly one instance of the hardware. Can I somehow create a worker, configure it to run tasks sequentially and enforce my tasks to only run on this worker?

Thank you for your thoughts on this.
Michael.


#2

Hello Michael,

First of all, from somebody who worked in the embedded space, kudos for wanting to do automated tests also for the hardware! Regarding your questions:

  1. I don’t have a precise answer. You can have a look if https://concourse-ci.org/task-step.html#task-step-privileged helps you. See also Exposing /dev/loop devices in privileged mode. I remember somebody already asked a similar question: how to run a task on the Linux host skipping the container, but I could not find it.

  2. You have to take locks. See https://github.com/concourse/pool-resource and [ANN] Concourse Pool Boy (detect and release stale pool resource locks).


#3

Hi Marco,

first of all, thank you for your valuable input. I will give both a try. Maybe someone else has a better idea, though, as the device access is a bit uncomfortable. Should I create a ticket, which asks for exposure of the devices argument to image_resource config?


#4

Sure, feel free to open a ticket. I suggest to focus on the rationale and use case, more than a specific feature. You can also try asking on the Concourse Discord chat server.


#5

Well I think I need more help on this. Without privileged:true my task runs and correctly emits the error Error: Error: No such file or directory, cannot open /dev/serial/by-use/tty-loopback, just because the device does not exist. When I add privileged: true to the task in my plan, the task (coverage) hangs forever. Here’s my configuration:

resources:
- name: repo
  ...
- name: ci-device-names
  ...

jobs:
- name: coverage
  plan:
  - get: repo
    trigger: true
  - get: ci-device-names
  - task: prepare
    file: repo/ci/tasks/prepare.yml
  - task: coverage
    privileged: true
    file: repo/ci/tasks/coverage.yml

Any ideas?


#6

I was thinking you might be able to address this communication problem from another point of view: virtualizing the serial port over TCP.

On the host (the physical host or VM on which you have access to /dev/serial/by-use/tty-loopback), start a server that makes that special file available as a TCP listening port.

On the Concourse task, open a TCP connection, get back a fd (file descriptor) representing the socket, pass it to your test suite.

Have a look at http://www.dest-unreach.org/socat/doc/README for a possible server tool.


#7

Hi,

that’s a good way as well, however, it would also be great to know, why the task just hangs forever as soon as I enable privileged for it.

I fear that I am going to try out your new idea and that one does not work as well because of a little something. Are there any known issues about the privileged flag? Maybe I should update concourse, but that’s always a hastle because of the stalled workers issue.


#8

Seems I am affected by https://github.com/concourse/concourse/issues/1404 / https://github.com/concourse/concourse/issues/1966

I will try to set the fs driver to btrfs as suggested ther.

By the way: I am on the most current concourse release (4.2.1).


#9

Another alternative would be to implement a locking mechanism in Redis, an instance of which can be run in a Docker container. Then it is fairly easy to acquire the lock, run your code/test, and then release the lock.

Keys in Redis can also have a Time to Live set when they are created, which means the Lock can be released if your Concourse CI fails or times out (i.e. Redis will delete the key after the timeout).


#10

Also a good idea, thank you trulede.

Status update: I managed to get the first thing working, using socat, as suggested by marco-m. I will post some example code here later.

For the locking: I also like your suggestion to try locker-resource from the other thread. ([ANN] Concourse Pool Boy (detect and release stale pool resource locks)) I will try out that one first.


#11

How to set up access to (usb) serial devices for ci:

Preparing the host (my host runs Debian 9.5):

  1. install ser2net
  2. (for usb serial devices): Create udev-Rules, which assign speaking names to your usb-tty devices:
SUBSYSTEM=="tty", ATTRS{serial}=="AI02MXXX", ACTION=="add", SYMLINK+="tty-meaningful-name"

Create one such line for each usb serial device. Just adapt the serial and the symlink. You can get the serial using

udevadm info -a /dev/ttyUSB*
  1. (for non-usb serial devices): simply create a meaningful symlink - this is optional.

  2. Run the following command to re-read udev rules:

udevadm control --reload-rules && udevadm trigger
  1. Replug your USB serial cable(s)

  2. edit /etc/ser2net.conf to contain the following:

PORT:raw:60:/dev/tty-meaningful-name:115200 8DATABITS NONE 1STOPBIT

Create one such line for each usb serial device. Just set a unique PORT and the device symlink and of course your serial device configuration.

  1. Restart the ser2net-service: service ser2net restart

  2. In the test script run the following command to get access to your port:

socat -d pty,raw,nonblock,echo=0,link=/dev/tty-meaningful-name tcp:HOSTNAME:PORT &

The HOSTNAME must be a name or IP address under which the host running ser2net is accessible from within the test container.

Now you can use /dev/tty-meaningful-name as serial interface. I am not yet sure, whether you can also remotely change settings like baud rate, flow control, etc using that virtual com port.

Hope that helps someone in the future.


#12

@0815fox happy it worked, and thanks a lot for publishing the details to help people in the future!

Security-wise, since this opens a TCP port, it should be firewalled accordingly.