Support systemd-nspawn (and maybe LXC) containers

Systemd provides a virtualization tool called systemd-nspawn. It is relatively easy to use and available almost everywhere. It would be nice if gitlab-ci-multi-runner could use it out-of-the-box.

systemd-nspawn has some interesting options:

  -x, --ephemeral
       If specified, the container is run with a temporary "btrfs"
       snapshot of its root directory (as configured with --directory=),
       that is removed immediately when the container terminates. This
       option is only supported if the root file system is "btrfs". [...]

   --tmpfs=
       Mount a tmpfs file system into the container. Takes a single
       absolute path argument that specifies where to mount the tmpfs
       instance to [...]

  --overlay=, --overlay-ro=
       Combine multiple directory trees into one overlay file system and
       mount it into the container. Takes a list of colon-separated paths
       to the directory trees to combine and the destination mount point. [...]

(Also there are some examples at the end of the man page.)

This basically mean we can have two ways of running the systemd-nspawn container:

1. systemd-nspawn --ephemeral -- this creates btrfs snapshot, runs the container and destroys the snapshot when done. Needs btrfs filesystem, but usable for very large projects.
2. Read-only root + tmpfs -- Everything in the container is read-only, except /var/lib/gitlab-runner. All changes are in memory, including checked out repository, so the clean up is very cheap. The downside is memory consumption, but nowdays we have gigabytes of memory so lets use it. Most projects need only few megabytes anyway. Advantage is very easy setup: debootstrap --include=dbus,git,build-essential stable /var/lib/container/gitlab-debian and then the runner can start the container using systemd-nspawn --read-only --directory /var/lib/container/gitlab-debian --tmpfs /var/lib/gitlab-runner.
3. RW overlay on RO root -- Like the previous approach, but instead of using tmpfs we use RW overlay (bind mount) to make /var/lib/gitlab-runner read-write while rest of the container is read only.

And since the container does not have to boot up, we only need to run the runner, implementation can be about the same as ssh executor, only using systemd-nspawn instead of ssh (as long as we can do everything in a single connection/command).

Alternative way to use systemd-nspawn containers is to use local socket-activated ephemeral container with SSH executor (the runner connects to localhost, host's systemd starts a sshd -i in a new ephemeral container without booting a container's systemd; when connection is closed the container is destroyed automatically because sshd is PID1), but we can drop ssh if the runner can call nspawn directly.

Also I found a question about LXC, where LXC container (lxc-start-ephemeral) is started by sshd and ForceCommand option, instead of using systemd's socket activation.

(Disclaimer: I'm not a systemd developer nor fan.)

 

---

 

Example of the local ephemeral (tmpfs variant) container with SSH executor:

Service which starts the container on incomming SSH connection:

#/etc/systemd/system/gitlab-runner-in-container@.service
[Unit]
Description=GitLab Runner in a Container (%i)

[Service]
ExecStart=-/bin/dash -c "exec /usr/bin/systemd-nspawn --quiet --read-only -M gitlab-runner-$$$$ -D /var/lib/container/gitlab-runner --link-journal=try-host --tmpfs=/var/lib/gitlab-runner /bin/
dash -c 'mkdir /var/run/sshd ; cp -ar /var/lib/gitlab-runner.template/. /var/lib/gitlab-runner ; exec /usr/sbin/sshd -i'"
StandardInput=socket

Note the -M argument which generates unique name of each instance. The cp populates runner's home with authorized keys and other stuff after tmpfs is mounted, but before sshd starts.

The socket:

#/etc/systemd/system/gitlab-runner-in-container.socket
[Unit]
Description=GitLab Runner in a Container

[Socket]
ListenStream=127.0.0.1:24
Accept=true

[Install]
WantedBy=sockets.target

And the runner configuration:

#/etc/gitlab-runner/config.toml
[[runners]]
  name = "my-lovely-container"
  url = "https://git.*****/ci"
  token = "*******"
  executor = "ssh"
  [runners.ssh]
    user = "gitlab-runner"
    host = "127.0.0.1"
    port = "24"
    identity_file = "/etc/gitlab-runner/id_rsa"
  [runners.cache]
    Insecure = false

The goal is to leave out ssh, so the setup is way simpler. Also it looks like ssh runner is executing individual commands using separate ssh connections, so it does not work at all...