Support systemd-nspawn (and maybe LXC) containers
Systemd provides a virtualization tool called
systemd-nspawn. It is relatively easy to use and available almost everywhere. It would be nice if gitlab-ci-multi-runner could use it out-of-the-box.
systemd-nspawn has some interesting options:
-x, --ephemeral If specified, the container is run with a temporary "btrfs" snapshot of its root directory (as configured with --directory=), that is removed immediately when the container terminates. This option is only supported if the root file system is "btrfs". [...] --tmpfs= Mount a tmpfs file system into the container. Takes a single absolute path argument that specifies where to mount the tmpfs instance to [...] --overlay=, --overlay-ro= Combine multiple directory trees into one overlay file system and mount it into the container. Takes a list of colon-separated paths to the directory trees to combine and the destination mount point. [...]
(Also there are some examples at the end of the man page.)
This basically mean we can have two ways of running the
systemd-nspawn --ephemeral-- this creates btrfs snapshot, runs the container and destroys the snapshot when done. Needs btrfs filesystem, but usable for very large projects.
Read-only root + tmpfs -- Everything in the container is read-only, except
/var/lib/gitlab-runner. All changes are in memory, including checked out repository, so the clean up is very cheap. The downside is memory consumption, but nowdays we have gigabytes of memory so lets use it. Most projects need only few megabytes anyway. Advantage is very easy setup:
debootstrap --include=dbus,git,build-essential stable /var/lib/container/gitlab-debianand then the runner can start the container using
systemd-nspawn --read-only --directory /var/lib/container/gitlab-debian --tmpfs /var/lib/gitlab-runner.
RW overlay on RO root -- Like the previous approach, but instead of using tmpfs we use RW overlay (bind mount) to make
/var/lib/gitlab-runnerread-write while rest of the container is read only.
And since the container does not have to boot up, we only need to run the runner, implementation can be about the same as ssh executor, only using
systemd-nspawn instead of
ssh (as long as we can do everything in a single connection/command).
Alternative way to use systemd-nspawn containers is to use local socket-activated ephemeral container with SSH executor (the runner connects to localhost, host's systemd starts a
sshd -i in a new ephemeral container without booting a container's systemd; when connection is closed the container is destroyed automatically because sshd is PID1), but we can drop ssh if the runner can call nspawn directly.
Also I found a question about LXC, where LXC container (lxc-start-ephemeral) is started by sshd and ForceCommand option, instead of using systemd's socket activation.
(Disclaimer: I'm not a systemd developer nor fan.)
Example of the local ephemeral (tmpfs variant) container with SSH executor:
Service which starts the container on incomming SSH connection:
#/etc/systemd/system/gitlab-runner-in-container@.service [Unit] Description=GitLab Runner in a Container (%i) [Service] ExecStart=-/bin/dash -c "exec /usr/bin/systemd-nspawn --quiet --read-only -M gitlab-runner-$$$$ -D /var/lib/container/gitlab-runner --link-journal=try-host --tmpfs=/var/lib/gitlab-runner /bin/ dash -c 'mkdir /var/run/sshd ; cp -ar /var/lib/gitlab-runner.template/. /var/lib/gitlab-runner ; exec /usr/sbin/sshd -i'" StandardInput=socket
Note the -M argument which generates unique name of each instance. The
cp populates runner's home with authorized keys and other stuff after tmpfs is mounted, but before sshd starts.
#/etc/systemd/system/gitlab-runner-in-container.socket [Unit] Description=GitLab Runner in a Container [Socket] ListenStream=127.0.0.1:24 Accept=true [Install] WantedBy=sockets.target
And the runner configuration:
#/etc/gitlab-runner/config.toml [[runners]] name = "my-lovely-container" url = "https://git.*****/ci" token = "*******" executor = "ssh" [runners.ssh] user = "gitlab-runner" host = "127.0.0.1" port = "24" identity_file = "/etc/gitlab-runner/id_rsa" [runners.cache] Insecure = false
The goal is to leave out ssh, so the setup is way simpler. Also it looks like ssh runner is executing individual commands using separate ssh connections, so it does not work at all...