Make sure GitLab Hosted Runners are the same as GitLab.com runners

from a customer: https://gitlab.zendesk.com/agent/tickets/56306

On Gitlab, the dependencies install properly and the build completes as it should.

$ sudo update-binfmts --enable qemu-arm 
update-binfmts: warning: qemu-arm already enabled in kernel. 
[...] 
I: Running command: chroot rootfs.debootstrap/ /debootstrap/debootstrap --second-stage 
I: Keyring file not available at /usr/share/keyrings/debian-archive-keyring.gpg; switching to https mirror https://mirrors.kernel.org/debian 
I: Installing core packages... 

However on the Githost runner:

$ sudo update-binfmts --enable qemu-arm 
update-binfmts: warning: Couldn't load the binfmt_misc module. 
[...] 
I: Running command: chroot rootfs.debootstrap/ /debootstrap/debootstrap --second-stage 
chroot: failed to run command '/debootstrap/debootstrap': Exec format error 
+ exit 126 

This functionality is critical to our operation, and we solely chose to use Githost because we knew binfmt was included in the Gitlab runners.

@dblessing @MrChrisW are you aware of a reason that they would be different? If anything, I might expect GitLab Hosted runners to be more flexible.

They're not autoscaled so that's one big difference. I'm not sure if there are differences in behavior beyond that.

Hi there, I'm the customer. Are they running the same kernel/image?

Furthermore,

Gitlab Runner:

$ ls /proc/sys/fs
[...]
aio-nr
binfmt_misc
dentry-state
[...]
$ ls /proc/sys/fs/binfmt_misc
[...]
qemu-alpha
qemu-arm
[...]
$ cat /proc/sys/fs/binfmt_misc/qemu-arm
enabled
interpreter /usr/bin/qemu-arm-static
flags: OC
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff

Githost Runner:

$ ls /proc/sys/fs
[...]
aio-nr
binfmt_misc
dentry-state
[...]
$ ls /proc/sys/fs/binfmt_misc
$ cat /proc/sys/fs/binfmt_misc/qemu-arm
cat: /proc/sys/fs/binfmt_misc/qemu-arm: No such file or directory

The kernel must have binfmt_misc support, but for some reason it's not enabled, and I can't enable it. I've found this Debian mailing list entry that seems to indicate that there's a mounting mechanism for binfmt_misc. That thread is quite old, on modern Debian systems it looks like it's managed by the systemd service binfmt-support, there's also an init script . I wonder if this is perhaps disabled on the host.