Skip to content

"cannot run SSH" when fetching changes in docker runner because of submodules

I often get this error at the very beginning of builds:

gitlab-ci-multi-runner 1.0.4 (014aa8c)
Using Docker executor with image docker.infinit.io/infinit:buildslave-jessie-i386 ...
Pulling docker image docker.infinit.io/infinit:buildslave-jessie-i386 ...

Running on runner-aa4c2067-project-1-concurrent-0 via 1.buildslave.infinit.one...
Fetching changes...
HEAD is now at c824bfa chrono: Fix circular dependencies on OSX.
From https://git.infinit.io/infinit/elle
   ea8ca30..eb50a40  master     -> origin/master
Fetching submodule drake
error: cannot run ssh: No such file or directory
fatal: unable to fork

ERROR: Build failed with: exit code 1

Note that this is run before any command from .gitlab-ci.yml, hence I can hardly debug it. This is the error reported by git when it cannot find the ssh executable - however, it definitely is installed in the docker image: when the step does not fail, our gitlab-ci.yml later run git submodule update --init --recursive which fetchs submodules successfully. Entering the image, I can confirm ssh is present and functional in /usr/bin/ssh.

The step most likely fail because, in the context of the "fetching change" step, ssh cannot be found - maybe a different docker images is used, the PATH is altered and does not include /usr/bin, ... ?

I suppose this step runs "git fetch". Git now has fetch.recurseSubmodules set to "on-demand", which means fetching a commit that updates a submodule will fetch that submodule, hence the failure. If the build is relaunched, it will not try fetching the submodule since the root repository is already fetched and proceed with the content of .gitlab-ci.yml (which will successfully run git submodule update over SSH). What make the problem worse is that if it uses another runner, the problem will happen again, forcing you to retry every build until it lands on a runner that was previously used for that build and already has fetched the commit.

So my suspicion is : it seems ssh is not available in the environment that fetch changes in the docker executor. This is a problem since a mere .gitmodules file with SSH remotes will make that step fail before we even get to the .gitlab-ci.yml content.

I'm willing to debug this further, but I'm sort of stuck and could use help with the following:

  • Is the "Fetching changes" built-in step run in the docker images specified for the rest of the build ?
  • What command does it exacly run ? Can I customize it ? (I would also suggesting displaying the exact command in the output)

Note that I have tried setting fetch.recurseSubmodule to "false" in the .gitconfig of the docker image as a workaround, which should inhibit the recursive fetching of submodules, but it doesn't seem to be taken in account. I'm in the dark here since I'm not sure what image/environment is used.

Thanks,