"cannot run SSH" when fetching changes in docker runner because of submodules
I often get this error at the very beginning of builds:
gitlab-ci-multi-runner 1.0.4 (014aa8c) Using Docker executor with image docker.infinit.io/infinit:buildslave-jessie-i386 ... Pulling docker image docker.infinit.io/infinit:buildslave-jessie-i386 ... Running on runner-aa4c2067-project-1-concurrent-0 via 1.buildslave.infinit.one... Fetching changes... HEAD is now at c824bfa chrono: Fix circular dependencies on OSX. From https://git.infinit.io/infinit/elle ea8ca30..eb50a40 master -> origin/master Fetching submodule drake error: cannot run ssh: No such file or directory fatal: unable to fork ERROR: Build failed with: exit code 1
Note that this is run before any command from .gitlab-ci.yml, hence I can hardly debug it. This is the error reported by git when it cannot find the
ssh executable - however, it definitely is installed in the docker image: when the step does not fail, our gitlab-ci.yml later run
git submodule update --init --recursive which fetchs submodules successfully. Entering the image, I can confirm ssh is present and functional in
The step most likely fail because, in the context of the "fetching change" step, ssh cannot be found - maybe a different docker images is used, the
PATH is altered and does not include /usr/bin, ... ?
I suppose this step runs "git fetch". Git now has fetch.recurseSubmodules set to "on-demand", which means fetching a commit that updates a submodule will fetch that submodule, hence the failure. If the build is relaunched, it will not try fetching the submodule since the root repository is already fetched and proceed with the content of .gitlab-ci.yml (which will successfully run
git submodule update over SSH). What make the problem worse is that if it uses another runner, the problem will happen again, forcing you to retry every build until it lands on a runner that was previously used for that build and already has fetched the commit.
So my suspicion is : it seems
ssh is not available in the environment that fetch changes in the docker executor. This is a problem since a mere
.gitmodules file with SSH remotes will make that step fail before we even get to the .gitlab-ci.yml content.
I'm willing to debug this further, but I'm sort of stuck and could use help with the following:
- Is the "Fetching changes" built-in step run in the docker images specified for the rest of the build ?
- What command does it exacly run ? Can I customize it ? (I would also suggesting displaying the exact command in the output)
Note that I have tried setting
fetch.recurseSubmodule to "false" in the
.gitconfig of the docker image as a workaround, which should inhibit the recursive fetching of submodules, but it doesn't seem to be taken in account. I'm in the dark here since I'm not sure what image/environment is used.