Custom docker images behaving differently on Runners
Summary
We have are running on-premise Omnibus GitLab. When using the internal docker registry and a custom image I'm getting different results when I run via a runner and when I run via my local machine. I've verified the same behavior is occurring on GitLab.com. I suspect that it is a configuration issue on my end however I've run through everything several times.
Steps to reproduce
Registry: https://gitlab.com/apou/build-images
Example CI Pipeline: https://gitlab.com/apou/custom-image-test/-/jobs/30094803
Actual behavior
Script exits because it's not able to find a path to grails
Expected behavior
Should return path to grails
Relevant logs and/or screenshots
Results from my local machine
$ docker run -it registry.gitlab.com/apou/build-images:grails-2.0.4
Unable to find image 'registry.gitlab.com/apou/build-images:grails-2.0.4' locally
grails-2.0.4: Pulling from apou/build-images
402ae6c1c9cf: Already exists
b9e2ecdf64ec: Already exists
94f147cb24f1: Already exists
d5be6b7b6169: Already exists
4cd7baadf299: Already exists
9f17971554b1: Already exists
1a05a4d24186: Pull complete
7e2f9e95458b: Pull complete
2badf8106f01: Pull complete
24f0af1b3238: Pull complete
bca135ec7195: Pull complete
b590ca89ea4b: Pull complete
Digest: sha256:3a27f09d3bb29ed498b1186200820f379c06961456c17c36c9da299c9d74fe0a
Status: Downloaded newer image for registry.gitlab.com/apou/build-images:grails-2.0.4
root@23fd2215f43f:/# which grails
/usr/local/sdkman/candidates/grails/current/bin/grails
root@23fd2215f43f:/#
Results from GitLab Runner
Running with gitlab-ci-multi-runner 9.4.2 (6d06f2e)
on docker-auto-scale (4e4528ca)
Using Docker executor with image registry.gitlab.com/apou/build-images:grails-2.0.4 ...
Using docker image sha256:a1d4a718a191e35918cec03c109e314bfcd4a070e65788e0956aaf1f6f585d08 for predefined container...
Pulling docker image registry.gitlab.com/apou/build-images:grails-2.0.4 ...
Using docker image registry.gitlab.com/apou/build-images:grails-2.0.4 ID=sha256:30bee1b15e8b30b2f3d05dd69e693e5d5d3f738717a53d3fb43a17fac0a2e0fe for build container...
Running on runner-4e4528ca-project-3983142-concurrent-0 via runner-4e4528ca-machine-1503499835-1f0b8bb4-digital-ocean-2gb...
Cloning repository...
Cloning into '/builds/apou/custom-image-test'...
Checking out 6c141f2c as master...
Skipping Git submodules setup
$ which grails
ERROR: Job failed: exit code 1
Environment description
This is happening on GitLab.com and on premise.
Used GitLab Runner version
gitlab-runner --version
Version: 9.5.0
Git revision: 413da38
Git branch: 9-5-stable
GO version: go1.8.3
Built: Tue, 22 Aug 2017 13:35:13 +0000
OS/Arch: linux/amd64