CI runner breaks on activated git-lfs

Summary

I have Docker CI enabled for a repository that has git-lfs enabled. Only the first run is successful. Every subsequent run fails with the following error:

Running with gitlab-ci-multi-runner 1.11.1 (a67a225)
  on dromedary-runner01 (c8ca563e)
Using Docker executor with image localhost:5000/codegenerator:latest ...
Pulling docker image localhost:5000/codegenerator:latest ...
Running on runner-c8ca563e-project-75-concurrent-0 via dromedary-runner01...
Fetching changes...
HEAD is now at 741dee6 add debug options for git and pip and make the quiet
From http://172.16.2.210/libraries/toolchain
   741dee6..c164a3a  development -> origin/development
Checking out c164a3a1 as development...
\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/post-checkout.\n
ERROR: Job failed: exit code 1

Issueing docker rm docker ps -aq`` allows for ONE more succesful run.

Assumptions: There is some Docker or Gitlab cache that is being reused and where the failure occurs.

Steps to reproduce

(How one can reproduce the issue - this is very important)

What is the current bug behavior?

CI fails

What is the expected correct behavior?

CI works fine with lfs enabled

Relevant logs and/or screenshots

In the Dockerfile, I do

# git-lfs
RUN \
    curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash && \
    apt-get install -y git-lfs

And in my test script I run

git lfs install --force

Running the CI chain with debug enabled, I get

++ [[ -d /home/gitlab-runner/builds/libraries/toolchain/.git ]]
++ echo 'Fetching changes...'
++ cd /home/gitlab-runner/builds/libraries/toolchain
++ git config fetch.recurseSubmodules false
Fetching changes...
++ rm -f .git/index.lock
++ rm -f .git/shallow.lock
++ git clean -ffdx
++ git reset --hard
HEAD is now at 76a3fa2 Merge branch 'development' of 172.16.2.210:libraries/toolchain into development
++ git remote set-url origin http://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@172.16.2.210/libraries/toolchain.git
++ git fetch origin --prune '+refs/heads/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*'
Checking out 76a3fa22 as development...
++ echo 'Checking out 76a3fa22 as development...'
++ git checkout -f -q 76a3fa225eb9d6ed51a8af07d6ad2f493d2a4a87
\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting .git/hooks/post-checkout.\n
ERROR: Job failed: exit code 1

Output of checks

Results of GitLab environment info

System information
System:		Ubuntu 14.04
Current User:	git
Using RVM:	no
Ruby Version:	2.3.3p222
Gem Version:	2.6.6
Bundler Version:1.13.7
Rake Version:	10.5.0
Redis Version:	3.2.5
Sidekiq Version:4.2.7

GitLab information
Version:	8.17.3
Revision:	f0c970e
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		http://172.16.2.210
HTTP Clone URL:	http://172.16.2.210/some-group/some-project.git
SSH Clone URL:	git@172.16.2.210:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no

GitLab Shell
Version:	4.1.1
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Checking GitLab Shell ...

GitLab Shell version >= 4.1.1 ? ... OK (4.1.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
2/1 ... ok
28/3 ... ok
28/4 ... ok
5/5 ... repository is empty
28/6 ... ok
2/7 ... ok
28/8 ... ok
28/9 ... ok
2/10 ... ok
5/11 ... ok
2/12 ... ok
15/13 ... ok
2/14 ... ok
2/15 ... ok
28/16 ... ok
2/17 ... ok
2/18 ... ok
2/19 ... ok
2/20 ... ok
28/21 ... ok
2/23 ... ok
2/24 ... ok
2/25 ... ok
2/26 ... ok
5/27 ... ok
38/30 ... ok
28/34 ... ok
28/46 ... ok
28/47 ... ok
2/48 ... ok
28/49 ... ok
28/51 ... ok
28/52 ... ok
2/53 ... ok
18/54 ... repository is empty
28/55 ... ok
73/56 ... repository is empty
73/57 ... repository is empty
69/58 ... ok
69/59 ... ok
2/62 ... ok
4/63 ... ok
84/64 ... ok
73/65 ... repository is empty
2/66 ... ok
82/68 ... ok
28/70 ... ok
28/71 ... ok
73/74 ... ok
82/75 ... ok
8/86 ... ok
28/100 ... repository is empty
82/115 ... ok
35/118 ... ok
82/121 ... ok
73/124 ... repository is empty
85/125 ... repository is empty
28/126 ... ok
28/127 ... repository is empty
28/130 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... no
  Try fixing it:
  sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
  sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} \;
  sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} \;
  For more information see:
  doc/install/installation.md in section "GitLab"
  Please fix the error above and rerun the checks.
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
2/1 ... yes
28/3 ... yes
28/4 ... yes
5/5 ... yes
28/6 ... yes
2/7 ... yes
28/8 ... yes
28/9 ... yes
2/10 ... yes
5/11 ... yes
2/12 ... yes
15/13 ... yes
2/14 ... yes
2/15 ... yes
28/16 ... yes
2/17 ... yes
2/18 ... yes
2/19 ... yes
2/20 ... yes
28/21 ... yes
2/23 ... yes
2/24 ... yes
2/25 ... yes
2/26 ... yes
5/27 ... yes
38/30 ... yes
28/34 ... yes
28/46 ... yes
28/47 ... yes
2/48 ... yes
28/49 ... yes
28/51 ... yes
28/52 ... yes
2/53 ... yes
18/54 ... yes
28/55 ... yes
73/56 ... yes
73/57 ... yes
69/58 ... yes
69/59 ... yes
2/62 ... yes
4/63 ... yes
84/64 ... yes
73/65 ... yes
2/66 ... yes
82/68 ... yes
28/70 ... yes
28/71 ... yes
73/74 ... yes
82/75 ... yes
8/86 ... yes
28/100 ... yes
82/115 ... yes
35/118 ... yes
82/121 ... yes
73/124 ... yes
85/125 ... yes
28/126 ... yes
28/127 ... yes
28/130 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.10.2)
Active users: 69

Checking GitLab ... Finished

Possible fixes

Clear Docker cache after every run.