Gitlab-Ci fails to submit builds to coordinator

After the 8.0 upgrade, the CI fails build:

WARN[0347] 33ba87b6 13 Submitting build to coordinator... failed

GitLab shows the runners as alive, even gitlab-ci-multi-runner verify shows them.

I have no clue, what i've done wrong, what can be wrong or how to fix the problem.

Here is the output of gitlab-ctl show-config: config.conf

Interesting, i did change the permission for /var/opt/gitlab/gitlab-ci/builds by hand first (as a workaround). But i reverted it before the upgrade to the final to see if omnibus does it right. Which seemed to be so.

Addittional Info: I didn't followed the migration docs (doesn't care about the history).

@jannickfahlbusch And if you reregister the runner?

@marin I registered all runners again and the problem is still there

Edit: In addition: gitlab-org/gitlab-ci-multi-runner#187 is related.

What are the permissions on /var/opt/gitlab/gitlab-ci and /var/opt/gitlab/gitlab-ci/builds?

/var/opt/gitlab/gitlab-ci:

4 drwxr-xr-x  8 git          root       4096 Sep 14 10:17 gitlab-ci

/var/opt/gitlab/gitlab-ci/builds:

4 drwx------  4 git       root       4096 Sep 14 01:14 builds

Whole dir:

4 drwxr-xr-x  8 git       root       4096 Sep 14 10:17 .
4 drwxr-xr-x 16 root      root       4096 Sep 22 11:18 ..
4 drwx------  4 git       root       4096 Sep 14 01:14 builds
4 drwx------  2 gitlab-ci root       4096 Aug  6 08:58 etc
4 drwx------  2 gitlab-ci root       4096 Jan  5  2015 home
4 drwxr-x---  2 gitlab-ci gitlab-www 4096 Sep 22 12:21 sockets
4 drwx------  4 gitlab-ci root       4096 Jun 22 18:22 tmp
4 -rw-r--r--  1 root      root          6 Sep 14 10:17 VERSION
4 drwx------  2 gitlab-ci root       4096 Jan  5  2015 working

That looks as expected.

What is the builds_path in /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml?

It's

builds_path: /var/opt/gitlab/gitlab-ci/builds

Can you show context of log/production.log when the Submitting build to coordinator... failed happens?

@ayufan Where is the file located?

If you are admin on your gitlab instance you can find it in the admin ui, otherwise i think in omnibus its here: /var/log/gitlab/gitlab-rails/production.log

I hope this is the right log:production.log

EDIT: production.log-2

Great. Please show:

ls -al /var/opt/gitlab
ls -al /var/opt/gitlab/gitlab-ci
ls -al /var/opt/gitlab/gitlab-ci/builds

Most likely this is permission problem when trying to access gitlab-ci folder.

root@git:~# ls -al /var/opt/gitlab
insgesamt 72
drwxr-xr-x 16 root         root       4096 Sep 22 11:18 .
drwxr-xr-x  3 root         root       4096 Okt 24  2014 ..
drwx------  2 git          root       4096 Sep 22 11:15 backups
-rw-------  1 root         root         38 Okt 24  2014 bootstrapped
drwxr-xr-x  2 gitlab-ci    root       4096 Sep 22 11:15 ci-backups
drwxr-x---  2 gitlab-redis gitlab-ci  4096 Sep 22 12:25 ci-redis
-rw-r--r--  1 git          git         279 Jul  7 10:29 .gitconfig
drwx------  4 git          root       4096 Okt 24  2014 git-data
drwxr-xr-x  8 git          root       4096 Sep 14 10:17 gitlab-ci
drwxr-x---  2 git          gitlab-www 4096 Sep 22 12:31 gitlab-git-http-server
drwxr-xr-x  7 git          root       4096 Sep 22 14:12 gitlab-rails
drwx------  2 git          root       4096 Apr 23 10:55 gitlab-shell
drwx------  3 root         root       4096 Sep 22 14:41 logrotate
drwxr-xr-x  3 mattermost   root       4096 Aug 24 10:24 mattermost
drwxr-x---  8 root         gitlab-www 4096 Sep 22 14:12 nginx
drwxr-xr-x  3 gitlab-psql  root       4096 Sep 22 14:12 postgresql
drwxr-x---  2 gitlab-redis git        4096 Sep 22 14:51 redis
drwx------  2 git          git        4096 Okt 24  2014 .ssh
root@git:~# ls -al /var/opt/gitlab/gitlab-ci
insgesamt 36
drwxr-xr-x  8 git       root       4096 Sep 14 10:17 .
drwxr-xr-x 16 root      root       4096 Sep 22 11:18 ..
drwx------  4 git       root       4096 Sep 14 01:14 builds
drwx------  2 gitlab-ci root       4096 Aug  6 08:58 etc
drwx------  2 gitlab-ci root       4096 Jan  5  2015 home
drwxr-x---  2 gitlab-ci gitlab-www 4096 Sep 22 12:21 sockets
drwx------  4 gitlab-ci root       4096 Jun 22 18:22 tmp
-rw-r--r--  1 root      root          6 Sep 14 10:17 VERSION
drwx------  2 gitlab-ci root       4096 Jan  5  2015 working
root@git:~# ls -al /var/opt/gitlab/gitlab-ci/builds
insgesamt 16
drwx------ 4 git       root      4096 Sep 14 01:14 .
drwxr-xr-x 8 git       root      4096 Sep 14 10:17 ..
drwxr-xr-x 6 gitlab-ci gitlab-ci 4096 Aug 27 10:56 2015_08
drwxr-xr-x 6 gitlab-ci gitlab-ci 4096 Sep 18 15:16 2015_09

@jannickfahlbusch Did you migrate the old CI data?

Nope. Had the CI before, but i do not need the old data - i reregistered the runners afterwards with gitlab-ci-multi-runner register

If you don't need the old data can you remove the two directories that are in the builds directory?

Done. It seems, that the Build-Tasks are executed correctly, but the 'logs' are not sent back to the CI-Frontend

EDIT: It is working for now. Thank you! :)

I'm leaving this open for now, because this seems to be a problem for more users after the upgrade to 8.0.0.

Yes, it's we are thinking how to fix that. /cc @marin

Milestone changed to 8.0

Same boat here: Was using gitlab-ci in 7.x, and don't care about migrations. Had to remove the contents of /var/opt/gitlab/gitlab-ci/builds before runners could submit logs to CI.

mentioned in issue #819 (closed)

Deleting the contents of /var/opt/gitlab/gitlab-ci/builds (3 folders) also did the trick for us. We were using GitLab CI previously and upgraded from 7.14 to 8.0.1.

mentioned in issue #826 (closed)

Status changed to closed

This was added to the migration doc https://gitlab.com/gitlab-org/gitlab-ce/commit/e2be419c9587b21b53c2d426d7783f99751df741

THanks for the tip of builds directory, look like a user permission issue

Before removing they were:

drwxr-xr-x.  3 gitlab-ci gitlab-ci 4096 Aug 31 14:58 2015_08
drwxr-xr-x.  3 gitlab-ci gitlab-ci 4096 Sep  1 09:32 2015_09
drwxr-xr-x. 12 gitlab-ci gitlab-ci 4096 Oct  7 12:38 2015_10

After removing new one created:

drwxr-xr-x. 3 git git 4096 Oct 15 11:55 2015_10

And it is returning output as expected

It seems like the other solution might just be a chown from gitlab-ci to gitlab user.