CI can't clone after 8.0 migration

I recentrly upgraded my gitlab server to 8.0, migrated my CI data using the CI migration guide, and push some commits, but now all my builds failed because it cannot fetch data.

docker runner states :

Cloning repository...
Cloning into '/builds/ns/project'...
fatal: Authentication failed for 'https://gitlab-ci-token:d33[...]a6@git.geekwu.org/ns/project.git/'

ERROR: Build failed with: exit code 128

I've tried to re-register runners but got no more success

I've tried to change the CI token without success

git clone the url fails too, as intended

NB: the projects are private, but was built with gitlab 7.x

1. Can you try to do git clone from cmd line and see if it works?
2. Can you check production.log of GitLab if there are any errors?
3. Can you try to add a new project to CI and see if it works?

1 -> git clone with ssh url works (git@git.geekwu.org:ns/project.git), git clone with https/token url fails:

git clone https://gitlab-ci-token:d33[...]a6@git.geekwu.org/ns/project.git/Cloning into 'project'...
fatal: Authentication failed for 'https://gitlab-ci-token:d33[...]a6@git.geekwu.org/ns/project.git/'

2 -> here are the logs when I trigger a rebuild via the retry button

root@gitlab:/var/log/gitlab/gitlab-rails# tail -f production.log| grep -v register.json
Started POST "/ci/projects/3/builds/125/retry" for 2001:41d0:fe09:7821:95eb:25fa:3fa0:58e9 at 2015-09-22 18:41:22 +0200
Processing by Ci::BuildsController#retry as HTML
  Parameters: {"authenticity_token"=>"[FILTERED]", "project_id"=>"3", "id"=>"125"}
Redirected to https://git.geekwu.org/ci/projects/3/builds/128
Completed 302 Found in 134ms (ActiveRecord: 62.3ms)
Started GET "/ci/projects/3/builds/128" for 2001:41d0:fe09:7821:95eb:25fa:3fa0:58e9 at 2015-09-22 18:41:22 +0200
Processing by Ci::BuildsController#show as HTML
  Parameters: {"project_id"=>"3", "id"=>"128"}
Completed 200 OK in 223ms (Views: 110.5ms | ActiveRecord: 18.9ms)
Started GET "/data/dsdb.git/info/refs?service=git-upload-pack" for 127.0.0.1 at 2015-09-22 18:41:25 +0200
Started GET "/data/dsdb.git/info/refs?service=git-upload-pack" for 127.0.0.1 at 2015-09-22 18:41:25 +0200
Started PUT "/ci/api/v1/builds/128.json" for 127.0.0.1 at 2015-09-22 18:41:26 +0200
Started GET "/ci/projects/3/builds/128" for 2001:41d0:fe09:7821:95eb:25fa:3fa0:58e9 at 2015-09-22 18:41:27 +0200
Processing by Ci::BuildsController#show as JSON
  Parameters: {"project_id"=>"3", "id"=>"128"}
Completed 200 OK in 31ms (Views: 0.1ms | ActiveRecord: 4.3ms)
Started GET "/ci/projects/3/builds/128" for 2001:41d0:fe09:7821:95eb:25fa:3fa0:58e9 at 2015-09-22 18:41:27 +0200
Processing by Ci::BuildsController#show as HTML
  Parameters: {"project_id"=>"3", "id"=>"128"}
Completed 200 OK in 79ms (Views: 24.7ms | ActiveRecord: 9.1ms)
^C

I cannot see any error (nor I find any error by grepping 'error' in log)

3 -> with a new project, it works (public or private)

Most probably you have different clone token in use: Can you go to database and verify that token is the same as in ci_projects and services for GitLabCiService?

The tokens are not the same. Putting the token from services into ci_projects make them works

Thanks @BastienDurel

We will fetch token from ci_projects instead of using the one stored in GitLabCiService

mentioned in issue #2572 (closed)

Can someone provide a step-by-step on how to fix this manually until it's available through the omnibus packages? I can't get any builds to run after updating due to this issue.

$ sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production
gitlabhq_production=# select properties from services where type like 'GitlabCiService' and properties like '%project_url%';

you get a list of tokens, urls from which you extract ids. Then for each project, run

gitlabhq_production=# update ci_projects set token='<the token from services>' where id=<the id from service>

if you have too many project you'll have to parse json in SQL to make an ensemblist update ;)

Thanks that did it.

Status changed to closed

This is fixed.

You can simply enter the new 'CI Settings' in GitLab, and enter the old token.

I experienced this bug in 8.3.0 as well, upgrading from 7.3.2. Is it still supposed to happen? Do I have to manually enter the token of a worker? What if I want to use more than one worker for the same project?

Hey @ayufan , I have the same problem here, I used Gitlab 8.0 (omnibus) before and wasn't able to get ci with multi-runner working so I updated to 8.3 with same result - ci not working due to authorisation failed while cloning the repository. I managed to get manually cloning working for existing project users and with the ci-token from the projects settings. So there is no problem with port 80 or http itself but probably with the access tokens. I tried to look for a solution inside the database but for this project there is no row in ci_projects-table so I cannot set the token there. Isn't it supposed to include a row for ci-enabled projects automatically in ci_projects table? I also tried to create a plain new project and enable runners and ci but no row in ci_projects is created and ci still not working.

Is there a solution or how can I fix this?

Nevermind, I found out it was'nt a gitlab ci issue any more. First it was a problem of gitlab 8.0 but after migration to 8.3 the problem was because of docker and localhost issue. Localhost of gitlab host was resolved to the docker container. I fixed it with real host name instead of localhost.