GitLab CI fails on merge requests

Summary

I recently migrated from a GitLab source install to the Omnibus install upgrading to GitLab 7.11.4. GitLab now fails to connect to GitLab CI on merge requests, but has no issues on normal commits/pushes. It is also worth noting that when I push to a non-standard branch (ie not master) GitLab CE does not run a GitLab CI build.

Steps to reproduce

Ensure CI is working (pressing the test button for example) then opening a merge request.

Expected behavior

GitLab CE connects to GitLab CI and runs a build for merge requests.

Observed behavior

GitLab CE fails to contact GitLab CI and asks me to check my settings, I also see the below in the logs:

==> /var/log/gitlab/gitlab-ci/production.log <==
Started GET "/projects/3/refs/drone/commits/b1a4ccae1cd77249316e7a0ee2db50779c5cfce8/status.json?token=<redacted>" for 127.0.0.1 at 2015-06-16 21:32:15 +1000
Processing by CommitsController#status as JSON
  Parameters: {"token"=>"<redacted>", "project_id"=>"3", "ref_id"=>"drone", "id"=>"b1a4ccae1cd77249316e7a0ee2db50779c5cfce8"}
Completed 404 Not Found in 4ms

ActiveRecord::RecordNotFound (ActiveRecord::RecordNotFound):
  app/controllers/commits_controller.rb:23:in `commit'

==> /var/log/gitlab/nginx/gitlab_ci_access.log <==
192.168.10.33 - - [16/Jun/2015:21:32:15 +1000] "GET /projects/3/refs/drone/commits/b1a4ccae1cd77249316e7a0ee2db50779c5cfce8/status.json?token=<redacted> HTTP/1.1" 404 36 "-" "Ruby"

Output of checks

All clear.

Current Setup

System information
System:		Ubuntu 14.04
Current User:	git
Using RVM:	no
Ruby Version:	2.1.6p336
Gem Version:	2.2.1
Bundler Version:1.5.3
Rake Version:	10.4.2
Sidekiq Version:3.3.0

GitLab information
Version:	7.11.4
Revision:	b725318
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		https://git.<redacted>.com
HTTP Clone URL:	https://git.<redacted>.com/some-project.git
SSH Clone URL:	git@git.<redacted>.com:some-project.git
Using LDAP:	no
Using Omniauth:	no

GitLab Shell
Version:	2.6.3
Repositories:	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:		/opt/gitlab/embedded/bin/git

mentioned in issue #1414 (closed)

Same steps, same Problem here

I can reproduce as well. Brand new and updated omnibus setup. "Cannot connect to CI Server" inside merge requests. Other builds seem to kick off and run ok, but merge requests fail and give this error for every project.

Can confirm the same behavior.

I tried removing and re-adding a repo and it didn't do anything, same state.

So I've just figured out the issue, it seems that when the commit is in the past, before the project was added to GitLab CI, and you do a merge request it can't find the tests for that commit since they never ran.

To reproduce this and confirm I did the following:

1. Split a branch and do a commit that causes CI to run after CI has been setup and is working
2. Open a merge request for this change
3. The merge request should show CI information as expected

This really isn't expected behaviour for me and is pretty annoying. I would expect that a merge request should run another full CI run as changes in the branch you're merging into could have changed and this new merge could break things. It shouldn't base whether CI tests pass or fail off old tests.

Another thought is that if the last commit in a merge request or the only commit has [skip ci] in the name then this will also fail. I've been able to reproduce this as well.

CI does not know anything about commits in past. So if your MR's were created before enabling CI. It's OK to see this error. I see only issue here is that we need to make error more explicit in this case. We will work on it.

What about [skip ci] tag. Yes, this is known issue and it has been alerady fixed in master and will be available in 7.12. Thank you.

TODO:

• make error explicit

/cc @dzaporozhets

CI is working fine here - Project builds on every commit. Now I create a MR -> failed to connect error

Thats a bit annoying, @vsizov. I would expect that a merge request would do another (independent) CI run. What if the branch that it is being merged from is extremely out of date?

@Mooash You can push an empty commit to that branch as a workaround.

@Ivyemm steps to reproduce, version, etc.?

@vsizov Current Omnibus Packages on Ubuntu 14.04.02 LTS Reproduce: Create Branch, Add File and Commit -> CI Build Fine, Create Merge Request -> Build fails

I've been able to repeat the above as well

I would expect that a merge request would do another (independent) CI run.

@Mooash I do not agree with you. CI result should be different only between commits. So if you already pushed commit and have CI build for it - why creating merge request should do new CI run which will cause same result?

@dzaporozhets I think the Merge Request's CI result should do another independent run. I think what @Mooash is saying is that if the branch that is being merged in did not first pull upstream changes, this test could fail where the just testing the original commit could succeed. Now, generally the branch will likely have merge conflicts and fixing those conflicts will probably require pulling in upstream changes, which will lead to a commit that is checked by the CI before it gets added to the merge request. Still, there will be instances where branches can be merged automatically, but the resulting code will not compile. You want to find that out before you finish the merge.

Also, I think it's important to realize that someone may be submitting a merge request from a fork that is not linked up to a project's CI. Are those commits never checked until after the merge has been completed? What happens in those instances?

Yeah, @csparker247 explained it a lot better then I did.

@csparker247 I might be wrong but if feels like you mix ideas of testing after-merge state and commit itself. Even if you trigger build on merge request creation - it will be build for last commit in this branch - same commit tested as you just pushed it few minutes before.

Also, I think it's important to realize that someone may be submitting a merge request from a fork that is not linked up to a project's CI. Are those commits never checked until after the merge has been completed? What happens in those instances?

In those instances you clearly see that there is no CI status in merge request page. Because fork does not have CI enabled. Even if you trigger CI on merge request creation - you should trigger CI of the fork. Because source code hosted in fork project. And if CI is not enabled for fork - nothing will happen.

Correct me if I am wrong but problem described in this issue occurs only when you:

1. have existing repository
2. enable CI
3. don't push to repository
4. immediately create merge request

As soon as you continue working - you will not see this error any more because all new pushes will receive CI status

1. have existing repository
2. enable CI
3. don't push to repository Commit -> CI build fine
4. immediately create merge request -> CI build fail

Sure, I create a branch before, commit (CI is fine), and create MergeRequest to master.

@Ivyemm ok now it looks like a bug. Can you create public project on gitlab.com where you can reproduce an issue and give me a link, please?

@dzaporozhets I'm not sure I follow but I think it's because I misunderstand what the CI status on a MR is supposed to indicate. This repo illustrates what I'm trying to say: https://gitlab.com/csparker247/merge-example/network/master

In the repo I set up, both feature and master in their pre-merge states will compile correctly, but their post-merge child will not. My expectation as a user is that when I create a merge request for feature into master, CI will test the compilation of the merged branch. But it seems like what you're saying is that that is not the case, that the CI status on the Merge Request is illustrating the status of the branch being merged in.

In this instance, I think what I'm experiencing is not a bug, but is more of a feature request. I personally feel the case I am proposing is more useful, more inline what the general user would expect, and would allow for code testing of MR's from forks that don't have CI enabled.

After all of that, I don't think I can reproduce what @Ivyemm is experiencing. Thanks for sticking through all that! I'll step out of this thread now.

My expectation as a user is that when I create a merge request for feature into master, CI will test the compilation of the merged branch.

But it does not.

But it seems like what you're saying is that that is not the case, that the CI status on the Merge Request is illustrating the status of the branch being merged in.

Correct. CI does not know anything about other branches/repositories and merge requests. It has only sha of commit it should test. Because of that CI can not pull 2 branches from different repositories, merge them and run build on top of that.

In this instance, I think what I'm experiencing is not a bug, but is more of a feature request.

Absolutely correct

@csparker247 Feel free to create request at http://feedback.gitlab.com/forums/176466-general/category/64310-gitlab-ci. And thank you for conversation

Haha, look what I found: http://feedback.gitlab.com/forums/176466-general/suggestions/5683182-merge-requests-should-trigger-a-build-on-the-main Thanks!

Still seeing this in 7.12!

Upgraded to 7.12 Keep in mind I also removed all projects from CI, and re-added the projects then ran through the steps:

Create branch, add files, commit, create merge request (can't connect to CI Server)

Similar events in the logs as original post.

Can you check if that branch exist in the CI? If not, please check logs after push to new branch in both sides: GitLab and CI.

The branch is not popping up in CI. I just created the branch after I added the project to CI.

upgraded to 7.12.0 on ubuntu 14+

still failing.

Config looks good though (tokens match)

builds work fine

@rick1

please check logs after push to new branch in both sides: GitLab and CI.

@chrisamoore You created MR with source branch - "Fix/CiMerge" and I can't find it in the CI screenshot you provided. So that, it is also related to you:

Can you check if that branch exist in the CI? If not, please check logs after push to new branch in both sides: GitLab and CI.

It's been at least 2 weeks (and a new release) since we heard from you. I'm closing this issue but if you still experience this problem, please open a new issue (but also reference the old issue(s)). Make sure to also include the necessary debugging information conforming to the issue tracker guidelines found in our contributing guidelines.

Status changed to closed

mentioned in merge request !14785