CI errors on gitlab-ce fork

Zendesk: https://gitlab.zendesk.com/agent/tickets/10721

I have a (old/long-running) fork of gitlab-ce repo. It has previously had CI enabled and set to use same runners as gitlab-ce. Since the CI merge it is no longer working and producing lots of errors. I have deactivated and reactivated CI yet the symptoms still exist.

Symptoms:

• 'Continuous Integration' does not appear in the fork's sidebar.
• Merge requests produce a 500 error on load ( @marin sees it is CI related in the logs)
• 'Test Settings' in GitLab CI service reports an error occurred.

@ayufan suggested creating this issue so we can further investigate. I may be able to resolve this by deleting my fork and re-forking but troubleshooting my issue may prove useful for fixing a bug or helping others.

Here are the logs when you attempt to open a merge request with this issue:

Completed 500 Internal Server Error in 2833ms (ActiveRecord: 42.9ms)

ActionView::Template::Error (bad URI(is not URI?): ):
     10:         %span CI build #{status}
     11:         for #{@merge_request.last_commit_short_sha}.
     12:         %span.ci-coverage
     13:         - if ci_build_details_path(@merge_request)
     14:           = link_to "View build details", ci_build_details_path(@merge_request), :"data-no-turbolink" => "data-no-turbolink"
     15:
     16:     .ci_widget
   app/helpers/merge_requests_helper.rb:39:in `ci_build_details_path'
   app/views/projects/merge_requests/widget/_heading.html.haml:13:in `block in _app_views_projects_merge_requests_widget__heading_html_haml__3628858818888431681_100574560'
   app/views/projects/merge_requests/widget/_heading.html.haml:3:in `each'
   app/views/projects/merge_requests/widget/_heading.html.haml:3:in `_app_views_projects_merge_requests_widget__heading_html_haml__3628858818888431681_100574560'

I hopped on a rails console to dig in to one of these merge requests a little. The following returns nil, so I would expect the view's conditional to be passed over. So...why the bad URI?

irb(main):012:0> mr.source_project.ci_service.build_page(mr.last_commit.sha, mr.source_branch)
=> nil

The ci_service for the source project is:

#<GitlabCiService id: 579845, type: "GitlabCiService", title: nil, project_id: 42984, created_at: "2015-05-05 09:32:57", updated_at: "2015-09-23 15:07:49", active: true, properties: {"token"=>"60e330828a6dfee488ef3078c19d9e", "project_url"=>"https://ci.gitlab.com/projects/1460"}, template: false, push_events: true, issues_events: true, merge_requests_events: true, tag_push_events: true, note_events: true>

This is now a customer issue, as well. See https://gitlab.zendesk.com/agent/tickets/8906

We know from my fork that removing the CI project entirely and re-adding it works. However, that is not an acceptable workaround for many users. Even if we need to do some rails console queries to correct something, that's better than just nuking everything.

Any ideas?

cc/ @ayufan @JobV

@dzaporozhets Maybe you have some idea why this happens?

@ayufan no I have no idea. Someone should go and debug it to find out the problem and provide a fix.

@ayufan I can do it only tomorrow so if you have time today - please take care of it

@ayufan We just had another customer report of this issue. Can you please take a look at why this method generates a Bad URI error?

https://gitlab.zendesk.com/agent/tickets/10721

Added ~113220 label

mentioned in issue #3078 (closed)

I'll give that priority and try to find the cause of this strange behavior.

@JobV shouldn't we assign a milestone to this issue?

@ayufan have you looked into this already?

Milestone changed to 8.2

Milestone changed to 8.3

We are currently having this issue, is there a workaround for this that anyone is aware of?

@dsolberg I'm looking for env where this is broken. Can you help me debug this problem?

@ayufan Also check the https://gitlab.zendesk.com/agent/tickets/10721. Customer may be willing to help debug this.

This problem will fade away with 8.3 since we plan to remove that part.

@ayufan things are merged, so close this?

@JobV @ayufan What was merged? I had a customer test 8.3 RC1 and the issue still exists.

https://gitlab.zendesk.com/agent/tickets/13699

The buggy code is removed, it should be fixed in RC2.

@ayufan This still exists after RC2. It just moved a bit. Now it occurs directly in the ci_build_details_path:

def ci_build_details_path(merge_request)
    build_url = merge_request.source_project.ci_service.build_page(merge_request.last_commit.sha, merge_request.source_branch)
    parsed_url = URI.parse(build_url) <---- 

    unless parsed_url.userinfo.blank?
      parsed_url.userinfo = ''
    end

    parsed_url.to_s
  end

I don't know what would be producing a bad build_url, but we should track it down and/or catch an error here so users don't get a 500.

Most likely the build_url is nil now, because we don't have the GitLabCiService.

2015-12-21T13:10:10.556567+00:00 10.1.0.9 97 <13>Dec 21 13:10:10 worker8 production.log: ActionView::Template::Error (bad URI(is not URI?): ):
2015-12-21T13:10:10.556570+00:00 10.1.0.9 76 <13>Dec 21 13:10:10 worker8 production.log:     26:         = succeed "." do
2015-12-21T13:10:10.556573+00:00 10.1.0.9 218 <13>Dec 21 13:10:10 worker8 production.log:     27:           = link_to commit.short_id, namespace_project_commit_path(@merge_request.source_project.namespace, @merge_request.source_project, commit), class: "monospace"
2015-12-21T13:10:10.556577+00:00 10.1.0.9 77 <13>Dec 21 13:10:10 worker8 production.log:     28:         %span.ci-coverage
2015-12-21T13:10:10.556579+00:00 10.1.0.9 117 <13>Dec 21 13:10:10 worker8 production.log:     29:         - if details_path = ci_build_details_path(@merge_request)
2015-12-21T13:10:10.556582+00:00 10.1.0.9 145 <13>Dec 21 13:10:10 worker8 production.log:     30:           = link_to "View details", details_path, :"data-no-turbolink" => "data-no-turbolink"
2015-12-21T13:10:10.556585+00:00 10.1.0.9 52 <13>Dec 21 13:10:10 worker8 production.log:     31:
2015-12-21T13:10:10.556588+00:00 10.1.0.9 66 <13>Dec 21 13:10:10 worker8 production.log:     32:     .ci_widget
2015-12-21T13:10:10.556591+00:00 10.1.0.9 112 <13>Dec 21 13:10:10 worker8 production.log:   app/helpers/merge_requests_helper.rb:31:in `ci_build_details_path'
2015-12-21T13:10:10.556593+00:00 10.1.0.9 214 <13>Dec 21 13:10:10 worker8 production.log:   app/views/projects/merge_requests/widget/_heading.html.haml:29:in `block in _app_views_projects_merge_requests_widget__heading_html_haml___3476946318292057374_61233120'
2015-12-21T13:10:10.556596+00:00 10.1.0.9 118 <13>Dec 21 13:10:10 worker8 production.log:   app/views/projects/merge_requests/widget/_heading.html.haml:18:in `each'
2015-12-21T13:10:10.556599+00:00 10.1.0.9 205 <13>Dec 21 13:10:10 worker8 production.log:   app/views/projects/merge_requests/widget/_heading.html.haml:18:in `_app_views_projects_merge_requests_widget__heading_html_haml___3476946318292057374_61233120'
2015-12-21T13:10:10.556602+00:00 10.1.0.9 202 <13>Dec 21 13:10:10 worker8 production.log:   app/views/projects/merge_requests/widget/_merged.html.haml:2:in `_app_views_projects_merge_requests_widget__merged_html_haml___2384104635445375962_61498840'
2015-12-21T13:10:10.556605+00:00 10.1.0.9 197 <13>Dec 21 13:10:10 worker8 production.log:   app/views/projects/merge_requests/widget/_show.html.haml:4:in `_app_views_projects_merge_requests_widget__show_html_haml__4420653386937396843_61624920'
2015-12-21T13:10:10.556608+00:00 10.1.0.9 185 <13>Dec 21 13:10:10 worker8 production.log:   app/views/projects/merge_requests/_show.html.haml:36:in `_app_views_projects_merge_requests__show_html_haml___1251847593700588828_63816260'
2015-12-21T13:10:10.556611+00:00 10.1.0.9 182 <13>Dec 21 13:10:10 worker8 production.log:   app/views/projects/merge_requests/show.html.haml:1:in `_app_views_projects_merge_requests_show_html_haml___1107166135428720116_63854420'
2015-12-21T13:10:10.556613+00:00 10.1.0.9 112 <13>Dec 21 13:10:10 worker8 production.log:   app/controllers/projects/merge_requests_controller.rb:51:in `show'
2015-12-21T13:10:10.556616+00:00 10.1.0.9 44 <13>Dec 21 13:10:10 worker8 production.log:
2015-12-21T13:10:10.556619+00:00 10.1.0.9 44 <13>Dec 21 13:10:10 worker8 production.log:

@ayufan This is happening on https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/401 if we need a test case.

mentioned in merge request !2167 (merged)

Status changed to closed by merge request !2167 (merged)

mentioned in commit c4eacbd2

It was happening here too https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/475 however this is no longer the case as of rc3.

mentioned in commit 25bf3eb8

mentioned in merge request !14785