Cannot connect to CI server error messages

This happened to @sytses and others lately: https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/6629

If you look at the JSON data, pipeline is null:

{"id":4056442,"iid":6629,"author_id":424775,"description":"Results in:\r\n\r\n![image](/uploads/6a8ea99b096112f644f5b65586a21584/image.png)","lock_version":1,"milestone_id":null,"position":0,"state":"opened","title":"Reorder comparisons","updated_by_id":424775,"created_at":"2017-06-29T21:41:39.936Z","updated_at":"2017-06-29T22:10:33.726Z","deleted_at":null,"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null,"in_progress_merge_commit_sha":null,"locked_at":null,"merge_commit_sha":null,"merge_error":null,"merge_params":{"force_remove_source_branch":"1"},"merge_status":"can_be_merged","merge_user_id":null,"merge_when_pipeline_succeeds":false,"source_branch":"reorder-comparisons","source_project_id":7764,"target_branch":"master","target_project_id":7764,"approvals_before_merge":null,"squash":false,"rebase_commit_sha":null,"rebase_in_progress":false,"should_be_rebased":false,"ff_only_enabled":false,"can_push_to_source_branch":true,"rebase_path":null,"approvals_path":null,"merge_event":null,"closed_event":null,"author":{"name":"Mark Pundsack","username":"markpundsack","id":424775,"state":"active","avatar_url":"https://secure.gravatar.com/avatar/26cf0d1d89d41b47dc9a9d70a20d721d?s=80\u0026d=identicon","web_url":"https://gitlab.com/markpundsack","path":"/markpundsack"},"merge_user":null,"diff_head_sha":"48d942a6218b957a706647b419b49b14e6c00764","merge_commit_message":"Merge branch 'reorder-comparisons' into 'master'\n\nReorder comparisons\n\nSee merge request !6629","pipeline":null,"work_in_progress":false,"source_branch_exists":true,"mergeable_discussions_state":true,"branch_missing":false,"commits_count":1,"has_conflicts":false,"can_be_merged":true,"remove_source_branch":true,"project_archived":false,"only_allow_merge_if_pipeline_succeeds":true,"has_ci":true,"ci_status":null,"issues_links":{"assign_to_closing":null,"closing":"","mentioned_but_not_closing":""},"source_branch_with_namespace_link":"\u003ca href=\"/gitlab-com/www-gitlab-com/commits/reorder-comparisons\"\u003ereorder-comparisons\u003c/a\u003e","source_branch_path":"/gitlab-com/www-gitlab-com/branches/reorder-comparisons","current_user":{"can_remove_source_branch":true,"can_revert_on_current_merge_request":null,"can_cherry_pick_on_current_merge_request":false},"target_branch_commits_path":"/gitlab-com/www-gitlab-com/commits/master","new_blob_path":"/gitlab-com/www-gitlab-com/new/reorder-comparisons","conflict_resolution_path":null,"remove_wip_path":"/gitlab-com/www-gitlab-com/merge_requests/6629/remove_wip","cancel_merge_when_pipeline_succeeds_path":"/gitlab-com/www-gitlab-com/merge_requests/6629/cancel_merge_when_pipeline_succeeds","create_issue_to_resolve_discussions_path":"/gitlab-com/www-gitlab-com/issues/new?merge_request_to_resolve_discussions_of=6629","merge_path":"/gitlab-com/www-gitlab-com/merge_requests/6629/merge","cherry_pick_in_fork_path":null,"revert_in_fork_path":null,"email_patches_path":"/gitlab-com/www-gitlab-com/merge_requests/6629.patch","plain_diff_path":"/gitlab-com/www-gitlab-com/merge_requests/6629.diff","status_path":"/gitlab-com/www-gitlab-com/merge_requests/6629.json","ci_environments_status_path":"/gitlab-com/www-gitlab-com/merge_requests/6629/ci_environments_status","merge_commit_message_with_description":"Merge branch 'reorder-comparisons' into 'master'\n\nReorder comparisons\n\nResults in:\r\n\r\n![image](/uploads/6a8ea99b096112f644f5b65586a21584/image.png)\n\nSee merge request !6629","diverged_commits_count":3,"commit_change_content_path":"/gitlab-com/www-gitlab-com/merge_requests/6629/commit_change_content"}

When I looked at the commit status of this commit, it was green.

I came back hours lately, and it appears to have been fixed.

I see a number of issues:

1. Should we change the text to be something more like, "Pipeline for merge request has not been created yet"?
2. Why is there such a long delay? Sidekiq queues did not seem out of control.

/cc: @ayufan, @fatihacet

@stanhu We have a merge request that should improve that in 9.4, see https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12127. The problem is that we create a pipeline asynchronously, and there are numerous problems caused by this, like https://gitlab.com/gitlab-org/gitlab-ce/issues/33663#note_33750822.

I'm not sure why it took so long to update, this is something we should check.

This problem is still present and is probably caused by some kind of a race condition. We should look into it again. Let's add a milestone here.

changed milestone to %9.5

added regression label

@rspeicher You were facing this error recently much more frequently. Could you describe how your flow did look like and also on which MR you had hit the problem?

I start to wonder if this is not really a race condition, but actually a dying sidekiq job that fails before executing update process.

@rspeicher I would like to correlate that with replication lag, do you have time when it did happen?

1. We execute POST on Unicorn to create a new merge request (Create button in UI),
2. We execute https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/services/merge_requests/create_service.rb#L15,
3. Concurrently CreatePipelineService is being executed on master, but due to replication lag we SELECT empty object,
4. We save NULL, as between SELECT ci_pipelines and INSERT merge_requests there is delay,
5. This is not picked by CreatePipelineService, because first, we execute: INSERT ci_pipelines, and then: UPDATE merge_requests WITH ci_pipelines, and before UPDATE there's no merge request yet created.

Would then the solution be to fix 5.?

1. INSERT merge_requests,
2. UPDATE merge_requests WITH ci_pipelines as in CreatePipelineService,

@ayufan Thanks for looking into it.

You were facing this error recently much more frequently. Could you describe how your flow did look like and also on which MR you had hit the problem?

Workflow is always the same -- git push, click the "New MR" link that shell spits out, fill in description/apply labels, submit.

Recent examples:

• https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13163
• https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13024
• https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13052
• https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13236
• https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13293
• https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13300

There have been more but a subsequent push to the branch, such as after a review, fixes things usually.

Thanks. So, it is likely that my thinking in this comment is likely to happen: https://gitlab.com/gitlab-org/gitlab-ce/issues/34547#note_36337769 due to ordering of operations.

The problem is with the ordering of operations.

We do:

• INSERT ci_pipelines, UPDATE merge_request WITH ci_pipeline,
• SELECT ci_pipelines, INSERT merge_request.

Where we should do:

• INSERT ci_pipelines, UPDATE merge_request WITH ci_pipeline,
• INSERT merge_request, UPDATE merge_request WITH ci_pipeline,

added Deliverable label

assigned to @zj

assigned to @dosuken123

unassigned @zj

mentioned in merge request !13236 (merged)

This is the overoverview.

I'll create an MR according to the @ayufan comment.

mentioned in merge request !13252 (merged)

@fatihacet Hi. We're considering to change <span>Could not connect to the CI server. Please check your settings and try again.</span> which you committed at https://gitlab.com/gitlab-org/gitlab-ce/commit/0151325dacebb99d54b6effb1d5842c0c712168c.

Here is a @stanhu's concern.

Should we change the text to be something more like, "Pipeline for merge request has not been created yet"?

Could you explain why we ended up the phrase briefly? Also, please let us know what do you think if we change it to <span>Pipeline for merge request has not been created yet.</span>?

/cc @stanhu @bikebilly

FYI, "Could not connect to the CI server. Please check your settings and try again" might be true for external CI systems (e.g. Jenkins), but for GitLab CI it doesn't really make sense.

I think <span>Could not connect to the CI server. Please check your settings and try again.</span> is a relic from when CI was a separate service.

Pipeline for merge request has not been created yet. sounds good, but I would add a few words what is expected or not from the user. If I see this message I'll wonder if it's my fault or GitLab's. Should I do something (re-push) or just wait?

@bikebilly Do we have a related/duplicated issue about this word phrasing? If no, we should create.

closed via merge request !13252 (merged)

mentioned in commit edc81ea6

mentioned in commit 14936d0c

@bikebilly If the error message(Could not connect to the CI server. Please check your settings and try again) persists like https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13024 or https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13052 even if you tried to reload the page, then we need to look this issue again that https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/13252 was not enough to resolve 3rd party integration. If the error message will be gone in a minute, then this is expected.

@bikebilly I mean this issue is not related to the problem. This issue is more like "a problem by internal race conditions". I assume what you encountered was related to UI/UX problem while a GitLab instance is fetching Jenkins results.

I think we need to create a new issue like "Improve "Could not connect to the CI server. Please check your settings and try again" message" (as @stanhu suggested at https://gitlab.com/gitlab-org/gitlab-ce/issues/34547#note_36745338).