GitHub import process is extremely slow

Overview

GitHub importing is currently extremely slow for large repositories. v8.16.4

Reproduce

1. Import https://github.com/gitlabhq/gitlabhq
   1. 45,568 commits
   2. x pull requests
2. Running the import takes well over 30 minutes (this does vary based on connection speed and processing power)

There's no progress shown to the user during this time.

Debugging

I've monkey patched the client.rb to print the last api response including the status code. It's clear we're not hitting the GitHub rate limit. The time between sending requests is slow. It seems the bottleneck may be when we create the merge requests/issues on the GitLab side.

After cycle analytics was introduced we see a large number of updates to the merge_request_metrics table when importing.

 SQL (0.4ms)  UPDATE "merge_request_metrics" SET "merged_at" = $1, "updated_at" = $2 
WHERE "merge_request_metrics"."id" = $3  [["merged_at", "2017-02-14 17:28:58.179914"], 
["updated_at", "2017-02-14 17:28:58.180620"], ["id", 564]]

https://gitlab.zendesk.com/agent/tickets/61132

@stanhu This is a blocker right now for a customer+ moving to GitLab from GitHub. The project they're importing has 14,000 commits, 20 branches, 400 releases, and 3000 pull requests. They've run the import job well over 12 hours and it still going.

Is this more of a CI/CD issue because we update the merge request metrics with a pipeline? Are we seeing lots of blocking queries?

If we comment out this update, does the import speed along?

/cc: @jameslopez

Related issue that Ahmad looked into: https://gitlab.com/gitlab-org/gitlab-ce/issues/27429

@ahmadsherif Can you take a look at this one?

For reference I started an import on GitLab.com of GitLabHQ. It's now been running for 11 hours and hasn't completed. I can't see the RepostioryImportWorker running in Sidekiq so it seems the job has stopped yet the import is still in a "processing" state.

@MrChrisW I'm pretty sure that got killed by the infamous MemoryKiller :D In that case, the import remains in the started state forever. This could be tricky since we want the memory to be very low to keep the import going after the 15mins grace period, and it's easy to exceed 1GB if we have other threads also doing some memory consuming stuff :/

So this would be an exercise of two things 1. Keep the memory low as possible. 2. Make it faster.

@stanhu Importing PRs is slow because fetching source ref for each PR from GH is slow, or at least adds up in time, not sure if recent code added another bottleneck or not. Maybe we should investigate running this stage (PR importing) in parallel jobs.

@MrChrisW's case is a bit weird because nothing seems to be imported except for the repo itself; the failed cases we've seen at least got some stuff imported before failing. Probably it got hit early by the memory killer like @jameslopez suggested.

I'm unable to reproduce the "MemoryKiller" theory although I did get half way there (469948kb). Funnily enough during this import test the worker finished successfully, however only the project was imported (no issues or pull requests) - API Log - https://drive.google.com/a/gitlab.com/file/d/0B_4wYK1qcPT1d3JCaFpwV19IUzg/

We're clearly no where near the rate limit yet it's so slow and MR/issues failed but the import is marked as a success. Also no MemKiller

---

The first person who can successfully import https://github.com/gitlabhq/gitlabhq to GitLab.com (all issues/mrs) gets a

---

2017-02-16T02:47:28.838Z 54012 TID-ow5zvwg50 
RepositoryImportWorker JID-b45ca42df4864cb9f860ccb7 INFO: start
2017-02-16T02:47:28.838Z 54012 TID-ow5zvwg50 
RepositoryImportWorker JID-b45ca42df4864cb9f860ccb7 INFO: arguments: [13]
2017-02-16T02:59:01.283Z 54012 TID-ow5zvwg50 
RepositoryImportWorker JID-b45ca42df4864cb9f860ccb7 INFO: done: 692.446 sec

ps -o rss= -p 54012 - MAX = 1000000

Wed 15 Feb 2017 20:54:48 CST|469948
Wed 15 Feb 2017 21:03:42 CST|507068

I, [2017-02-15T20:46:05.651543 #54042]  INFO -- : Status: 200
I, [2017-02-15T20:46:06.122365 #54042]  INFO -- : Rate Limit: 4991
I, [2017-02-15T20:46:06.122619 #54042]  INFO -- : Time: 2017-02-15 20:46:06 -0600

I, [2017-02-15T21:02:56.067253 #54041]  INFO -- : Status: 200
I, [2017-02-15T21:02:56.527811 #54041]  INFO -- : Rate Limit: 4972
I, [2017-02-15T21:02:56.527992 #54041]  INFO -- : Time: 2017-02-15 21:02:56 -0600

irb(main):002:0> Project.find_by_full_path('MrChrisW/gitlabhq')
=> #<Project id: 2703519, name: "gitlabhq", path: "gitlabhq", description: "GitLab CE | Please

I'm also unable to identify the reason the import failed on GitLab.com. Searching the project ID 2703519 in Kibana shows nothing https://log.gitlap.com/goto/bca8037ccb699da264db7090b4234d37 - The RepositoryImportWorker is logged successfully with arguments (which should use the above project ID) https://log.gitlap.com/goto/1e3b973df0fdbdaaf04abf17e5da5dab

I don't think we have a MemoryKiller + RepositoryImportWorker issue here. See the last 30 days on GitLab.com https://log.gitlap.com/goto/9e22384ed1dfaf71979633b0401481d0

None of the kills line up with my import of project 2703519

added ~1071780 label

@ahmadsherif Can you do a benchmark the import https://github.com/gitlabhq/gitlabhq and see where the time is going? That would help inform whether your theory of pulling in individual PRs is causing the problem.

@stanhu Sure. Would you please add me (same handle as here) to the organization/repo so I can get started?

@rspeicher Can you add @ahmadsherif to the team in GitHub? @ahmadsherif What is your handle there?

@stanhu @rspeicher Same handle as here "ahmadsherif".

@stanhu @ahmadsherif Done.

@stanhu Here is a breakdown of an import of our repo on GH:

Resources	Time
Labels	1.8 seconds
Milestones	2.75 seconds
Pull Requests	11.7 hours
Issues	24 minutes
Issues Comments	2 hours
Pull Requests Comments	20 minutes
Wiki	2 seconds
Releases	0.12 second

Restoring source branches took 6.2 hours out of the 11.7, each restore operation is plotted below:

Wow! Is this slow because we have to do a git fetch for each pull request that references a deleted commit? How do we make this faster? Could we, for example, do a bulk git fetch? http://stackoverflow.com/a/25098004/1992201

Is this slow because we have to do a git fetch for each pull request that references a deleted commit?

That's right.

Could we, for example, do a bulk git fetch? http://stackoverflow.com/a/25098004/1992201

I am not sure if that would fetch them in parallel; otherwise I don't think we would gain much.

@ahmadsherif I think we can fetch all the branches easily in batches. Shall we just go ahead and do that?

$ git fetch origin pull/10391/head:pr-10391 pull/10385/head:pr-10385
remote: Counting objects: 6, done.
remote: Total 6 (delta 3), reused 3 (delta 3), pack-reused 3
Unpacking objects: 100% (6/6), done.
From https://github.com/gitlabhq/gitlabhq
 * [new ref]               refs/pull/10391/head -> pr-10391
 * [new ref]               refs/pull/10385/head -> pr-10385

@stanhu Alright, I'll try to whip some code together tomorrow and run another import with it, see how much of a difference does it make.

Fetching refs in bulk helped a little; it brought down the time of importing PRs to 8.8 hours. I think there's still room for improvements so I'll continue working on it.

@DouweM Support team would like to prioritize this for 9.1. If you agree, please assign the milestone and deliverable label.

changed milestone to %9.1

added ~480950 label

added Deliverable label

@ahmadsherif Are you working on this? I'm happy to put someone on it for %9.1 to help out/finish anything you've started, which would be next week.

@DouweM I am, but not full-time because of my commitment to Gitaly. Next Monday I shall push my progress for anyone who would like to use/finish it.

@DouweM Here is the branch I was working on https://gitlab.com/gitlab-org/gitlab-ce/tree/fix/speed-gh-pr-import.

assigned to @jameslopez

We saw a customer today who had a slow import for one 24000 commit repo, and then that repo was dying on new commits with ProcessCommitWorker throwing a Stack Level Too Deep Error:

https://gitlab.zendesk.com/agent/tickets/67741

I think that error was related to the fact they re-imported the repo multiple times and the DB was in some weird state. Logging here for posterity.

we can fetch all refs in a single run, we just need to change git config to do that. It's similar to do a git clone --mirror and how we replicate data in Geo. See:

• https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/models/repository.rb#L1016-1020
• https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/models/concerns/repository_mirroring.rb#L27-34

cc @ahmadsherif @dbalexandre

added Strategic Partnerships label

We have a high priority strategic partner who is considering migrating from GitHub and wants to do a POC of the migration and of GitLab. Currently, @dbalexandre has been assisting them as they failed on the import, as did @dbalexandre. I would really like to see this getting done as soon as possible. @jameslopez cc/ @DouweM

@eliran.mesika this is scheduled for 9.1, and it's a priority. In case there's a lot of work to do here, the idea is to split it and ship something on 9.1 anyway. I am not full-time on this since I was release manager on 8.17 and now on 9.0.

Thanks @jameslopez. @DouweM can we get someone to be full-time with this?

For this partner specifically, we will discuss in our private Slack: https://gitlab.slack.com/archives/C4N66DV1S/p1490201952024085

Importer speed improvements will be in CE as normal

mentioned in issue #17709 (closed)

mentioned in merge request !10273 (merged)

mentioned in merge request gitlab-shell!125 (merged)

mentioned in merge request !10308 (merged)

closed via merge request !10273 (merged)

We seeing same issue when trying to import repo by URL from bitbucket. We are running Hosted Gitlab EE. It would be great if we can resolve it. We are blocked :( https://support.gitlab.com/hc/en-us/requests/71686