Optimize GitHub importer for speed and memory
What does this MR do?
It applies some improvements to GitHub importer. Importing of a private GH repo (~1500 issues, ~180 PRs, ~3600 commits), was taking over 30 minutes and memory usage reached 950 MB, after these changes it took 8 minutes and memory was at 700 MB max (for both cases, Sidekiq starts at ~400 MB).
Are there points in the code the reviewer needs to double check?
N/A
Why was this MR needed?
People are asking for it, see issues below.
Screenshots (if relevant)
N/A
Does this MR meet the acceptance criteria?
-
CHANGELOG entry added -
Documentation created/updated -
API support added - Tests
-
Added for this feature/bug -
All builds are passing
-
-
Conform by the merge request performance guides -
Conform by the style guides -
Branch has no merge conflicts with master
(if you do - rebase it please) -
Squashed related commits together
What are the relevant issue numbers?
https://gitlab.com/gitlab-com/support-forum/issues/996 and https://gitlab.com/gitlab-org/gitlab-ce/issues/19124#note_13588843