- Jan 24, 2018
-
-
Ahmad Sherif authored
Closes gitaly#929
-
- Jan 18, 2018
-
-
- Jan 17, 2018
-
-
Jacob Vosmaer (GitLab) authored
-
- Jan 11, 2018
-
-
Ahmad Sherif authored
Closes gitaly#851
-
- Jan 05, 2018
-
-
Alejandro Rodríguez authored
-
Lin Jen-Shin (godfat) authored
-
- Jan 04, 2018
-
-
Alejandro Rodríguez authored
-
Alejandro Rodríguez authored
-
- Jan 03, 2018
-
-
Jacob Vosmaer (GitLab) authored
-
Jacob Vosmaer (GitLab) authored
-
- Dec 27, 2017
-
-
Ahmad Sherif authored
-
- Dec 22, 2017
-
-
Alejandro Rodríguez authored
-
Alejandro Rodríguez authored
-
blackst0ne authored
-
- Dec 14, 2017
-
-
Nick Thomas authored
By importing this Ruby code into gitlab-rails (and gitaly-ruby), we avoid 200ms of startup time for each gitlab_projects subprocess we are eliminating. By not having a gitlab_projects subprocess between gitlab-rails / sidekiq and any git subprocesses (e.g. for fork_project, fetch_remote, etc, calls), we can also manage these git processes more cleanly, and avoid sending SIGKILL to them
-
- Dec 07, 2017
-
-
Zeger-Jan van de Weg authored
-
- Dec 06, 2017
-
-
Alejandro Rodríguez authored
This also allows us to simplify the naming since we can make some fetching methods private.
-
- Dec 01, 2017
-
-
Stan Hu authored
This was failing regularly with an Error 500 when the API branches endpoint was used. Closes #40615
-
- Nov 23, 2017
-
-
Douwe Maan authored
-
Lin Jen-Shin authored
-
Lin Jen-Shin authored
-
- Nov 20, 2017
-
-
Alejandro Rodríguez authored
-
- Nov 17, 2017
-
-
Alejandro Rodríguez authored
-
- Nov 10, 2017
-
-
Jacob Vosmaer (GitLab) authored
-
- Nov 07, 2017
-
-
Yorick Peterse authored
Prior to this MR there were two GitHub related importers: * Github::Import: the main importer used for GitHub projects * Gitlab::GithubImport: importer that's somewhat confusingly used for importing Gitea projects (apparently they have a compatible API) This MR renames the Gitea importer to Gitlab::LegacyGithubImport and introduces a new GitHub importer in the Gitlab::GithubImport namespace. This new GitHub importer uses Sidekiq for importing multiple resources in parallel, though it also has the ability to import data sequentially should this be necessary. The new code is spread across the following directories: * lib/gitlab/github_import: this directory contains most of the importer code such as the classes used for importing resources. * app/workers/gitlab/github_import: this directory contains the Sidekiq workers, most of which simply use the code from the directory above. * app/workers/concerns/gitlab/github_import: this directory provides a few modules that are included in every GitHub importer worker. == Stages The import work is divided into separate stages, with each stage importing a specific set of data. Stages will schedule the work that needs to be performed, followed by scheduling a job for the "AdvanceStageWorker" worker. This worker will periodically check if all work is completed and schedule the next stage if this is the case. If work is not yet completed this worker will reschedule itself. Using this approach we don't have to block threads by calling `sleep()`, as doing so for large projects could block the thread from doing any work for many hours. == Retrying Work Workers will reschedule themselves whenever necessary. For example, hitting the GitHub API's rate limit will result in jobs rescheduling themselves. These jobs are not processed until the rate limit has been reset. == User Lookups Part of the importing process involves looking up user details in the GitHub API so we can map them to GitLab users. The old importer used an in-memory cache, but this obviously doesn't work when the work is spread across different threads. The new importer uses a Redis cache and makes sure we only perform API/database calls if absolutely necessary. Frequently used keys are refreshed, and lookup misses are also cached; removing the need for performing API/database calls if we know we don't have the data we're looking for. == Performance & Models The new importer in various places uses raw INSERT statements (as generated by `Gitlab::Database.bulk_insert`) instead of using Rails models. This allows us to bypass any validations and callbacks, drastically reducing the number of SQL queries and Gitaly RPC calls necessary to import projects. To ensure the code produces valid data the corresponding tests check if the produced rows are valid according to the model validation rules.
-
- Nov 03, 2017
-
-
Alejandro Rodríguez authored
-
Alejandro Rodríguez authored
-
micael.bergeron authored
also, I refactored the MergeRequest#fetch_ref method to express the side-effect that this method has. MergeRequest#fetch_ref -> MergeRequest#fetch_ref! Repository#fetch_source_branch -> Repository#fetch_source_branch!
-
- Nov 02, 2017
-
-
- Nov 01, 2017
-
-
James Edwards-Jones authored
Gitlab::Git::Blob.batch_lfs_metadata can be used to check for LFS pointers. It uses a lazy enumorator and filters by blob size
-
- Oct 31, 2017
-
-
Alejandro Rodríguez authored
-
- Oct 27, 2017
-
-
Lin Jen-Shin (godfat) authored
-
- Oct 24, 2017
-
-
Alejandro Rodríguez authored
We also delete some unused code related to the aforementioned feature.
-
- Oct 13, 2017
-
-
Jacob Vosmaer (GitLab) authored
-
- Oct 12, 2017
-
-
Ahmad Sherif authored
Fixes #39046
-
- Oct 11, 2017
-
-
Ahmad Sherif authored
Fixes gitaly#629
-
- Oct 09, 2017
-
-
Alejandro Rodríguez authored
-
- Oct 07, 2017
-
-
Jacopo authored
Replaces all the explicit include metadata syntax in the specs (tag: true) into the implicit one (:tag). Added a cop to prevent future errors and handle autocorrection.
-
- Oct 04, 2017
-
-
Jacob Vosmaer (GitLab) authored
-
Alejandro Rodríguez authored
-