Consider switching `Repository#fetch_remote` to use Rugged

One issue we've seen on GitLab.com is that RepositoryUpdateMirrorWorker blocks because it shells out to git to fetch a remote (gitlab-com/infrastructure#698 (closed)).

I believe it's possible that Sidekiq will get blocked waiting for the git command to complete. If we used Rugged here, I think our Sidekiq process will be able to take advantage of multiple threads. @yorickpeterse Is this a valid assessment?

/cc: @pcarranza

@stanhu I'm not entirely sure, this depends on how CRuby handles shelling out. Having said that, if Rugged provides a way to do fetching I don't see why we shouldn't use it.

Mentioned in issue #24187 (closed)

Related: https://gitlab.com/gitlab-org/gitlab-ce/issues/21657

Yep, this would solve the issue nicely:

[1] pry(main)> require 'rugged'
=> true
[2] pry(main)> repo = Rugged::Repository.new('.')
=> #<Rugged::Repository:70117128518480 {path: "/Users/tsigo/tmp/release-tools-test-gitlab/.git/"}>
[3] pry(main)> repo.remotes['github']
=> #<Rugged::Remote:0x007f8ad487c128 @owner=#<Rugged::Repository:70117128518480 {path: "/Users/tsigo/tmp/release-tools-test-gitlab/.git/"}>>
[4] pry(main)> repo.fetch('github')
Rugged::NetworkError: authentication required but no callback set
from /Users/tsigo/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/rugged-0.24.0/lib/rugged/repository.rb:210:in `fetch'

There are cases where we shell out because it's faster, and there are cases where we shell out because Rugged doesn't support something, but I think there are also cases like this where we shell out because we didn't know Rugged did something, or it didn't at the time, but it does now 😞

I'd like every place we shell out to git to have either a comment on why we're doing it, or a technical debt issue telling us to remove it.

https://gitlab.com/gitlab-org/gitlab-ce/issues/24374 will help A LOT with this, because the places we shell out will be in:

1. This repo.
2. EE.
3. gitlab-shell.

mentioned in issue #25779 (moved)

moved to gitlab#6936

Hey! 👋

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

• https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
• https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue