Improve diff lines count efficiency

This will help diagnose the source of excessive I/O from Rugged
calls. To implement this, we need to obtain the full list of arguments
sent to each request method.

This adds `rugged_duration_ms` and `rugged_calls` fields to
`api_json.log` and `production_json.log`. This will make it easier to
identify performance issues caused by excessive I/O.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/64676

When passing start_branch on committing from the WebIDE, it's possible
that the branch has changed since editing started, which results in the
change being applied on top of the latest commit in the branch and
overwriting the new changes.

By passing the start_sha instead we can make sure that the change is
applied on top of the commit which the user started editing from.

Igor Drozdov authored 5 years ago and Nick Thomas committed 5 years ago

This is part of migration of Folder View to Vue

Whenever we use the rugged implementation, we are going straight to disk
so we want to bypass the disk access check.

Suggests to use a JSON structured log instead

Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/54102

Add a module we use as a singleton to determine whether or not rails is
able to access the disk

Currently, MergeToRefService is specifically designed for
createing merge commits from source branch and target branch of
merge reqeusts. We extend this behavior to source branch and any
target ref paths.

GitalyBot authored 5 years ago and Bob Van Landuyt :neckbeard: committed 5 years ago

The setup was wrong, and due to Gitaly being a bit more strict it had to
match up. The test is now accurate and passing again.

Luke Duncalfe authored 5 years ago and Mayra Cabrera committed 5 years ago

This backports to CE changes to allow the EE model
DesignManagement::Repository to override the #attributes_at method to
provide its own git attributes.

The #attributes_at method was freely available, as it's never called by
anything in the app. It looks like the code that called it got
refactored out of existence in ca66a04f. It was still being called in a
spec
https://gitlab.com/gitlab-org/gitlab-ce/blob/85b29c1c2fa3b94d7371cf454c485457a0756cb1/spec/services/files/create_service_spec.rb#L40
which I've left because with the change in Lfs::FileTransformer in fact
is now again the perfect test!

See EE MR
https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14017

And these comment threads
https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/13894#note_178002089
https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/13894#note_178049984

CE port of https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/13460

It used to be the case that GitLab created symlinks for each repository
to one copy of the Git hooks, so these ran when required. This changed
to set the hooks dynamically on Gitaly when invoking Git.

The side effect is that we didn't need all these symlinks anymore, which
Gitaly doesn't create anymore either. Now that means that the tests in
GitLab-Rails should test for it either.

Related: https://gitlab.com/gitlab-org/gitaly/issues/1392#note_175619926

Attempting to use the API endpoint
/projects/:id/repository/tree?recursive=true would only return a subset
of the results since the full recursive list wasn't actually being
returned.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/61979

We want to optimize the query for the CountDivergingCommits rpc by
removing the --max-count argument now that we have commit graphs
enabled for all repositories during housekeeping. However, we want to
test this first behind a feature flag.

DisconnectGitAlternates is a Gitaly RPC that will delete a repository's
objects/info/alternates file. This commit adds client support for that
RPC.

Luke Duncalfe authored 5 years ago and Douwe Maan committed 5 years ago

The new two-step Gitaly `Rebase` RPC yields the rebase commit SHA to the
client before proceeding with the rebase.

This avoids an issue where the rebase commit SHA was returned when the
RPC had fully completed, and in some cases this would be after the Rails
`post_receive` worker services had already run. In these situations,
the merge request did not yet have its rebase_commit_sha attribute set
introducing the possibility for bugs (such as previous approvals being
reset).

https://gitlab.com/gitlab-org/gitlab-ee/issues/5966

Gitaly's FetchIntoObjectPool RPC will idempotently fetch objects into an
object pool. If the pool doesn't exist, it will create an empty pool
before attempting the fetch. This change adds client code as well as
specs to cover this behavior.

This ports the changes from
https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10462/
to CE

Inside a wiki, when we show the sidebar or browse to the `pages`,
all page contents are retrieved from Gitaly and that is a waste
of resources, since no content from that pages are going to be
showed.

This MR introduces the method `ProjectWiki#list_pages`,
which uses new wiki_list_pages RPC call to retrieve
pages without content

Also in the `WikisController` we're using the method to show
pages in the sidebar and also on the `pages` page.

This should already be done in Gitlab::Git:Compare.

A temp reference is only needed to fetch a branch from another project,
as in the case for forked repositories. For branch comparisons within
the same project, we can just use the existing branch names to do the
comparison.

Relates to https://gitlab.com/gitlab-org/gitlab-ce/issues/38689#note_126107862

This brings back changes in
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20432.

For users using Gitaly on top of NFS, accessing the Git data directly
via Rugged may be faster than going through than Gitaly. This merge
request introduces the feature flag `rugged_list_commits_by_oid` to
activate the Rugged method.

For one customer, we saw that ListCommitsByOid was the second highest
used endpoint that may be causing increased load.

This reverts commit 171818df.

This reverts commit 6c75bd01, reversing
changes made to 1be7f5aa.

Igor Drozdov authored 5 years ago and Douwe Maan committed 5 years ago

- Add controls for sorting by title and date
- Execute Gitaly call which now accepts sorting params for wikis

Add `GetArchiveRequest` to git-archive params.

Modifies `Git::Repository#archive_metadata` to append `path`
to `ArchivePrefix` so it'll not hit the cache of repository archive
when it already exists.

Jacob Vosmaer (GitLab) authored 5 years ago and Grzegorz Bizon committed 5 years ago

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/59777.

In earlier iterations of our implementation of Git object deduplication
we thought we would be making extensive use of Git remotes in pool
repositories in the future, and that we should manage these remotes
carefully from the start. We now expect we only care about one remote,
namely the source project. The other remotes are there only for forensic
purposes.

Before this MR we tried to also remove pool remotes when member projects
got deleted, with the UnlinkRepositoryFromObjectPool RPC. This is
fragile when there are race conditions (see
https://gitlab.com/gitlab-org/gitaly/issues/1568#note_153955926). We
have spent some time making this RPC less fragile in
https://gitlab.com/gitlab-org/gitaly/merge_requests/1151 but looking at
this problem again, I think we should just stop calling it.

The Rugged implementation was recursively scanning the repository to
create `flat_path` because the post-process step was being called from
with a loop. For large repositories, this was significantly slowing
things down. Break the call to `rugged_populate_flat_path` out of this
loop to make this work efficiently.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/59759

Error messages from custom pre-receive hooks now appear in the GitLab
UI.

This is re-enabling a feature that had been disabled in merge request
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18646

The feature had been disabled due to security concerns that information
which was not intended to be public (like stack traces) would leak into
public view.

PreReceiveErrors (from pre-receive, post-receive and update custom
hooks) are now filtered for messages that have been prefixed in a
particular way.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/48132

Allow incomplete commit records to load their full data from gitaly.

Commits can be based on a Hash of data retrieved from PostgreSQL, and
this data can be intentionally incomplete in order to save space.

A new method #gitaly? has been added to Gitlab::Git::Commit, which
returns true if the underlying data source of the Commit is a
Gitaly::GitCommit.

CommitCollection now has a method #enrich which replaces non-gitaly
commits in place with commits from gitaly.

CommitCollection#without_merge_commits has been updated to call this
method, as in order to determine a merge commit we need to have parent
data.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/58805

Revert "Merge branch '58805-allow-incomplete-commit-data-to-be-fetched-from-collection' into 'master'"

This reverts merge request !26144

Allow incomplete commit records to load their full data from gitaly.

Commits can be based on a Hash of data retrieved from PostgreSQL, and
this data can be intentionally incomplete in order to save space.

A new method #gitaly? has been added to Gitlab::Git::Commit, which
returns true if the underlying data source of the Commit is a
Gitaly::GitCommit.

CommitCollection now has a method #enrich which replaces non-gitaly
commits in place with commits from gitaly.

CommitCollection#without_merge_commits has been updated to call this
method, as in order to determine a merge commit we need to have parent
data.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/58805

EE added the ability to set timeouts when handling blobs. Since there's
no particular reason for this to be EE specific we can just backport
this to CE.