For zero-downtime deployed in a mixed code environment between 12.2 and
12.3, the branch and tag name cache is incorrectly invalidated - a push
to an old machine will not clear the redis set version of the cache on
the new machine.

This commit ensures that, in 12.3, both set and non-set versions of the
cache are invalidated, but does not write or consult the set version of
the cache. . In 12.4, it will be safe to switch branch and tag names to
the redis set cache both it and the legacy cache will be invalidated
appropriately in such a mixed code environment.

This delays the full implementation of the feature by one release, but
in the absence of a credible feature-flagging strategy, and amidst an
abundance of caution about the effects of too-eager cache expiration, I
believe this is the best approach available to us.

This reverts commit c6ccc07f.

It performs an expensive operation, but isn't required in
most of tests

This reverts commit 0eff75fa.

Stan Hu authored 5 years ago and Douwe Maan committed 5 years ago

Previously `ProjectCacheWorker` would be scheduled once per ref, which
would generate unnecessary I/O and load on Sidekiq, especially if many
tags or branches were pushed at once. `ProjectCacheWorker` would expire
three items:

1. Repository size: This only needs to be updated once per push.
2. Commit count: This only needs to be updated if the default branch
   is updated.
3. Project method caches: This only needs to be updated if the default
   branch changes, but only if certain files change (e.g. README,
   CHANGELOG, etc.).

Because the third item requires looking at the actual changes in the
commit deltas, we schedule one `ProjectCacheWorker` to handle the first
two cases, and schedule a separate `ProjectCacheWorker` for the third
case if it is needed. As a result, this brings down the number of
`ProjectCacheWorker` jobs from N to 2.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/52046

This allows us to check inclusion for the *_exists? methods without
downloading the full list of branch names, which is over 100KiB in size
for gitlab-ce at the moment.

Previously each tag in a push would invoke the Gitaly `FindAllTags` RPC
since the tag cache would be invalidated with every tag.

We can eliminate those extraneous calls by expiring the tag cache once
in `PostReceive` and taking advantage of the cached tags.

Relates to https://gitlab.com/gitlab-org/gitlab-ce/issues/65795

Whenever `PostReceive` is enqueued, `UpdateMergeRequestsWorker`
is enqueued and `MergeRequests::RefreshService` is called, it'll
check if the source branch of each MR asssociated to the push exists
or not via `MergeRequest#source_branch_exists?`. The said method will
call `Repository#branch_exists?` which is cached in `Rails.cache`.

When the cache contains outdated data and the source branch actually
exists, the `MergeRequests#RefreshService` job will close associated
MRs which is not correct.

The fix is to expire the branches cache of the project so we have
updated data during the post receive hook which will help in the
accuracy of the check if we need to close associated MRs or not.

We use the relative path of the archive to check for archive staleness.

Suggests to use a JSON structured log instead

Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/54102

Today, Pipelines for merge train run on `refs/merge`,
however, this causes a race condition that it can be
overwritten by CheckMergeabilityService.

This patch fixes the problem by generating `refs/train`
for those pipelines.

Currently, MergeToRefService is specifically designed for
createing merge commits from source branch and target branch of
merge reqeusts. We extend this behavior to source branch and any
target ref paths.

In 12.0, we turned the feature flag on that effectively turned off the
--max-count flag for the count diverging commits call. Since we have
commit graphs turned on, this did not affect preformance negatively.
Thus, we want to deprecate the call that passes --max-count

Extract diverging_commit_counts into a service class

Douwe Maan authored 5 years ago and Mayra Cabrera committed 5 years ago

This change makes sure Gitaly includes a fix to make rebase work again
properly.

Part of: https://gitlab.com/gitlab-org/gitlab-ce/issues/62353

Douwe Maan authored 5 years ago and Mayra Cabrera committed 5 years ago

Disabling because of https://gitlab.com/gitlab-org/gitlab-ce/issues/62353

If `merge_error` is ever set after a failure, it is never cleared, even
after a subsequent, successful rebase. We now clear this field whenever
a successful rebase occurs.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/56139

We want to optimize the query for the CountDivergingCommits rpc by
removing the --max-count argument now that we have commit graphs
enabled for all repositories during housekeeping. However, we want to
test this first behind a feature flag.

Fixes #60425

Luke Duncalfe authored 5 years ago and Douwe Maan committed 5 years ago

The new two-step Gitaly `Rebase` RPC yields the rebase commit SHA to the
client before proceeding with the rebase.

This avoids an issue where the rebase commit SHA was returned when the
RPC had fully completed, and in some cases this would be after the Rails
`post_receive` worker services had already run. In these situations,
the merge request did not yet have its rebase_commit_sha attribute set
introducing the possibility for bugs (such as previous approvals being
reset).

https://gitlab.com/gitlab-org/gitlab-ee/issues/5966

Updates the EnvironmentController#metrics_dashboard endpoint
to support a "dashboard" param, which can be used to specify
the filepath of a dashboard configuration from a project
repository. Dashboard configurations are expected to be
stored in .gitlab/dashboards/.

Updates dashboard post-processing steps to exclude custom
metrics, which should only display on the system dashboard.

This ports the changes from
https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10462/
to CE

This reverts commit 171818df.

This reverts commit 6c75bd01, reversing
changes made to 1be7f5aa.

Add `GetArchiveRequest` to git-archive params.

Modifies `Git::Repository#archive_metadata` to append `path`
to `ArchivePrefix` so it'll not hit the cache of repository archive
when it already exists.

This changes the repository type from a binary `wiki?` to a type. So
we can have more than 2 repository types.

Now everywhere we called `.wiki?` and expected a boolean, we check
that type.

This reverts merge request !26183

Signed-off-by: Rémy Coutable <remy@rymai.me>

When an empty project is loaded in the UI, there are 15 separate Gitaly
FindDefaultBranch calls to determine the root_ref. Previously, it was
not cached even within the request. This change caches it within the
request so only a single FindDefaultBranch RPC is needed.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/58684

This style change enforces `return if ...` instead of
`return nil if ...` to save maintainers a few minor review points

Adds the ground work for writing into
the merge ref refs/merge-requests/:iid/merge the
merge result between source and target branches of
a MR, without further side-effects such as
mailing, MR updates and target branch changes.

Adds the client call for the gitaly rpc CountDivergingCommits

fixing signature

simplifying commit logic

adding test for max-count

refactoring tests