For zero-downtime deployed in a mixed code environment between 12.2 and
12.3, the branch and tag name cache is incorrectly invalidated - a push
to an old machine will not clear the redis set version of the cache on
the new machine.

This commit ensures that, in 12.3, both set and non-set versions of the
cache are invalidated, but does not write or consult the set version of
the cache. . In 12.4, it will be safe to switch branch and tag names to
the redis set cache both it and the legacy cache will be invalidated
appropriately in such a mixed code environment.

This delays the full implementation of the feature by one release, but
in the absence of a credible feature-flagging strategy, and amidst an
abundance of caution about the effects of too-eager cache expiration, I
believe this is the best approach available to us.

This reverts commit c6ccc07f.

It performs an expensive operation, but isn't required in
most of tests

This reverts commit 0eff75fa.

Stan Hu authored 5 years ago and Douwe Maan committed 5 years ago

Previously `ProjectCacheWorker` would be scheduled once per ref, which
would generate unnecessary I/O and load on Sidekiq, especially if many
tags or branches were pushed at once. `ProjectCacheWorker` would expire
three items:

1. Repository size: This only needs to be updated once per push.
2. Commit count: This only needs to be updated if the default branch
   is updated.
3. Project method caches: This only needs to be updated if the default
   branch changes, but only if certain files change (e.g. README,
   CHANGELOG, etc.).

Because the third item requires looking at the actual changes in the
commit deltas, we schedule one `ProjectCacheWorker` to handle the first
two cases, and schedule a separate `ProjectCacheWorker` for the third
case if it is needed. As a result, this brings down the number of
`ProjectCacheWorker` jobs from N to 2.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/52046

This allows us to check inclusion for the *_exists? methods without
downloading the full list of branch names, which is over 100KiB in size
for gitlab-ce at the moment.

Previously each tag in a push would invoke the Gitaly `FindAllTags` RPC
since the tag cache would be invalidated with every tag.

We can eliminate those extraneous calls by expiring the tag cache once
in `PostReceive` and taking advantage of the cached tags.

Relates to https://gitlab.com/gitlab-org/gitlab-ce/issues/65795

Whenever `PostReceive` is enqueued, `UpdateMergeRequestsWorker`
is enqueued and `MergeRequests::RefreshService` is called, it'll
check if the source branch of each MR asssociated to the push exists
or not via `MergeRequest#source_branch_exists?`. The said method will
call `Repository#branch_exists?` which is cached in `Rails.cache`.

When the cache contains outdated data and the source branch actually
exists, the `MergeRequests#RefreshService` job will close associated
MRs which is not correct.

The fix is to expire the branches cache of the project so we have
updated data during the post receive hook which will help in the
accuracy of the check if we need to close associated MRs or not.

We use the relative path of the archive to check for archive staleness.

Suggests to use a JSON structured log instead

Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/54102

Today, Pipelines for merge train run on `refs/merge`,
however, this causes a race condition that it can be
overwritten by CheckMergeabilityService.

This patch fixes the problem by generating `refs/train`
for those pipelines.

Currently, MergeToRefService is specifically designed for
createing merge commits from source branch and target branch of
merge reqeusts. We extend this behavior to source branch and any
target ref paths.

In 12.0, we turned the feature flag on that effectively turned off the
--max-count flag for the count diverging commits call. Since we have
commit graphs turned on, this did not affect preformance negatively.
Thus, we want to deprecate the call that passes --max-count

Extract diverging_commit_counts into a service class

Douwe Maan authored 5 years ago and Mayra Cabrera committed 5 years ago

This change makes sure Gitaly includes a fix to make rebase work again
properly.

Part of: https://gitlab.com/gitlab-org/gitlab-ce/issues/62353

Douwe Maan authored 5 years ago and Mayra Cabrera committed 5 years ago

Disabling because of https://gitlab.com/gitlab-org/gitlab-ce/issues/62353

If `merge_error` is ever set after a failure, it is never cleared, even
after a subsequent, successful rebase. We now clear this field whenever
a successful rebase occurs.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/56139

We want to optimize the query for the CountDivergingCommits rpc by
removing the --max-count argument now that we have commit graphs
enabled for all repositories during housekeeping. However, we want to
test this first behind a feature flag.