During a project import, `LfsLinkService` attempts to link `LfsObjects`
that have not already been associated with a project.  It's possible for
a large repo to have thousands of OIDs, which can cause long database
query and parsing times.  By processing a batch of 1000 at a time, we
can reduce that time at the expense of a few more SQL queries.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/66274

- Adds UI to configure in group and project settings
- Removes notification configuration for users when
disabled at group or project level

Add Gitlab::VisibilityLevelChecker that verifies
selected project visibility level (or overridden param)
is not restricted when creating or importing a project

Bob Van Landuyt :neckbeard: authored 5 years ago and Douwe Maan committed 5 years ago

**Prevention of running 2 simultaneous updates**

Instead of using `RemoteMirror#update_status` and raise an error if
it's already running to prevent the same mirror being updated at the
same time we now use `Gitlab::ExclusiveLease` for that.

When we fail to obtain a lease in 3 tries, 30 seconds apart, we bail
and reschedule. We'll reschedule faster for the protected branches.

If the mirror already ran since it was scheduled, the job will be
skipped.

**Error handling: Remote side**

When an update fails because of a `Gitlab::Git::CommandError`, we
won't track this error in sentry, this could be on the remote side:
for example when branches have diverged.

In this case, we'll try 3 times scheduled 1 or 5 minutes apart.

In between, the mirror is marked as "to_retry", the error would be
visible to the user when they visit the settings page.

After 3 tries we'll mark the mirror as failed and notify the user.

We won't track this error in sentry, as it's not likely we can help
it.

The next event that would trigger a new refresh.

**Error handling: our side**

If an unexpected error occurs, we mark the mirror as failed, but we'd
still retry the job based on the regular sidekiq retries with
backoff. Same as we used to

The error would be reported in sentry, since its likely we need to do
something about it.

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21679 moved the
deletion of registry tags outside of a transaction, but introduced an
issue where Sidekiq would attempt to contact the container registry
during project destruction even if it were disabled.

Relates to:

* https://gitlab.com/charts/gitlab/issues/1455
* https://gitlab.com/gitlab-org/gitlab-ce/issues/45941

We should just ignore these errors and move along with the deletion
since the repositories are going to be trashed anyway.

Closes https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31164

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/65054

We dropped MySQL support and a lot of mysql specific code has been
removed in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29608.

This comes in from the other direction and removes any `if postgresql?`
branches.

Suggests to use a JSON structured log instead

Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/54102

When the SQL query cache is active, the SELECT query for finding
projects to apply service templates returns the same values. This causes
an infinite loop because even though bulk INSERT queries are made, the
cached results never reflect that progress.  To fix this, we call
`Project.uncached` around the query to ensure new data is retrieved.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/63595

Enable Get and Update of ci_default_git_depth for
Project API.

Renaming Project#default_git_depth to :ci_default_git_depth
to give more context through the API usage.

Add API documentation

and simplify ProjectCiCdSetting#set_default_git_depth

If the origin project has no default_git_depth set (i.e. nil) set the
fork's default_git_depth to 0

Krasimir Angelov authored 5 years ago and Fabio Pitino committed 5 years ago

Introduce default_git_depth in project's CI/CD settings and set it to
50. Use it if there is no GIT_DEPTH variable specified. Apply this
default only to newly created projects and keep it nil for old ones
in order to not break pipelines that rely on non-shallow clones.

default_git_depth can be updated from CI/CD Settings in the UI, must be
either nil or integer between 0 and 1000 (incl).

Inherit default_git_depth from the origin project when forking projects.

MR pipelines are run on a MR ref (refs/merge-requests/:iid/merge) and it
contains unique commit (i.e. merge commit) which doesn't exist in the
other branch/tags refs. We need to add it cause otherwise it may break
pipelines for old projects that have already enabled Pipelines for merge
results and have git depth 0.

Document new default_git_depth project CI/CD setting

Delete Todos for guest users when project visibility
level is updated to private.

GitDeduplicationService performs idempotent operations on deduplicated
projects.

Immediate configuration is not ideal for group and instance
level clusters as projects that may never be deployed would
still have Kubernetes namespaces and service accounts created
for them.

As of https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25586
we now create only the resources that are required for the
project being deployed, at the time of deployment.

Pull mirrors would run the `Projects::AfterImportService`, which would
force a `git gc` each time it finished. This is overkill and not
necessary now that we have refs packed more frequently
(https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27826).

Closes https://gitlab.com/gitlab-org/gitlab-ee/issues/11556

Project imports were failing with `undefined method each_with_object for
String` because the import was attempting to parse the LFS Batch API and
failing due to the fact that the Content-Type wasn't a supported format
(e.g. application/vnd.git-lfs+json instead of application/json). We now
parse the body as JSON.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/61624

Fixes #60425

PackRefs is not an expensive gitaly call - we want to
call it more often (than as part of full `gc`) because
it helps to keep number of refs files small - too many
refs file may be a problem for deployments with
slow storage.

Francisco Javier López authored 5 years ago and James Lopez committed 5 years ago

In order to make `LfsImportService` more reusable, we
need to extract the logic inside `ImportService` and
encapsulate it into the service.

Prepares us for upgrade to Rails 5.2

This new table will be used to store the external_dashboard_url which
allows users to add a link to their external dashboards (ex Grafana)
to the Metrics dashboard.

Jasper Maes authored 6 years ago and Heinrich Lee Yu committed 5 years ago

Model.new.attributes now also returns encrypted attributes.

- Externalize strings in milestones_helper
- Externalize strings in app/services
- Update PO file

The `#reload` makes to load all objects into memory,
and the main purpose of `#reload` is to drop the association cache.

The `#reset` seems to solve exactly that case.

It could happen that there's a cached (in Redis) ApplicationSetting
record, and calling
`Gitlab::CurrentSettings.current_application_settings` only returns it
instead of creating a new DB record, which makes the
`ApplicationSetting.current_without_cache.update!` call fail.

Signed-off-by: Rémy Coutable <remy@rymai.me>

Imre (Admin) authored 5 years ago and Andreas Brandl committed 5 years ago

Move Contribution Analytics related spec in spec/features/groups/group_page_with_external_authorization_service_spec to EE

This reverts merge request !26823

Imre (Admin) authored 5 years ago and Andreas Brandl committed 5 years ago

Move Contribution Analytics related spec in spec/features/groups/group_page_with_external_authorization_service_spec to EE

During a project import, it's possible that new branches are created by
the importer to handle pull requests that have been created from forked
projects, which would increment the `pushes_since_gc` value via
`HousekeepingService.increment!` before a full garbage collection gets
to run. This causes HousekeepingService to skip the full `git gc` and
move to the incremental repack mode. To ensure that a garbage collection
is run to pack refs and objects, explicitly execute the task.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/59477