Fabio Pitino authored 2 years ago and Marius Bobin committed 2 years ago

Allow merge request pipeline creation to take in input
Git push options in order to support `ci.skip`.

Changelog: other
EE: true

RepositoryRemoveRemoteWorker's logic has been removed in 14.2 but
the worker itself needs to stay until 15.0 to drain the job queue.
It can be removed now.

Extract ServicePing payload computation and persistance
out of SubmitService into GitlabServicePingWorker
in order to pre compute SerivcePing data for preview
for instance admins even if those instances does not submit
this data later on to GitLab

Changelog: performance

The Guard worker will now log the repository id and the repository path
to ease debugging.

* Add `batched_background_migrations.gitlab_schema` column.
* When fetching the active migration scope to schemas available for the
current connection.

For DML migrations, which are restricted to given gitlab_schema, use this
value when enqueuing batched background migration, if there is not one
provided explicitly.

For v1.0 migrations, set `gitlab-schema` to `gitlab_main`.

Skip batched backfround migration worker if it shares database config
with another database.

https://gitlab.com/gitlab-org/gitlab/-/issues/359951

Changelog: changed

Updates the enqueuery next_repository query to
use LIMIT 2 to force the postgresql query planner
to use the index rather than a Seq Scan

Re-enqueuerer should support custom lease keys because
 some workers gies through few different scopes

Changelog: fixed
EE: true

Sean Arnold authored 2 years ago and Alexis Kalderimis committed 2 years ago

- Removes code to automatically create PromtheusAlerts.
This includes when the Prometheus integration is created.
- Also removes updates to the Prometheus Cluster. As adding and editing alerts is removed this is no longer needed.

This commit removes a worker used to remove artifacts for
a given job. Prefer using ExpireBuildArtifactsWorker which
is more performant.

when those are read from application settings. We need to convert values
to seconds first.

This sets the data_consistency for the Ci::ArchiveTraceWorker to
sticky. Allowing the read queries before any write to be handled by a
replica.

The job is scheduled from the Ci::BuildFinishedWorker with some delay
built in. Which means this job can take the 0.8s wait in the
beginning, and is very likely to find a replica that is up to date
with the WAL locations at the time of scheduling.

To increase the effect of this, this also preloads some associations
in the worker. That way, we can take advantage of loading them from a
replica rather than doing it lazily from the primary.

David Fernandez authored 2 years ago and Bob Van Landuyt :neckbeard: committed 2 years ago

In loop mode, the Enqueuer will loop on migration starts until either:
* it fills the capacity
* it hits the loop deadline
* the loop mode feature flag is disabled

* Encapsulate the redis logic for inactive projects deletion feature
inside InactiveProjectsDeletionWarningTracker.
* Make the workers idempotent.
* Update the feature_category to the correct owners.

Steve Abrams authored 2 years ago and Adam Hegyi committed 2 years ago

Add application settings to define timeouts for
long running container registry migration
pre-imports and imports.

Changelog: added

Administrators of large GitLab instances often find that over time
projects become inactive and are no longer used.  These projects take
up unnecessary disk space. This feature adds an automated way to
identify these projects, warn the maintainers ahead of time, and then
delete the projects if they are still inactive.

Changelog: added

When closing issues errors out, it can stop the execution of
`PostMergeService` which can result to other operations not being
executed.

To ensure that we still run the rest of logic in `PostMergeService`
when that happens, we close issues asynchronously.

This is behind the `async_mr_close_issue` feature flag as a new
worker is being introduced.

John Cai authored 2 years ago and Igor Drozdov committed 2 years ago

The repository size calculation is changing in Gitaly and will be more
expensive to compute. Recomputing the size on an incremental repack is
not necessary, so we can save some system resources by not computing it.

Changelog: changed

The deployment hooks worker can run against
a runner's job request API call which lacks user,
project or group identities in certain cases.

In such cases it is difficult to locate the exact
log that was responsible for firing a webhook or
integration event related to a deployment because
the volume of overall deployments per minute is
pretty high and no distinguishable fields are
present to filter it any further.

This change adds ID values to the logging
that allows us to track the deployment by ID
or its associated project by ID. The expectation
is that these IDs are always printed regardless
of the scope of the runners.

Changelog: added

Steve Abrams authored 2 years ago and Bob Van Landuyt :neckbeard: committed 2 years ago

Adds container_registry_migration_phase2_enqueue_twice ff
to allow the Enqueuer to be called twice each time a
container repository finishes the import path

This reverts MR
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86669.

David Fernandez authored 2 years ago and Bob Van Landuyt :neckbeard: committed 2 years ago

to take the repository id into account. This way, if two executions in
parallel select different repositories, both can continue their
execution.

This removes all `default_enabled: :yaml | true | false`

Update Projects::RecordTargetPlatformsWorker to handle detection of
projects built for the Android platform. It executes a new service
(Projects::AndroidTargetPlatformDetectorService) to determine if a given
project that uses Java or Kotlin is an Android project by checking if
the file AndroidManifest.xml is present in the project repository.

Add the flag file_extraction_pipeline! to the
ProjectAttributesPipeline in bulk import to keep re-enqueuing the
pipeline until the exported file is generated and download from the
source instance

Changelog: fixed

We have jobs to check the consistency between Namespace/Project
tables and their mirror tables.
With this MR, we will detect them and fix them automatically

Issue: https://gitlab.com/gitlab-org/gitlab/-/issues/359855

Markus Koller authored 2 years ago and Etienne Baqué committed 2 years ago

- Add a `#log_exception` helper to format exceptions.
- Also log the integration record ID.
- Log through the integration model in `Jira::Requests::Base` instead
  of including `Integrations::Loggable`.

Besides the formatting changes, there are two behaviour changes in
`ProjectServiceWorker`:

- The log message now goes to `integrations_json` instead of
  `exceptions_json`.
- We stop logging errors when the integration record can't be found,
  and instead just exit silently.

Changelog: changed

Changelog: other

Steve Abrams authored 2 years ago and David Fernandez committed 2 years ago

Allow a value of 0 for container_registry_import_max_tags_count
to be used as a "no limit" option.

Changelog: removed

Remove AuthorizedProjectRefresh::ProjectCreateWorker and its service

Remove project-group link creation services
& workers

Peter Leitzen authored 2 years ago and Adam Hegyi committed 2 years ago

Do not rely on implicit ID ordering when cleaning up package files but
order by ID explicitly.