In case when `needs:` is missing, but when requested by service,
we would not save the pipeline with config_error.

This makes it explicit that we want to persist the error
as `config_error` failure reason.

This makes BuildQueueService to force refresh runners
that are considered to have recent queue.

Such runners are the ones that connected within online
interval + time to expire runner cache.

This changes the `needs:` logic to require
that all jobs to be present. Instead of skipping
do fail the pipeline creation if `needs:` dependency
is not found.

Currently, some of the jobs with `needs:`
would be processed in stages, it means
that `when:` for such jobs would not be
respected.

This changes the behavior to have a separate
execution paths for jobs with `needs:`.

This changes `needs:` from weak reference
to have a strong reference.

This means that job will not be created
unless all needs are present as part of
a pipeline.

This implements the support for `needs:` keyword
as part of GitLab CI. That makes some of the jobs
to be run out of order.

Adds Job specific variables to facilitate specifying variables when
running manual jobs.

Often live traces are removed even though the archived trace
doesn't exist. This commit checkes the existence strictly.

This commit reverts the previously introduced concurrent pipeline
schedule creation which was a viable solution for mitigating
inconsistent pipeline schedule by Sidekiq Memory Killer.

This backports all EE schema changes to CE, including EE migrations,
ensuring both use the same schema.

== Updated tests

A spec related to ghost and support bot users had to be modified to make
it pass. The spec in question assumes that the "support_bot" column
exists when defining the spec. In the single codebase setup this is not
the case, as the column is backported in a later migration. Any attempt
to use a different schema version or use of "around" blocks to
conditionally disable specs won't help, as reverting the backport
migration would also drop the "support_bot" column. Removing the
"support_bot" tests entirely appears to be the only solution.

We also need to update some foreign key tests now that we have
backported the EE columns. Fortunately, these changes are very minor.

== Backporting migrations

This commit moves EE specific migrations (except those for the Geo
tracking database) and related files to CE, and also removes any traces
of the ee/db directory.

Some migrations had to be modified or removed, as they no longer work
with the schema being backported. These migrations were all quite old,
so we opted for removing them where modifying them would take too much
time and effort.

Some old migrations were modified in EE, while also existing in CE. In
these cases we took the EE code, and in one case removed them entirely.
It's not worth spending time trying to merge these changes somehow as we
plan to remove old migrations around the release of 12.0, see
https://gitlab.com/gitlab-org/gitlab-ce/issues/59177 for more details.

Fixing the bug

updated several specs and factories to accomodate new permissions

Currently, pipeline schedule worker is unstable because it's
sometimes killed by excessive memory consumption.
In order to improve the performance, we add the following fixes:

1. next_run_at is always real_next_run, which means the value
   always takes into account of worker's cron schedule
1. Remove exlusive lock. This is already covered by real_next_run
   change.
1. Use RunPipelineScheduleWorker for avoiding memory killer.
   Memory consumption is spread to the multiple sidekiq worker.

We've already migrated all the legacy artifacts to the new realm,
which is ci_job_artifacts table.
It's time to remove the old code base that is no longer used.

Currently, schedule head pipeline update method which executed after
pipeline creation does not take into account of merge reqeust
pipelines. We should use dedicated `all_merge_requests` method
in this case.

Continuation of 426488b7

- Adds an endpoint on PipelinesController
- Adds a service that iterates over every build in a stage and
plays it.
- Includes 'play_manual' details on EntitySerializer
- Builds a new Stage state: PlayManual. An stage can take this status if
it has manual builds or an skipped, scheduled or manual status
- Includes FE modifications and specs

Due to the nature of pipelines for merge requests, deployments.ref can
be a merge request ref instead of a branch name.

We support the environment auto-stop hook for this case

This backports the changes from
https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/10452

If no pipeline is created we currently have IID gap.
This is due to fact that we generate IID not on save,
but rather ahead of time. This results, us,
losing IIDs.

When deleting a pipeline, only some of the cache structures were being
expired, but not the full pipeline list. We have to synchronously
schedule a pipeline cache expiration because the pipeline will be
deleted if the Sidekiq expiration job picks it up. To do this, properly
extract all the logic buried in the Sidekiq worker into a service, and
then call the service.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/60469

https://gitlab.com/gitlab-org/gitlab-ce/issues/60480

Probably useful as we often move these files to "new" files.

Previously the raw push option Array was sent to Pipeline::Chain::Skip.

This commit updates this class (and the chain of classes that pass the
push option parameters from the API internal `post_receive` endpoint to
that class) to treat push options as a Hash of options parsed by
GitLab::PushOptions.

The GitLab::PushOptions class takes options like this:

    -o ci.skip -o merge_request.create -o merge_request.target=branch

and turns them into a Hash like this:

    {
      ci: {
        skip: true
      },
      merge_request: {
        create: true,
        target: 'branch'
      }
    }

This now how Pipeline::Chain::Skip is determining if the `ci.skip` push
option was used.

In order to implement https://gitlab.com/gitlab-org/gitlab-ee/issues/10179
we need several modifications on the CI config file. We are
adding a new ports section in the default Image object.

Each of these ports will accept: number, protocol and name.

By default this new configuration will be only enabled in
the Web IDE config file.

`project.pipeline_status.has_status?` is cached, which can lead to Error
500s in the UI if the this condition is used to check whether a pipeline
exists for a commit. We now expire the cache to ensure that the
information is consistent.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/59453

Introduces the concept of Prerequisites for a CI build.
If a build has unmet prerequisites it will go through the
:preparing state before being made available to a runner.

There are no actual prerequisites yet, so current
behaviour is unchanged.

Fix

ok

Add spec

Fix

ok

Fix

Add changelog

Fix

Add memoization

a

fix

source_sha and target_sha are used for merge request pipelines

Rename

Introduce Destroy expired job artifacts service

Revert a bit

Add changelog

Use expired

Improve

Fix spec

Fix spec

Use bang for destroy

Introduce iteration limit

Update comment

Simplify more

Refacor

Remove unnecessary thing

Fix comments

Fix coding offence

Make loop helper exception free

AuditEventService isn't equipped to handle logging of the destruction of
entities such as CI pipelines. It's a project-level event that operates
on a pipeline. The current log doesn't even indicate that the pipeline
is being destroyed.

This is a CE backport of
https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9105.  We're
removing the auditing call because it breaks the EE implementation.

These are data columns that store runtime configuration
of build needed to execute it on runner and within pipeline.

The definition of this data is that once used, and when no longer
needed (due to retry capability) they can be freely removed.

They use `jsonb` on PostgreSQL, and `text` on MySQL (due to lacking
support for json datatype on old enough version).

Sort out some logic

gitlab-org/gitlab-shell!166 added support for collecting push options
from the environment, and passing them along to the
/internal/post_receive API endpoint.

This change handles the new push_options JSON element in the payload,
and passes them on through to the GitPushService and GitTagPushService
services.

Futhermore, it adds support for the first push option, ci.skip.  With
this change, one can use 'git push -o ci.skip' to skip CI pipe
execution. Note that the pipeline is still created, but in the "skipped"
state, just like with the 'ci skip' commit message text.

Implements #18667