Artifacts are in the middle of being migrated from ci_builds to
ci_job_artifacts. The expiration date is currently visible in both of
these tables and the test for whether an expired artifact is present
for a job is complex as it requires checking both the of the tables.

Add two new indexes, one on ci_builds.artifacts_expire_at and one on
ci_job_artifacts.expire_at to enable finding expired artifacts
efficiently.

And until the migration is finished, replace the SQL for finding
expired and non-expired artifacts with a hand-crafted UNION ALL based
query instead of using OR. This overcomes a database optimizer
limitation that prevents it from using these indexes.

When the migration is finished the next version should remove this
query and replace it with a much simpler query on just
ci_job_artifacts. See
https://gitlab.com/gitlab-org/gitlab-ce/issues/42561 for followup.

- Move the exception handling as close to the source as possible to
avoid leaking Psych ahstraction
- Also remove unnecessary rescue all statement from LintsController.
This should not be necessary anymore since any YAML errors should all be
caught by the #validation_message method.

This is a small refactor to avoid querying Redis when we know there's
nothing in it.

This should reduce the number of SQL queries and lookups needed to look up a
project of a build and pipeline and vice versa.

Before:

```
[1] pry(main)> Ci::Build.reflect_on_association(:project).has_inverse?
=> false
[2] pry(main)> Project.reflect_on_association(:builds).has_inverse?
=> false
[3] pry(main)> Ci::Pipeline.reflect_on_association(:project).has_inverse?
=> false
[4] pry(main)> Project.reflect_on_association(:pipelines).has_inverse?
=> :project
```

After:

```
[1] pry(main)> Ci::Build.reflect_on_association(:project).has_inverse?
=> :builds
[2] pry(main)> Project.reflect_on_association(:builds).has_inverse?
=> :project
[3] pry(main)> Ci::Pipeline.reflect_on_association(:project).has_inverse?
=> :pipelines
[4] pry(main)> Project.reflect_on_association(:pipelines).has_inverse?
=> :project
```

This removes all usage of soft removals except for the "pending delete"
system implemented for projects. This in turn simplifies all the query
plans of the models that used soft removals. Since we don't really use
soft removals for anything useful there's no point in keeping it around.

This _does_ mean that hard removals of issues (which only admins can do
if I'm not mistaken) can influence the "iid" values, but that code is
broken to begin with. More on this (and how to fix it) can be found in
https://gitlab.com/gitlab-org/gitlab-ce/issues/31114.

Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/37447

Uses `list_commits_by_oid` on the CommitService, to request the needed
commits for pipelines. These commits are needed to display the user that
created the commit and the commit title.

This includes fixes for tests failing that depended on the commit
being `nil`. However, now these are batch loaded, this doesn't happen
anymore and the commits are an instance of BatchLoader.

Use Class.new(StandardError) instead of custom extended error class. Bring back specified_dependencies?.