To finish migrating untracked files to uploads for installations that were affected by https://gitlab.com/gitlab-org/gitlab-ce/issues/42881

Or just to delete the temp table if it is empty and left behind.

Fix a transient failure in db/post_migrate/20170717111152_cleanup_move_system_upload_folder_symlink.rb where symlink already exists

Signed-off-by: Rémy Coutable <remy@rymai.me>

Artifacts are in the middle of being migrated from ci_builds to
ci_job_artifacts. The expiration date is currently visible in both of
these tables and the test for whether an expired artifact is present
for a job is complex as it requires checking both the of the tables.

Add two new indexes, one on ci_builds.artifacts_expire_at and one on
ci_job_artifacts.expire_at to enable finding expired artifacts
efficiently.

And until the migration is finished, replace the SQL for finding
expired and non-expired artifacts with a hand-crafted UNION ALL based
query instead of using OR. This overcomes a database optimizer
limitation that prevents it from using these indexes.

When the migration is finished the next version should remove this
query and replace it with a much simpler query on just
ci_job_artifacts. See
https://gitlab.com/gitlab-org/gitlab-ce/issues/42561 for followup.

Since populating the fork networks was scheduled multiple times
because of bugs that needed to be fixed:

- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15595/
  Creating fork networks for projects that were missing the root of
  the fork network.

- https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15709
  The API allowed creating forked_project_links without
  fork_network_members.

Scheduling this migration multiple times would case it to run
concurrently. Which in turn would try to insert the same data into
`fork_network_members` causing duplicate key errors.

This avoids running the migration multiple times.

This makes it possible to heavily optimize this migration, because we
need an outdated index to remove redundant stages faster.

Fixes #32282.

Currently we still need to run EnsureStageService within a transaction,
because when it runs within in a transaction we are going to stick to
the primary database when using database load balancing. Extracting this
out of the transaction makes it possible to hit into problems with
replication lag in pipeline commit status API, which can cause a lot of
trouble.

EE seems to have had an outdated schema at some point, leading to some
environments not having the right columns in place. This adjusts the
migration for `issues.closed_at` so it takes care of those cases,
ensuring data can be migrated properly.

Fixes https://gitlab.com/gitlab-org/gitlab-ee/issues/4803

In the event of Sidekiq jobs getting lost there may be some rows left to
migrate. This migration ensures any remaining jobs are completed and
that all data has been migrated.

This fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/41595