All Sidekiq workers must use their own queue

Currently we have way too many workers using the "default" queue. Not only does this make monitoring harder, it also leads to the queue getting clogged by slow jobs. The easiest and most robust solution is to simply have every worker use its own queue. The "default" queue in turn must not be used, and we should add some sort of Rubocop rule to enforce this. The queue names should be similar to the worker class names, though they can be something different if it makes sense; as long as the name is descriptive enough.

Keep in mind that adding a new queue will require Omnibus changes to make Sidekiq read from these queues: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/templates/default/sv-sidekiq-run.erb#L7-24

Perhaps we need a better way to get this information into Omnibus?

/cc: @marin

Mentioned in merge request !6802 (merged)

The following workers are using the "default" queue:

AdminEmailWorker
BuildCoverageWorker
BuildEmailWorker
BuildFinishedWorker
BuildHooksWorker
BuildSuccessWorker
ClearDatabaseCacheWorker
DeleteUserWorker
ExpireBuildArtifactsWorker
ExpireBuildInstanceArtifactsWorker
GroupDestroyWorker
ImportExportProjectCleanupWorker
IrkerWorker
MergeWorker
NewNoteWorker
PipelineHooksWorker
PipelineProcessWorker
PipelineSuccessWorker
PipelineUpdateWorker
ProjectCacheWorker
ProjectDestroyWorker
PruneOldEventsWorker
RemoveExpiredGroupLinksWorker
RemoveExpiredMembersWorker
RepositoryArchiveCacheWorker
RepositoryCheck::BatchWorker
RepositoryCheck::ClearWorker
RepositoryCheck::SingleRepositoryWorker
RequestsProfilesWorker
StuckCiBuildsWorker
UpdateMergeRequestsWorker

Mentioned in issue gitlab-com/infrastructure#584 (closed)

@stanhu I would go as far as saying that this needs to be tackled in GitLab itself too: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-13-stable/bin/background_jobs#L40. Is there a way we could do this in one location? Maybe have a file with all queues listed (for EE too) and then import it into the source installation script and we can do the same for omnibus?

@marin So for https://gitlab.com/gitlab-org/gitlab-ce/issues/23352 we'll probably need a configuration file specifying what settings to use for the various queues. How do you feel about also using that file to build the list of jobs to start in Omnibus? I'm thinking of using a YAML file of sorts that per queue lists the settings, even if none are needed. We can then enforce these queues being present using something like Rubocop or https://github.com/danger/danger. The config file would look something like:

---
post_receive:
  workers: 5
  jobs_per_minute: XXX
default:
  workers: 2
...

Added ~307491 label

Milestone changed to %8.14

Moving this to 8.14 since this will require Omnibus changes, and I'd rather not make those so close to a release.

Also worth mentioning: this change will require downtime. By changing workers to use different queues any existing jobs in the old queues won't be processed. This means we need to either move the jobs to the new queues, or take Unicorns down and let Sidekiq process everything; then deploy.

Mentioned in merge request !6839 (merged)

Milestone changed to %8.13

Reassigned to @yorickpeterse

This is becoming critical as queues are filling up all the time, I'll try to get this done ASAP CE/Omnibus wise so we can cram it into 8.13.

Mentioned in commit e1391ed5

Mentioned in merge request !7006 (merged)

Merge request: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7006

Mentioned in commit cfc4eda1

Mentioned in commit dcf333a9

Mentioned in commit 01e3903c

Mentioned in commit 57dabf82

Mentioned in commit cd0eec82

Mentioned in commit 2eeda446

Mentioned in commit 05f4d6f6

Mentioned in commit dd27ea85

Mentioned in commit 9c514fef

Mentioned in commit 1d316624

Mentioned in commit 7d99e874

Mentioned in commit ef46a799

Mentioned in commit 97731760

Status changed to closed by merge request !7006 (merged)

Mentioned in commit 083f9f8c

Mentioned in commit 0aed998d

mentioned in commit test-nick/gitlab-ce@382a6a00

mentioned in merge request !14785