Add backpressure mechanism to mirror cronjob registration when under threshold
After adding capacity for the mirrors we are noticing that the amount of queued cronjobs is growing insanely.
The current theory is that this is happening because we have a lower threshold in which we register a cronjob to optimize job registration and resource usage, and we are under this threshold all the time because we have enough capacity to take the jobs fast enough.
Additionally, this was noticeable when we had to fly for a while without enough best effort capacity (https://gitlab.com/gitlab-com/infrastructure/issues/2466) so jobs were just piling up. This explains the final drop in the graph.
Even though these jobs are quick to execute, they hit the DB, so I think we should be adding a backpressure mechanism in the form only registering cronjobs once per couple of seconds or so, this way if we are so fast in executing mirror, we would still avoid registering so many.
cc/ @DouweM