Configure sidekiq-cron via gitlab.yml

In !1982 (merged) we added a hard-coded config/schedule.yml file. I think it would be better if the schedule was defined in config/gitlab.yml.

• GitLab already has way too many config files
• sidekiq-cron does not force us to use that file, we wrote our own initalizer that sucks in config/schedule.yml
• if we do add a config file it should be a .example file so that people can edit it without losing their changes during upgrades from source

The simplest thing is just to put this configuration in gitlab.yml somewhere.

cc @brodock @dzaporozhets @vsizov

Milestone changed to 8.2

Putting this configuration in gitlab.yml also makes packaging simpler. cc @marin

@jacobvosmaer . I don't think that !1982 (merged) in this form will even work with omnibus.

Thanks @jacobvosmaer I was about to create a similar issue. @brodock claims that GitLab config object is available in schedule.yml (I'm not sure how). If that's the case, we should be able to do this?

@brodock brought up in chat whether we should change the configuration key to something where the user defines a full cron syntax value, or whether we re-use the existing configuration for hour and minute and add additional for the other fields.

@dblessing we can have access to it because it is loaded by config/initializers/sidekiq.rb after initial Rails bootstrap.

If we move that logic to the Gitlab config file, we will have to make a few changes into the sidekiq (cron) initializer to decide whether or not run LdapSyncWorker (which I think its okay).

I vote for cron syntax, as its lot more powerful IMHO, ex:

# every 6 hours, interleaving full hours with halves, on workdays only
*/30 */6 * 1-5 *

It makes sense to move it to gitlab.yml, I haven't done that because of the scary warning

Will make a proposal draft here so we can discuss, ok?

@jacobvosmaer @marin @dblessing :

production: &base
  #
  # 1. GitLab app settings
  # ==========================

  # ...

  ## Gravatar
  ## For Libravatar see: http://doc.gitlab.com/ce/customization/libravatar.html
  gitlab:
  # ...

  ## Auxiliary jobs
  # Periodically executed jobs, to self-heal Gitlab, do external synchronizations, etc.
  # Please read here for more information: https://github.com/ondrejbartas/sidekiq-cron#adding-cron-job
  cron_jobs:
    stuck_ci_builds_worker:
      cron: "0 0 * * *"
      class: "StuckCiBuildsWorker"
      queue: "default"

    #
    # GitLab EE only jobs:
    #

    historical_data_worker:
      cron: "0 12 * * *"
      class: "HistoricalDataWorker"
      queue: "default"

    update_all_mirrors_worker:
      cron: "0 * * * *"
      class: "UpdateAllMirrorsWorker"
      queue: "default"

    # In addition to refreshing users when they log in,
    # enabling this setting will refresh LDAP user membership once a day.
    ldap_sync_worker:
      cron: "30 1 * * *"
      class: "LdapSyncWorker"
      queue: "default"

Thanks for explaining @brodock

Even if we choose to implement a config key per field it can still handle all the same values. I'm not sure about the best way to go here. Breaking changes are not ideal.

@dblessing are you talking about the existing config options for the LDAP Sync worker? I think in this case it is OK to deprecate that setting in favor of the new cron format.

@brodock that looks much nicer. I would only change it so that config/initializers/1_settings.rb fills in the queue and class fields. The class field should not even be configurable in gitlab.yml.

  cron_jobs:
    stuck_ci_builds_worker:
      cron: "0 0 * * *"

    #
    # GitLab EE only jobs:
    #

    historical_data_worker:
      cron: "0 12 * * *"

    update_all_mirrors_worker:
      cron: "0 * * * *"

    # In addition to refreshing users when they log in,
    # enabling this setting will refresh LDAP user membership once a day.
    ldap_sync_worker:
      cron: "30 1 * * *"

mentioned in merge request !2087 (merged)

Milestone changed to 8.3

@rspeicher @brodock So we release 8.3.0 without fixes for this?

mentioned in commit e6e338f0

@marin It wasn't merged and no one was assigned to it, not sure what I should have done differently.

mentioned in commit b0f1453f

Milestone changed to 8.6

Status changed to closed

Mentioned in commit pfjason/gitlab-ce@683a7954

mentioned in issue #31674 (moved)