Support multiple Redis instances based on queue type

Proposed implementation for support for multiple Redis key value stores

  • Add support for multiple key-value stores
    • allows different policies for retention, backup, deployment topologies by key namespace
  • Abstract key-value store name away from name of underlying technology such as "Redis"
  • note that config files, unix port numbers, and unix sockets for communication with Redis have been cloned and reconfigured

  • this MR is primarily submitted for review of
    • design approach
    • scope of changes
    • size of MR (1000+ lines -- could be broken into a few smaller step-wise pieces)
  • GDK testing can be performed with WIP MR for gdk
  • the Gitlab::Redis class was deliberately removed/deprecated to force the runtime to find any uses that were not readily apparent from grep/search of git repository
  • the modified GDK launches 3 separate KV servers (redis instances at present moment) and wraps each instance in a separate class (Cache, Queues, SharedState), each with bindings to appropriate namespaces within the KV store.
  • the modified CI launches 1 redis instance, and assigns each KV store to a different database number

Areas considered:

  • CI support
  • Build Support - deferred to issue omnibus-gitlab#2389
  • Deployment (Omnibus) - deferred to issue omnibus-gitlab#2389
  • Developer Support (GDK) - deferred to issue gitlab-development-kit#238
  • Documentation
  • retention policy for different namespaces
  • abstract class between new KV objects and Redis for shared behaviors
  • Migration from single instance to multiple instances - deferred to issue gitlab-org/gitlab-ce#33185
  • deferred to issue gitlab-org/gitlab-ce#33181 - * [ ] using atomic transactions
  • deferred to investigation issue gitlab-org/gitlab-ce#33182 - * [ ] whether certain key-values can be local to one FE node, or MUST be shared with other FE nodes
  • deferred to release management process - * [ ] Changelog
  • deferred - * [ ] using actual Model objects for encapsulation of underlying technology
  • deferred - * [ ] support for key-value pairs to participate in Rails Transactions with rollback on exception

Partially solves:

  • gitlab-org/gitlab-ce#30392

See also:

Edited by Robert Speicher