Run only one pipeline and project processing when scheduled multiple times

Open Kamil Trzcińśki requested to merge unique-sidekiq-jobs into master

What does this MR do?

Run only one Sidekiq job at single time with the same parameters when scheduled multiple times.

Why was this MR needed?

We are now hit by the issue that we schedule multiple jobs of the same context. All of them are being processed making it very computing intensive to process.

In all these cases it's completely fine to execute one, if there's more asking to queue, drop duplicates in queue and execute it second time.

I'm not sure about the stability of this gem, but this seems to be what we exactly need to easy the load on Sidekiq.

How it should work

If I understand that correctly this configuration should:

  1. You perform_async, nothing is in queue, it gets scheduled,
  2. The perform is executing,
  3. You perform_async, nothing is in queue, it gets scheduled, but since it's performing already it's added to queue,
  4. You perform_async, something is in queue, it gets dropped,
  5. The perform does finish,
  6. The new queued job gets scheduled.

At single time at most one job should be executed and queued in sidekiq in specific context.

Does this MR meet the acceptance criteria?

What are the relevant issue numbers?

cc @stanhu @yorickpeterse @pcarranza