Set smaller number of retries for `WebHookWorker`

By default Sidekiq will retry a job 25 times over about 21 days (https://github.com/mperham/sidekiq/wiki/Error-Handling). With web hooks, when a remote service is down for whatever reason, this can cause Sidekiq jobs to pile up. In the case of one customer in https://gitlab.zendesk.com/agent/tickets/73354 there were so many retries dropping back into the main queue that newer hooks and jobs were not executing in a timely manner.

I think we should set a smaller number of retries for web hooks. It's reasonable to try a few times, but if we can't execute the hook after a few tries, it's less likely that we'll ever be able to. On top of that, at what point is an old web hook no longer useful?

cc/ @stanhu @DouweM What do you think a sane number of retries is?

cc/ @mydigitalself For eventual scheduling. Once we decide on a sane number, implementation is a 5 minute job.

Didn't we set this number to 4, or does this retry limit not apply?

https://gitlab.com/gitlab-org/gitlab-ce/blob/v9.0.5/app/workers/project_web_hook_worker.rb#L5

@stanhu Yep, looks like we did. This customer is having the problem with system hooks, which falls under SystemHookWorker. We also have WebHookWorker which appears to be used by CI? #toomanywebhookworkers

We at least should set retries to 4 for all workers.

@stanhu I can put together an MR. Is this something we can ship in a patch, or slip in to 9.1? customer+ would like to get this in ASAP since it caused other more important/timely tasks to be delayed.

@dblessing Slipping this into 9.1 would be okay with me. @DouweM should sign off too, as well as @godfat or @felipe_artur.

added webhooks label

Sure, I thought we already do so, and I don't see this could be blocking releases at all.

changed milestone to %9.1

mentioned in merge request !10801 (merged)

closed via merge request !10801 (merged)

mentioned in commit tnir/gitlab-ce@bfb635a4

mentioned in commit 356a797b