Retry all docker operations

We are seeing some random failures that are docker-related. It looks like most of them can be avoided and in the end will not fail a build if we were to simply retry pull images, inspects, container starts, attaching to output.

Current failure rate is fairly low: according to logs from our shared-runners manager it is less than 1% (20 to 3000 processed builds).

@tmaczukin What do you think?

Snippet with a grep from logs: https://gitlab.com/gitlab-com/operations/snippets/17727

Retry is a good idea, build should be as stable as possible.

@ayufan Yes, let's try to do a retry of docker operations. In the snippet I see errors, that were already reported, and that are failing almost each build (see: gitlab-com/support-forum#652 (closed)).

This sounds like a sane idea. Note, though, that we had none of these errors whatsoever for months. They're just popping up everywhere now for some reason.

Having this method as a general rule sounds like a great improvement, but at the same time I would investigate where all these network/connectivity issues are coming from.

@mvdan Some of them are timeouts due to non-correctly working keep alives, some of them just docker engine errors.

failure rate can be very low, but if it happend builds failing again and again

mentioned in issue #1317 (closed)

I have the same problem , I would say my failure rate is over 50% dependent of what time I run a build. I noticed that after midnight BST it works pretty good, daytime the failure rate comes very close to 100%.

I'm getting this too. Seems like it happens reliably when the runner has been sitting idle for a while.

@tmaczukin Can we give this high priority?

Milestone changed to %v1.5

Milestone changed to %Backlog