Improve docker images pulling

We've seen situations when docker images pulling is hanging (because of networking/mirroring/registry problems). We should improve images pulling with:

• adding context abort (which will fail job fast if image pulling is hanging), (done with !559 (merged))
• adding a configurable retry of images pull (like we do for git clone/fetch).

changed milestone to %v9.2

Yes indeed. Also is there a way GitLab CI can cancel these jobs that aren't progressing?

@stanhu I don't think so. There is StuckCiJobsWorker, but it will drop a running job if it was not updated (basing on updated_at field) for more than hour. In this case Runner still process such job, even if nothing is happening inside of the executor, and sends an update (even if nothing was added to the trace) each 30 seconds. This was added because other way GitLab CI would treat a job as stuck and drop it even in case, when a job is not generating any output. But there is timeout set in project settings which should force Runner to fail such job. We're seeing it from time to time e.g. for GitLab CE/EE jobs.

Were jobs, hanging because of this problem, running longer than timeout configured for GitLab CE and GitLab EE projects?

Were jobs, hanging because of this problem, running longer than timeout configured for GitLab CE and GitLab EE projects?

Yes. I saw some jobs running for hours.

mentioned in merge request !559 (merged)

added Deliverable label

changed milestone to %Backlog