Set timeouts for cache operations requests

We should configure additional timeouts for cache operations requests. Currently we use http.DefaultClient with its default timeouts. In case of cache server problems (like heavy load when cache server is responding much slower than usual) cache uploading or downloading may unnecessarily extend the duration of job. Finally, in a heavy used CI environment, it may end with pending jobs queue growth and CI system outage.

One of ideas is to set the http.Client.Timeout field to a reasonable time after which whole request will be canceled. But what time is reasonable? 1, 5, 10 minutes?

Other idea is to set the http.Transport.ResponseHeaderTimeout. This will work with every size of the cache file and fail only if cache server is slow on processing and sending the response.

What else could be proposed?

changed milestone to %Backlog

added cache improvement labels

/cc @ayufan

Another Idea I have is that we could set the total timeout with an environment variable like CACHE_REQUESTS_TIMEOUT. It could be set to a value like 60 seconds and anyone could overwrite it on Runner's configuration or .gitlab-ci.yml level.

mentioned in issue gitlab-com/infrastructure#2022 (closed)

mentioned in merge request !608 (closed)

I would start with config-level configuration.

@ayufan But config.toml or .gitlab-ci.yml? Or both?

config.toml

@bikebilly I totally agree! :)