Setup-test-env running for more than 101 minutes
When I was going through the CE => EE merge for 9-3-stable, Setup-test-env job continued running after 101 minutes.
https://gitlab.com/gitlab-org/gitlab-ee/builds/18569573
I manually killed the job and re-triggered the pipeline.
/cc @ayufan
Activity
It was caused by Runners cache server problem. It seems that Minio server was hanging. We've restarted the process and jobs started to be processed normally:
As we can see jobs hanging on cache download/upload stage caused Runner to fill up it limits. This ended with rate of created/deleted/used autoscaled machines to be very low:
First it suggested that there is an issue with our machines provider. But logs didn't confirm it. Also our cloud provider responded for our question that they are not seeing any issues. In meanwhile @ayufan investigated cache server because it seemed that all jobs that are hanging, are hanging on cache operations. This lead us to the solution described at the beginning.
After a quick talk on Slack we've started gitlab-org/gitlab-ci-multi-runner#2518 to discuss how we could improve cache operations to protect from such problems in the future.
