Add troubleshooting guideline for git stuck processes

README.md

@@ -43,6 +43,10 @@ The aim of this project is to have a quick guide of what to do when an emergency
 * [Runners registry is down](troubleshooting/runners_registry_is_down.md)
 * [Runners cache free disk space is less than 20%](troubleshooting/runners_cache_disk_space.md)
+### Git Processes
+
+* [Troubleshoot git stuck processes](troubleshooting/git-stuck-processes.md)
+
 ### CephFS
 * [CephFS warns "failing to respond to cache pressure"](troubleshooting/cephfs.md)

troubleshooting/git-stuck-processes.md

+# Git Stuch Processes
+
+## First and foremost
+
+*Don't Panic*
+
+## Symptoms
+
+There is a large amount of git receive-pack processes in the fleet overview dashboard (at the bottom)
+
+## Reason
+
+Workhorse is not killing connections on a deadline after the client went away, this means that these processes are dangling blocked on IO, effectively doing nothing.
+
+## Prechecks
+
+Count how many processes are dangling, more than 10 is way too much for our current load (this may change over time)
+
+`knife ssh roles:gitlab-fe-git 'ps -eo cmd,pid,etimes= | grep receive-pack | wc -l'`
+
+## Resolution
+
+Kill all the processes that are dangling for more than one hour
+
+`knife ssh roles:gitlab-fe-git 'ps -eo etimes=,pid,cmd,pid | grep receive-pack | awk "{ if (\$1 > 1800) { print \$2 }}" | xargs sudo kill '`
+
+## Postchecks
+
+Review the dashboard to see a drop, and consider running the prechecks again