Post-mortem for the nfs011 issue where repositories went temporarily missing for customers

Context

A set of users with repositories stored on a single storage node saw some of their repositories disappear on gitlab.com. This was due to the fact that there was an issue related to repository caching on redis. At the same time there was a single user that caused heavy I/O resulting in a CPU spike on the same storage node.

Timeline

On date: 2017-08-15

• 00:15 UTC - First report by a user reporting a missing repo https://gitlab.com/gitlab-com/support-forum/issues/2320
• 07:00 UTC - Slowness on gitlab.com noticed by Gitlab employees - https://gitlab.slack.com/archives/C101F3796/p1502780359000006
• 07:00 UTC - Hackernews post - https://news.ycombinator.com/item?id=15016173
• 07:15 UTC - oncall paged via pagerduty and https://gitlab.com/gitlab-com/infrastructure/issues/2503 opened to track issue.
• 07:18 UTC - first tweet sent to alert users of the issue - https://gitlab.slack.com/archives/C101F3796/p1502781533000208
• 08:38 UTC - root cause identified as a single user causing problems but sending us a large amount of data
• 08:52 UTC - script started to clear the redis cache on nfs011
• 08:57 UTC - script finished to clear redis cache on nfs011, issue with missing repos resolved

Incident Analysis

• How was the incident detected? - By a user and then later by gitlab employees looking at the fleet overview / slack.
• Is there anything that could have been done to improve the time to detection? - Better alarming
• How was the root cause discovered? - Production engineer examining nfs011, ssh.
• Was this incident triggered by a change? - No
• Was there an existing issue that would have either prevented this incident or reduced the impact? - No

Root Cause Analysis

• TBD

...

What went well

• Coordination and the team working together.

What can be improved

• It took more time than it should to run the cache clean script, this should probably be a rake command and delivered and tested with the application
• We were unable to view the gitaly logs in ELK - https://gitlab.com/gitlab-com/infrastructure/issues/2505
• Better gitaly metrics and monitoring and per process monitoring for gitaly and git on the storage nodes
  • Gitaly Child Process accounting: https://gitlab.com/gitlab-org/gitaly/merge_requests/284: with this we'll be able to know about repos that are taking a disproportional amount of system resources.
• Repository limits: the user had a 91GB git repository. We should impose limits on this type of usage
• Gitaly Process CGroup: The NFS servers should be protected from Gitaly and it's child processes: https://gitlab.com/gitlab-com/infrastructure/issues/2364
• Additional Runbook Information: On-call engineers should be better informed of when and how to kill Gitaly git child processes - https://gitlab.com/gitlab-com/runbooks/merge_requests/330
  • (Particularly zombie git processes -- who's parent processes have been killed). Any git process with a PPID of 1 can almost always be killed (we could even script this?)
• Runbook Scripts Out of Date: if possible, runbook utilities like the one used to clear the cache, should be checked into the Git repo as methods and tested as part of CI, to ensure that they don't get out of date and break due to application code changes
  • This happened in this issue and prolonged the incident
• Repository Exists cache TTL should be set to seconds TTL: especially once it moves over to Gitaly.

Corrective actions

• https://gitlab.com/gitlab-com/infrastructure/issues/1498
• https://gitlab.com/gitlab-com/infrastructure/issues/2511
• https://gitlab.com/gitlab-com/runbooks/merge_requests/330

Guidelines

• Blameless Postmortems Guideline
• 5 whys

assigned to @jarv

added toil label

changed milestone to %WoW ending 2017-08-15

changed the description

marked this issue as related to #2491

marked this issue as related to #2505 (closed)

Some more issues....

What can be improved

• Gitaly Child Process accounting: https://gitlab.com/gitlab-org/gitaly/merge_requests/284: with this we'll be able to know about repos that are taking a disproportional amount of system resources.
• Repository limits: the user had a 91GB git repository. We should impose limits on this type of usage
• Gitaly Process CGroup: The NFS servers should be protected from Gitaly and it's child processes: https://gitlab.com/gitlab-com/infrastructure/issues/2364
• Additional Runbook Information: On-call engineers should be better informed of when and how to kill Gitaly git child processes
  • (Particularly zombie git processes -- who's parent processes have been killed). Any git process with a PPID of 1 can almost always be killed (we could even script this?)
• Runbook Scripts Out of Date: if possible, runbook utilities like the one used to clear the cache, should be checked into the Git repo as methods and tested as part of CI, to ensure that they don't get out of date and break due to application code changes
  • This happened in this issue and prolonged the incident
• Repository Exists cache TTL should be set to seconds TTL: especially once it moves over to Gitaly. This is a very cheap call and it's TTL doesn't match the cost of the underlying call at all.

I'd also like to reiterate how important logging is becoming to helping us find issues like this. Without our ELK logging infrastructure, we're effectively flying blind on what's going on in Gitaly (and to a large degree, on the NFS servers)

changed the description

@andrewn thanks! I updated the section with your additions.

Can this be made public?

This issue is fine to be made public as it only references other issues that have the repo names that were involved.

made the issue visible to everyone

mentioned in issue #2493 (closed)

added (perceived) data loss storage labels

changed milestone to %WoW ending 2017-08-22

removed toil label

@jarv

One further action is: https://gitlab.com/gitlab-com/infrastructure/issues/1498 (cc/ @sitschner )

I think there was an issue to improve the cache handling for failures, but I've no idea where was it (cc/ @stanhu and I think that @jacobvosmaer-gitlab was involved in that one)

Gitaly cgroups limits: https://gitlab.com/gitlab-com/infrastructure/issues/2511

• Additional Runbook Information: On-call engineers should be better informed of when and how to kill Gitaly git child processes

  • (Particularly zombie git processes -- who's parent processes have been killed). Any git process with a PPID of 1 can almost always be killed (we could even script this?)

Please, let's do it.

changed the description

The circuit breaker was supposed to avoid this problem, but we would have to activate the git_storage_circuit_breaker feature flag. We'd want Prometheus metrics to monitor and alert on this.

@pchojnacki Do we have a version bump to get https://gitlab.com/gitlab-org/prometheus-client-mmap/merge_requests/6 into 9.5? I don't see it at the moment.

Does it make sense to revisit gitlab-org/gitlab-ce#33117?

changed the description

removed the relation with #2505 (closed)

removed the relation with #2491

@stanhu I just cut the gem: 0.7.0.beta12 that gives almost 10x improvement in my testing. Also I've been working on additional performance improvements that I hope will give additional 2x improvement. https://gitlab.com/gitlab-org/prometheus-client-mmap/merge_requests/7

@pchojnacki Great, thanks. Can we bump the version for the next RC ASAP so that we at least have the initial set of improvements?

mentioned in issue #2511 (closed)

I think we can close this one as we have further actions and some of them are already ongoing.

Thanks all!

closed