Add Filesystem Performance Benchmarking documentation

Filesystem performance can have a big impact on overall GitLab
performance, especially for actions that read or write Git
repositories. This information will help benchmark filesystem
performance against known good and bad real-world systems.

doc/administration/high_availability/nfs.md

@@ -3,6 +3,11 @@
 You can view information and options set for each of the mounted NFS file
 systems by running `nfsstat -m` and `cat /etc/fstab`.
+NOTE: **Note:** Filesystem performance has a big impact on overall GitLab
+performance, especially for actions that read or write to Git repositories. See
+[Filesystem Performance Benchmarking](../operations/filesystem_benchmarking.md)
+for steps to test filesystem performance.
+
 ## NFS Server features
 ### Required features

doc/administration/operations/filesystem_benchmarking.md

+# Filesystem Performance Benchmarking
+
+Filesystem performance has a big impact on overall GitLab performance,
+especially for actions that read or write to Git repositories. This information
+will help benchmark filesystem performance against known good and bad real-world
+systems.
+
+Normally when talking about filesystem performance the biggest concern is
+with Network Filesystems (NFS). However, even some local disks can have slow
+IO. The information on this page can be used for either scenario.
+
+## Write Performance
+
+The following one-line command is a quick benchmark for filesystem write
+performance. This will write 1,000 small files to the directory in which it is
+executed.
+
+1. Change into the root of the appropriate
+   [repository storage path](../repository_storage_paths.md).
+1. Create a temporary directory for the test so it's easy to remove the files later:
+
+    ```sh
+    mkdir test; cd test
+    ```
+1. Run the command:
+
+    ```sh
+    time for i in {0..1000}; do echo 'test' > "test${i}.txt"; done
+    ```
+1. Remove the test files:
+
+   ```sh
+   cd ../; rm -rf test
+   ```
+
+The output of the `time for ...` command will look similar to the following. The
+important metric is the `real` time.
+
+```sh
+$ time for i in {0..1000}; do echo 'test' > "test${i}.txt"; done
+
+real	0m0.116s
+user	0m0.025s
+sys	0m0.091s
+```
+
+From experience with multiple customers, the following are ranges that indicate
+whether your filesystem performance is satisfactory or less than ideal:
+
+| Rating    | Benchmark result        |
+|:----------|:------------------------|
+| Best      | Less than 10 seconds    |
+| OK        | 10-18 seconds           |
+| Poor      | 18-25 seconds           |
+| Very poor | Greater than 25 seconds |

doc/administration/operations/index.md

@@ -16,3 +16,7 @@ to restart Sidekiq.
 indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or
 by [doing away with user SSH keys stored on GitLab entirely in favor
 of SSH certificates](ssh_certificates.md).
+- [Filesystem Performance Benchmarking](filesystem_benchmarking.md): Filesystem
+performance can have a big impact on GitLab performance, especially for actions
+that read or write Git repositories. This information will help benchmark
+filesystem performance against known good and bad real-world systems.