Merge branch 'shared-file-access' into 'master'

Start putting shared files in "shared"



See merge request !1691

CONTRIBUTING.md

@@ -83,6 +83,7 @@ If you can, please submit a merge request with the fix or improvements including
 1. Be prepared to answer questions and incorporate feedback even if requests for this arrive weeks or months after your MR submission
 1. If your MR touches code that executes shell commands, make sure it adheres to the [shell command guidelines](    doc/development/shell_commands.md).
 1. Also have a look at the [shell command guidelines](doc/development/shell_commands.md) if your code reads or opens files, or handles paths to files on disk.
+1. If your code creates new files on disk please read the [shared files guidelines](doc/development/shared_files.md).
 The **official merge window** is in the beginning of the month from the 1st to the 7th day of the month. The best time to submit a MR and get feedback fast.
 Before this time the GitLab B.V. team is still dealing with work that is created by the monthly release such as regressions requiring patch releases.

config/gitlab.yml.example

@@ -310,6 +310,10 @@ production: &base
       #       application_name: 'YOUR_APP_NAME',
       #       application_password: 'YOUR_APP_PASSWORD' } }
+  # Shared file storage settings
+  shared:
+    # path: /mnt/gitlab # Default: shared
+

config/initializers/1_settings.rb

@@ -125,6 +125,9 @@ Settings.omniauth['auto_link_ldap_user'] = false if Settings.omniauth['auto_link
 Settings.omniauth['providers']  ||= []
+Settings['shared'] ||= Settingslogic.new({})
+Settings.shared['path'] = File.expand_path(Settings.shared['path'] || "shared", Rails.root)
+
 Settings['issues_tracker']  ||= {}
 #
@@ -169,7 +172,7 @@ Settings.gitlab.default_projects_features['merge_requests'] = true if Settings.g
 Settings.gitlab.default_projects_features['wiki']           = true if Settings.gitlab.default_projects_features['wiki'].nil?
 Settings.gitlab.default_projects_features['snippets']       = false if Settings.gitlab.default_projects_features['snippets'].nil?
 Settings.gitlab.default_projects_features['visibility_level']    = Settings.send(:verify_constant, Gitlab::VisibilityLevel, Settings.gitlab.default_projects_features['visibility_level'], Gitlab::VisibilityLevel::PRIVATE)
-Settings.gitlab['repository_downloads_path'] = File.absolute_path(Settings.gitlab['repository_downloads_path'] || 'tmp/repositories', Rails.root)
+Settings.gitlab['repository_downloads_path'] = File.join(Settings.shared['path'], 'cache/archive') if Settings.gitlab['repository_downloads_path'].nil?
 Settings.gitlab['restricted_signup_domains'] ||= []
 Settings.gitlab['import_sources'] ||= ['github','bitbucket','gitlab','gitorious','google_code','fogbugz','git']
@@ -248,6 +251,7 @@ Settings.git['timeout']   ||= 10
 Settings['satellites'] ||= Settingslogic.new({})
 Settings.satellites['path'] = File.expand_path(Settings.satellites['path'] || "tmp/repo_satellites/", Rails.root)
+
 #
 # Extra customization
 #

doc/development/shared_files.md

+# Shared files
+
+Historically, GitLab has been storing shared files in many different
+directories: `public/uploads`, `builds`, `tmp/repositories`, `tmp/rebase` (EE),
+etc. Having so many shared directories makes it difficult to deploy GitLab on
+shared storage (e.g. NFS). Working towards GitLab 9.0 we are consolidating
+these different directories under the `shared` directory.
+
+This means that if GitLab will start storing puppies in some future version
+then we should put them in `shared/puppies`. Temporary puppy files should be
+stored in `shared/tmp`.
+
+In the GitLab application code you can get the full path to the `shared`
+directory with `Gitlab.config.shared.path`.
+
+## What is not a 'shared file'
+
+Files that belong to only one process, or on only one server, should not go in
+`shared`. Examples include PID files and sockets.
+
+## Temporary files and shared storage
+
+Sometimes you create a temporary file on disk with the intention of it becoming
+'official'. For example you might be first streaming an upload from a user to
+disk in a temporary file so you can perform some checks on it. When the checks
+pass, you make the file official. In scenarios like this please follow these
+rules:
+
+- Store the temporary file under `shared/tmp`, i.e. on the same filesystem you
+  want the official file to be on.
+- Use move/rename operations when operating on the file instead of copy
+  operations where possible, because renaming a file is much faster than
+  copying it.