Add latest changes from gitlab-org/gitlab@master

doc/administration/geo/replication/object_storage.md

@@ -30,7 +30,7 @@ To enable GitLab replication, you must:
    checkbox.
 For LFS, follow the documentation to
-[set up LFS object storage](../../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage).
+[set up LFS object storage](../../lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage).
 For CI job artifacts, there is similar documentation to configure
 [jobs artifact object storage](../../job_artifacts.md#using-object-storage)

doc/administration/git_annex.md

+---
+disqus_identifier: 'https://docs.gitlab.com/ee/workflow/git_annex.html'
+---
+
+# Git annex
+
+> **Warning:** GitLab has [completely
+removed][deprecate-annex-issue] in GitLab 9.0 (2017/03/22).
+Read through the [migration guide from git-annex to Git LFS][guide].
+
+The biggest limitation of Git, compared to some older centralized version
+control systems, has been the maximum size of the repositories.
+
+The general recommendation is to not have Git repositories larger than 1GB to
+preserve performance. Although GitLab has no limit (some repositories in GitLab
+are over 50GB!), we subscribe to the advice to keep repositories as small as
+you can.
+
+Not being able to version control large binaries is a big problem for many
+larger organizations.
+Videos, photos, audio, compiled binaries and many other types of files are too
+large. As a workaround, people keep artwork-in-progress in a Dropbox folder and
+only check in the final result. This results in using outdated files, not
+having a complete history and increases the risk of losing work.
+
+This problem is solved in GitLab Enterprise Edition by integrating the
+[git-annex] application.
+
+`git-annex` allows managing large binaries with Git without checking the
+contents into Git.
+You check-in only a symlink that contains the SHA-1 of the large binary. If you
+need the large binary, you can sync it from the GitLab server over `rsync`, a
+very fast file copying tool.
+
+## GitLab git-annex Configuration
+
+`git-annex` is disabled by default in GitLab. Below you will find the
+configuration options required to enable it.
+
+### Requirements
+
+`git-annex` needs to be installed both on the server and the client side.
+
+For Debian-like systems (e.g., Debian, Ubuntu) this can be achieved by running:
+
+```
+sudo apt-get update && sudo apt-get install git-annex
+```
+
+For RedHat-like systems (e.g., CentOS, RHEL) this can be achieved by running:
+
+```
+sudo yum install epel-release && sudo yum install git-annex
+```
+
+### Configuration for Omnibus packages
+
+For Omnibus GitLab packages, only one configuration setting is needed.
+The Omnibus package will internally set the correct options in all locations.
+
+1. In `/etc/gitlab/gitlab.rb` add the following line:
+
+   ```ruby
+   gitlab_shell['git_annex_enabled'] = true
+   ```
+
+1. Save the file and [reconfigure GitLab][] for the changes to take effect.
+
+### Configuration for installations from source
+
+There are 2 settings to enable git-annex on your GitLab server.
+
+One is located in `config/gitlab.yml` of the GitLab repository and the other
+one is located in `config.yml` of GitLab Shell.
+
+1. In `config/gitlab.yml` add or edit the following lines:
+
+   ```yaml
+   gitlab_shell:
+     git_annex_enabled: true
+   ```
+
+1. In `config.yml` of GitLab Shell add or edit the following lines:
+
+   ```yaml
+   git_annex_enabled: true
+   ```
+
+1. Save the files and [restart GitLab][] for the changes to take effect.
+
+## Using GitLab git-annex
+
+> **Note:**
+> Your Git remotes must be using the SSH protocol, not HTTP(S).
+
+Here is an example workflow of uploading a very large file and then checking it
+into your Git repository:
+
+```bash
+git clone git@example.com:group/project.git
+
+git annex init 'My Laptop'       # initialize the annex project and give an optional description
+cp ~/tmp/debian.iso ./           # copy a large file into the current directory
+git annex add debian.iso         # add the large file to git annex
+git commit -am "Add Debian iso"  # commit the file metadata
+git annex sync --content         # sync the Git repo and large file to the GitLab server
+```
+
+The output should look like this:
+
+```
+commit
+On branch master
+Your branch is ahead of 'origin/master' by 1 commit.
+  (use "git push" to publish your local commits)
+nothing to commit, working tree clean
+ok
+pull origin
+remote: Counting objects: 5, done.
+remote: Compressing objects: 100% (4/4), done.
+remote: Total 5 (delta 2), reused 0 (delta 0)
+Unpacking objects: 100% (5/5), done.
+From example.com:group/project
+   497842b..5162f80  git-annex  -> origin/git-annex
+ok
+(merging origin/git-annex into git-annex...)
+(recording state in git...)
+copy debian.iso (checking origin...) (to origin...)
+SHA256E-s26214400--8092b3d482fb1b7a5cf28c43bc1425c8f2d380e86869c0686c49aa7b0f086ab2.iso
+     26,214,400 100%  638.88kB/s    0:00:40 (xfr#1, to-chk=0/1)
+ok
+pull origin
+ok
+(recording state in git...)
+push origin
+Counting objects: 15, done.
+Delta compression using up to 4 threads.
+Compressing objects: 100% (13/13), done.
+Writing objects: 100% (15/15), 1.64 KiB | 0 bytes/s, done.
+Total 15 (delta 1), reused 0 (delta 0)
+To example.com:group/project.git
+ * [new branch]      git-annex -> synced/git-annex
+ * [new branch]      master -> synced/master
+ok
+```
+
+Your files can be found in the `master` branch, but you'll notice that there
+are more branches created by the `annex sync` command.
+
+Git Annex will also create a new directory at `.git/annex/` and will record the
+tracked files in the `.git/config` file. The files you assign to be tracked
+with `git-annex` will not affect the existing `.git/config` records. The files
+are turned into symbolic links that point to data in `.git/annex/objects/`.
+
+The `debian.iso` file in the example will contain the symbolic link:
+
+```
+.git/annex/objects/ZW/1k/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.png/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.iso
+```
+
+Use `git annex info` to retrieve the information about the local copy of your
+repository.
+
+---
+
+Downloading a single large file is also very simple:
+
+```bash
+git clone git@gitlab.example.com:group/project.git
+
+git annex sync             # sync Git branches but not the large file
+git annex get debian.iso   # download the large file
+```
+
+To download all files:
+
+```bash
+git clone git@gitlab.example.com:group/project.git
+
+git annex sync --content  # sync Git branches and download all the large files
+```
+
+By using `git-annex` without GitLab, anyone that can access the server can also
+access the files of all projects, but GitLab Annex ensures that you can only
+access files of projects you have access to (developer, maintainer, or owner role).
+
+## How it works
+
+Internally GitLab uses [GitLab Shell] to handle SSH access and this was a great
+integration point for `git-annex`.
+There is a setting in GitLab Shell so you can disable GitLab Annex support
+if you want to.
+
+## Troubleshooting tips
+
+Differences in version of `git-annex` on the GitLab server and on local machines
+can cause `git-annex` to raise unpredicted warnings and errors.
+
+Consult the [Annex upgrade page][annex-upgrade] for more information about
+the differences between versions. You can find out which version is installed
+on your server by navigating to <https://pkgs.org/download/git-annex> and
+searching for your distribution.
+
+Although there is no general guide for `git-annex` errors, there are a few tips
+on how to go around the warnings.
+
+### `git-annex-shell: Not a git-annex or gcrypt repository`
+
+This warning can appear on the initial `git annex sync --content` and is caused
+by differences in `git-annex-shell`. You can read more about it
+[in this git-annex issue][issue].
+
+One important thing to note is that despite the warning, the `sync` succeeds
+and the files are pushed to the GitLab repository.
+
+If you get hit by this, you can run the following command inside the repository
+that the warning was raised:
+
+```
+git config remote.origin.annex-ignore false
+```
+
+Consecutive runs of `git annex sync --content` **should not** produce this
+warning and the output should look like this:
+
+```
+commit  ok
+pull origin
+ok
+pull origin
+ok
+push origin
+```
+
+[annex-upgrade]: https://git-annex.branchable.com/upgrades/
+[deprecate-annex-issue]: https://gitlab.com/gitlab-org/gitlab/issues/1648
+[git-annex]: https://git-annex.branchable.com/ "git-annex website"
+[gitlab shell]: https://gitlab.com/gitlab-org/gitlab-shell "GitLab Shell repository"
+[guide]: lfs/migrate_from_git_annex_to_git_lfs.html
+[issue]: https://git-annex.branchable.com/forum/Error_from_git-annex-shell_on_creation_of_gcrypt_special_remote/ "git-annex issue"
+[reconfigure GitLab]: restart_gitlab.md#omnibus-gitlab-reconfigure
+[restart GitLab]: restart_gitlab.md#installations-from-source

doc/administration/gitaly/index.md

@@ -551,7 +551,7 @@ a few things that you need to do:
    to eliminate the need for a shared authorized_keys file.
 1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage)
    including [incremental logging](../job_logs.md#new-incremental-logging-architecture).
-1. Configure [object storage for LFS objects](../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage).
+1. Configure [object storage for LFS objects](../lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage).
 1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only).
 NOTE: **Note:**

doc/administration/index.md

@@ -161,9 +161,10 @@ Learn how to install, configure, update, and maintain your GitLab instance.
 ## Git configuration options
 - [Custom Git hooks](custom_hooks.md): Custom Git hooks (on the filesystem) for when webhooks aren't enough.
-- [Git LFS configuration](../workflow/lfs/lfs_administration.md): Learn how to configure LFS for GitLab.
+- [Git LFS configuration](lfs/lfs_administration.md): Learn how to configure LFS for GitLab.
 - [Housekeeping](housekeeping.md): Keep your Git repositories tidy and fast.
 - [Configuring Git Protocol v2](git_protocol.md): Git protocol version 2 support.
+- [Manage large files with `git-annex` (Deprecated)](git_annex.md)
 ## Monitoring GitLab

doc/administration/lfs/lfs_administration.md

+---
+disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html'
+---
+
+# GitLab Git LFS Administration
+
+Documentation on how to use Git LFS are under [Managing large binary files with Git LFS doc](manage_large_binaries_with_git_lfs.md).
+
+## Requirements
+
+- Git LFS is supported in GitLab starting with version 8.2.
+- Support for object storage, such as AWS S3, was introduced in 10.0.
+- Users need to install [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up.
+
+## Configuration
+
+Git LFS objects can be large in size. By default, they are stored on the server
+GitLab is installed on.
+
+There are various configuration options to help GitLab server administrators:
+
+- Enabling/disabling Git LFS support
+- Changing the location of LFS object storage
+- Setting up object storage supported by [Fog](http://fog.io/about/provider_documentation.html)
+
+### Configuration for Omnibus installations
+
+In `/etc/gitlab/gitlab.rb`:
+
+```ruby
+# Change to true to enable lfs - enabled by default if not defined
+gitlab_rails['lfs_enabled'] = false
+
+# Optionally, change the storage path location. Defaults to
+# `#{gitlab_rails['shared_path']}/lfs-objects`. Which evaluates to
+# `/var/opt/gitlab/gitlab-rails/shared/lfs-objects` by default.
+gitlab_rails['lfs_storage_path'] = "/mnt/storage/lfs-objects"
+```
+
+### Configuration for installations from source
+
+In `config/gitlab.yml`:
+
+```yaml
+# Change to true to enable lfs
+  lfs:
+    enabled: false
+    storage_path: /mnt/storage/lfs-objects
+```
+
+## Storing LFS objects in remote object storage
+
+> [Introduced][ee-2760] in [GitLab Premium][eep] 10.0. Brought to GitLab Core in 10.7.
+
+It is possible to store LFS objects in remote object storage which allows you
+to offload local hard disk R/W operations, and free up disk space significantly.
+GitLab is tightly integrated with `Fog`, so you can refer to its [documentation](http://fog.io/about/provider_documentation.html)
+to check which storage services can be integrated with GitLab.
+You can also use external object storage in a private local network. For example,
+[MinIO](https://min.io/) is a standalone object storage service, is easy to set up, and works well with GitLab instances.
+
+GitLab provides two different options for the uploading mechanism: "Direct upload" and "Background upload".
+
+**Option 1. Direct upload**
+
+1. User pushes an lfs file to the GitLab instance
+1. GitLab-workhorse uploads the file directly to the external object storage
+1. GitLab-workhorse notifies GitLab-rails that the upload process is complete
+
+**Option 2. Background upload**
+
+1. User pushes an lfs file to the GitLab instance
+1. GitLab-rails stores the file in the local file storage
+1. GitLab-rails then uploads the file to the external object storage asynchronously
+
+The following general settings are supported.
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| `enabled` | Enable/disable object storage | `false` |
+| `remote_directory` | The bucket name where LFS objects will be stored| |
+| `direct_upload` | Set to true to enable direct upload of LFS without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. | `false` |
+| `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` |
+| `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` |
+| `connection` | Various connection options described below | |
+
+The `connection` settings match those provided by [Fog](https://github.com/fog).
+
+Here is a configuration example with S3.
+
+| Setting | Description | example |
+|---------|-------------|---------|
+| `provider` | The provider name | AWS |
+| `aws_access_key_id` | AWS credentials, or compatible | `ABC123DEF456` |
+| `aws_secret_access_key` | AWS credentials, or compatible | `ABC123DEF456ABC123DEF456ABC123DEF456` |
+| `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 |
+| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true |
+| `region` | AWS region | us-east-1 |
+| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com |
+| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) |
+| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false |
+| `use_iam_profile` | Set to true to use IAM profile instead of access keys | false
+
+Here is a configuration example with GCS.
+
+| Setting | Description | example |
+|---------|-------------|---------|
+| `provider` | The provider name | `Google` |
+| `google_project` | GCP project name | `gcp-project-12345` |
+| `google_client_email` | The email address of the service account | `foo@gcp-project-12345.iam.gserviceaccount.com` |
+| `google_json_key_location` | The json key path | `/path/to/gcp-project-12345-abcde.json` |
+
+NOTE: **Note:**
+The service account must have permission to access the bucket.
+[See more](https://cloud.google.com/storage/docs/authentication)
+
+Here is a configuration example with Rackspace Cloud Files.
+
+| Setting | Description | example |
+|---------|-------------|---------|
+| `provider` | The provider name | `Rackspace` |
+| `rackspace_username` | The username of the Rackspace account with access to the container | `joe.smith` |
+| `rackspace_api_key` | The API key of the Rackspace account with access to the container | `ABC123DEF456ABC123DEF456ABC123DE` |
+| `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://developer.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` |
+| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for temporary URLs. Read more [here](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl) | `ABC123DEF456ABC123DEF456ABC123DE` |
+
+NOTE: **Note:**
+Regardless of whether the container has public access enabled or disabled, Fog will
+use the TempURL method to grant access to LFS objects. If you see errors in logs referencing
+instantiating storage with a temp-url-key, ensure that you have set they key properly
+on the Rackspace API and in `gitlab.rb`. You can verify the value of the key Rackspace
+has set by sending a GET request with token header to the service access endpoint URL
+and comparing the output of the returned headers.
+
+### Manual uploading to an object storage
+
+There are two ways to manually do the same thing as automatic uploading (described above).
+
+**Option 1: rake task**
+
+```sh
+rake gitlab:lfs:migrate
+```
+
+**Option 2: rails console**
+
+```sh
+$ sudo gitlab-rails console            # Login to rails console
+
+> # Upload LFS files manually
+> LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object|
+>   lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists?
+> end
+```
+
+### S3 for Omnibus installations
+
+On Omnibus installations, the settings are prefixed by `lfs_object_store_`:
+
+1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with
+   the values you want:
+
+   ```ruby
+   gitlab_rails['lfs_object_store_enabled'] = true
+   gitlab_rails['lfs_object_store_remote_directory'] = "lfs-objects"
+   gitlab_rails['lfs_object_store_connection'] = {
+     'provider' => 'AWS',
+     'region' => 'eu-central-1',
+     'aws_access_key_id' => '1ABCD2EFGHI34JKLM567N',
+     'aws_secret_access_key' => 'abcdefhijklmnopQRSTUVwxyz0123456789ABCDE',
+     # The below options configure an S3 compatible host instead of AWS
+     'host' => 'localhost',
+     'endpoint' => 'http://127.0.0.1:9000',
+     'path_style' => true
+   }
+   ```
+
+1. Save the file and [reconfigure GitLab]s for the changes to take effect.
+1. Migrate any existing local LFS objects to the object storage:
+
+   ```bash
+   gitlab-rake gitlab:lfs:migrate
+   ```
+
+   This will migrate existing LFS objects to object storage. New LFS objects
+   will be forwarded to object storage unless
+   `gitlab_rails['lfs_object_store_background_upload']` is set to false.
+
+### S3 for installations from source
+
+For source installations the settings are nested under `lfs:` and then
+`object_store:`:
+
+1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following
+   lines:
+
+   ```yaml
+   lfs:
+   enabled: true
+   object_store:
+     enabled: false
+     remote_directory: lfs-objects # Bucket name
+     connection:
+       provider: AWS
+       aws_access_key_id: 1ABCD2EFGHI34JKLM567N
+       aws_secret_access_key: abcdefhijklmnopQRSTUVwxyz0123456789ABCDE
+       region: eu-central-1
+       # Use the following options to configure an AWS compatible host such as Minio
+       host: 'localhost'
+       endpoint: 'http://127.0.0.1:9000'
+       path_style: true
+   ```
+
+1. Save the file and [restart GitLab][] for the changes to take effect.
+1. Migrate any existing local LFS objects to the object storage:
+
+   ```bash
+   sudo -u git -H bundle exec rake gitlab:lfs:migrate RAILS_ENV=production
+   ```
+
+   This will migrate existing LFS objects to object storage. New LFS objects
+   will be forwarded to object storage unless `background_upload` is set to
+   false.
+
+### Migrating back to local storage
+
+In order to migrate back to local storage:
+
+1. Set both `direct_upload` and `background_upload` to false under the LFS object storage settings. Don't forget to restart GitLab.
+1. Run `rake gitlab:lfs:migrate_to_local` on your console.
+1. Disable `object_storage` for LFS objects in `gitlab.rb`. Remember to restart GitLab afterwards.
+
+## Storage statistics
+
+You can see the total storage used for LFS objects on groups and projects
+in the administration area, as well as through the [groups](../../api/groups.md)
+and [projects APIs](../../api/projects.md).
+
+## Troubleshooting: `Google::Apis::TransmissionError: execution expired`
+
+If LFS integration is configred with Google Cloud Storage and background uploads (`background_upload: true` and `direct_upload: false`),
+Sidekiq workers may encouter this error. This is because the uploading timed out with very large files.
+LFS files up to 6Gb can be uploaded without any extra steps, otherwise you need to use the following workaround.
+
+```shell
+$ sudo gitlab-rails console            # Login to rails console
+
+> # Set up timeouts. 20 minutes is enough to upload 30GB LFS files.
+> # These settings are only in effect for the same session, i.e. they are not effective for sidekiq workers.
+> ::Google::Apis::ClientOptions.default.open_timeout_sec = 1200
+> ::Google::Apis::ClientOptions.default.read_timeout_sec = 1200
+> ::Google::Apis::ClientOptions.default.send_timeout_sec = 1200
+
+> # Upload LFS files manually. This process does not use sidekiq at all.
+> LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object|
+>   lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists?
+> end
+```
+
+See more information in [!19581](https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/19581)
+
+## Known limitations
+
+- Support for removing unreferenced LFS objects was added in 8.14 onwards.
+- LFS authentications via SSH was added with GitLab 8.12.
+- Only compatible with the Git LFS client versions 1.1.0 and up, or 1.0.2.
+- The storage statistics currently count each LFS object multiple times for
+  every project linking to it.
+
+[reconfigure gitlab]: ../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab"
+[restart gitlab]: ../restart_gitlab.md#installations-from-source "How to restart GitLab"
+[eep]: https://about.gitlab.com/pricing/ "GitLab Premium"
+[ee-2760]: https://gitlab.com/gitlab-org/gitlab/merge_requests/2760

doc/administration/lfs/manage_large_binaries_with_git_lfs.md

+---
+disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/manage_large_binaries_with_git_lfs.html'
+---
+
+# Git LFS
+
+Managing large files such as audio, video and graphics files has always been one
+of the shortcomings of Git. The general recommendation is to not have Git repositories
+larger than 1GB to preserve performance.
+
+![Git LFS tracking status](img/lfs-icon.png)
+
+An LFS icon is shown on files tracked by Git LFS to denote if a file is stored
+as a blob or as an LFS pointer.
+
+## How it works
+
+Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication
+to authorize client requests. Once the request is authorized, Git LFS client receives
+instructions from where to fetch or where to push the large file.
+
+## GitLab server configuration
+
+Documentation for GitLab instance administrators is under [LFS administration doc](lfs_administration.md).
+
+## Requirements
+
+- Git LFS is supported in GitLab starting with version 8.2
+- Git LFS must be enabled under project settings
+- [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up
+
+## Known limitations
+
+- Git LFS v1 original API is not supported since it was deprecated early in LFS
+  development
+- When SSH is set as a remote, Git LFS objects still go through HTTPS
+- Any Git LFS request will ask for HTTPS credentials to be provided so a good Git
+  credentials store is recommended
+- Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have
+  to add the URL to Git config manually (see [troubleshooting](#troubleshooting))
+
+NOTE: **Note:**
+With 8.12 GitLab added LFS support to SSH. The Git LFS communication
+still goes over HTTP, but now the SSH client passes the correct credentials
+to the Git LFS client, so no action is required by the user.
+
+## Using Git LFS
+
+Lets take a look at the workflow when you need to check large files into your Git
+repository with Git LFS. For example, if you want to upload a very large file and
+check it into your Git repository:
+
+```bash
+git clone git@gitlab.example.com:group/project.git
+git lfs install                       # initialize the Git LFS project
+git lfs track "*.iso"                 # select the file extensions that you want to treat as large files
+```
+
+Once a certain file extension is marked for tracking as a LFS object you can use
+Git as usual without having to redo the command to track a file with the same extension:
+
+```bash
+cp ~/tmp/debian.iso ./                # copy a large file into the current directory
+git add .                             # add the large file to the project
+git commit -am "Added Debian iso"     # commit the file meta data
+git push origin master                # sync the git repo and large file to the GitLab server
+```
+
+**Make sure** that `.gitattributes` is tracked by Git. Otherwise Git
+LFS will not be working properly for people cloning the project:
+
+```bash
+git add .gitattributes
+```
+
+Cloning the repository works the same as before. Git automatically detects the
+LFS-tracked files and clones them via HTTP. If you performed the `git clone`
+command with a SSH URL, you have to enter your GitLab credentials for HTTP
+authentication.
+
+```bash
+git clone git@gitlab.example.com:group/project.git
+```
+
+If you already cloned the repository and you want to get the latest LFS object
+that are on the remote repository, eg. for a branch from origin:
+
+```bash
+git lfs fetch origin master
+```
+
+### Migrate an existing repo to Git LFS
+
+Read the documentation on how to [migrate an existing Git repo with Git LFS](../../topics/git/migrate_to_git_lfs/index.md).
+
+## File Locking
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/35856) in GitLab 10.5.
+
+The first thing to do before using File Locking is to tell Git LFS which
+kind of files are lockable. The following command will store PNG files
+in LFS and flag them as lockable:
+
+```bash
+git lfs track "*.png" --lockable
+```
+
+After executing the above command a file named `.gitattributes` will be
+created or updated with the following content:
+
+```bash
+*.png filter=lfs diff=lfs merge=lfs -text lockable
+```
+
+You can also register a file type as lockable without using LFS
+(In order to be able to lock/unlock a file you need a remote server that implements the  LFS File Locking API),
+in order to do that you can edit the `.gitattributes` file manually:
+
+```bash
+*.pdf lockable
+```
+
+After a file type has been registered as lockable, Git LFS will make
+them readonly on the file system automatically. This means you will
+need to lock the file before editing it.
+
+### Managing Locked Files
+
+Once you're ready to edit your file you need to lock it first:
+
+```bash
+git lfs lock images/banner.png
+Locked images/banner.png
+```
+
+This will register the file as locked in your name on the server:
+
+```bash
+git lfs locks
+images/banner.png  joe   ID:123
+```
+
+Once you have pushed your changes, you can unlock the file so others can
+also edit it:
+
+```bash
+git lfs unlock images/banner.png
+```
+
+You can also unlock by id:
+
+```bash
+git lfs unlock --id=123
+```
+
+If for some reason you need to unlock a file that was not locked by you,
+you can use the `--force` flag as long as you have a `maintainer` access on
+the project:
+
+```bash
+git lfs unlock --id=123 --force
+```
+
+## Troubleshooting
+
+### error: Repository or object not found
+
+There are a couple of reasons why this error can occur:
+
+- You don't have permissions to access certain LFS object
+
+Check if you have permissions to push to the project or fetch from the project.
+
+- Project is not allowed to access the LFS object
+
+LFS object you are trying to push to the project or fetch from the project is not
+available to the project anymore. Probably the object was removed from the server.
+
+- Local Git repository is using deprecated LFS API
+
+### Invalid status for `<url>` : 501
+
+Git LFS will log the failures into a log file.
+To view this log file, while in project directory:
+
+```bash
+git lfs logs last
+```
+
+If the status `error 501` is shown, it is because:
+
+- Git LFS is not enabled in project settings. Check your project settings and
+  enable Git LFS.
+
+- Git LFS support is not enabled on the GitLab server. Check with your GitLab
+  administrator why Git LFS is not enabled on the server. See
+  [LFS administration documentation](lfs_administration.md) for instructions
+  on how to enable LFS support.
+
+- Git LFS client version is not supported by GitLab server. Check your Git LFS
+  version with `git lfs version`. Check the Git config of the project for traces
+  of deprecated API with `git lfs -l`. If `batch = false` is set in the config,
+  remove the line and try to update your Git LFS client. Only version 1.0.1 and
+  newer are supported.
+
+### getsockopt: connection refused
+
+If you push a LFS object to a project and you receive an error similar to:
+`Post <URL>/info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`,
+the LFS client is trying to reach GitLab through HTTPS. However, your GitLab
+instance is being served on HTTP.
+
+This behaviour is caused by Git LFS using HTTPS connections by default when a
+`lfsurl` is not set in the Git config.
+
+To prevent this from happening, set the lfs url in project Git config:
+
+```bash
+git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs"
+```
+
+### Credentials are always required when pushing an object
+
+NOTE: **Note:**
+With 8.12 GitLab added LFS support to SSH. The Git LFS communication
+still goes over HTTP, but now the SSH client passes the correct credentials
+to the Git LFS client, so no action is required by the user.
+
+Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing
+the LFS object on every push for every object, user HTTPS credentials are required.
+
+By default, Git has support for remembering the credentials for each repository
+you use. This is described in [Git credentials man pages](https://git-scm.com/docs/gitcredentials).
+
+For example, you can tell Git to remember the password for a period of time in
+which you expect to push the objects:
+
+```bash
+git config --global credential.helper 'cache --timeout=3600'
+```
+
+This will remember the credentials for an hour after which Git operations will
+require re-authentication.
+
+If you are using OS X you can use `osxkeychain` to store and encrypt your credentials.
+For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases).
+
+More details about various methods of storing the user credentials can be found
+on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage).
+
+### LFS objects are missing on push
+
+GitLab checks files to detect LFS pointers on push. If LFS pointers are detected, GitLab tries to verify that those files already exist in LFS on GitLab.
+
+Verify that LFS in installed locally and consider a manual push with `git lfs push --all`.
+
+If you are storing LFS files outside of GitLab you can disable LFS on the project by setting `lfs_enabled: false` with the [projects API](../../api/projects.md#edit-project).
+
+### Hosting LFS objects externally
+
+It is possible to host LFS objects externally by setting a custom LFS url with `git config -f .lfsconfig lfs.url https://example.com/<project>.git/info/lfs`.
+
+You might choose to do this if you are using an appliance like a Sonatype Nexus to store LFS data. If you choose to use an external LFS store,
+GitLab will not be able to verify LFS objects which means that pushes will fail if you have GitLab LFS support enabled.
+
+To stop push failure, LFS support can be disabled in the [Project settings](../../user/project/settings/index.md). This means you will lose GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS).

doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md

+# Migration guide from Git Annex to Git LFS
+
+>**Note:**
+Git Annex support [has been removed][issue-remove-annex] in GitLab Enterprise
+Edition 9.0 (2017/03/22).
+
+Both [Git Annex][] and [Git LFS][] are tools to manage large files in Git.
+
+## History
+
+Git Annex [was introduced in GitLab Enterprise Edition 7.8][post-3], at a time
+where Git LFS didn't yet exist. A few months later, GitLab brought support for
+Git LFS in [GitLab 8.2][post-2] and is available for both Community and
+Enterprise editions.
+
+## Differences between Git Annex and Git LFS
+
+Some items below are general differences between the two protocols and some are
+ones that GitLab developed.
+
+- Git Annex works only through SSH, whereas Git LFS works both with SSH and HTTPS
+  (SSH support was added in GitLab 8.12).
+- Annex files are stored in a sub-directory of the normal repositories, whereas
+  LFS files are stored outside of the repositories in a place you can define.
+- Git Annex requires a more complex setup, but has much more options than Git
+  LFS. You can compare the commands each one offers by running `man git-annex`
+  and `man git-lfs`.
+- Annex files cannot be browsed directly in GitLab's interface, whereas LFS
+  files can.
+
+## Migration steps
+
+>**Note:**
+Since Git Annex files are stored in a sub-directory of the normal repositories
+(`.git/annex/objects`) and LFS files are stored outside of the repositories,
+they are not compatible as they are using a different scheme. Therefore, the
+migration has to be done manually per repository.
+
+There are basically two steps you need to take in order to migrate from Git
+Annex to Git LFS.
+
+### TL; DR
+
+If you know what you are doing and want to skip the reading, this is what you
+need to do (we assume you have [git-annex enabled](../git_annex.md#using-gitlab-git-annex) in your
+repository and that you have made backups in case something goes wrong).
+Fire up a terminal, navigate to your Git repository and:
+
+1. Disable `git-annex`:
+
+   ```bash
+   git annex sync --content
+   git annex direct
+   git annex uninit
+   git annex indirect
+   ```
+
+1. Enable `git-lfs`:
+
+   ```
+   git lfs install
+   git lfs track <files>
+   git add .
+   git commit -m "commit message"
+   git push
+   ```
+
+### Disabling Git Annex in your repo
+
+Before changing anything, make sure you have a backup of your repository first.
+There are a couple of ways to do that, but you can simply clone it to another
+local path and maybe push it to GitLab if you want a remote backup as well.
+Here you'll find a guide on
+[how to back up a **git-annex** repository to an external hard drive][bkp-ext-drive].
+
+Since Annex files are stored as objects with symlinks and cannot be directly
+modified, we need to first remove those symlinks.
+
+NOTE: **Note:**
+Make sure the you read about the [`direct` mode][annex-direct] as it contains
+useful information that may fit in your use case. Note that `annex direct` is
+deprecated in Git Annex version 6, so you may need to upgrade your repository
+if the server also has Git Annex 6 installed. Read more in the
+[Git Annex troubleshooting tips](../git_annex.md#troubleshooting-tips) section.
+
+1. Backup your repository
+
+   ```bash
+   cd repository
+   git annex sync --content
+   cd ..
+   git clone repository repository-backup
+   cd repository-backup
+   git annex get
+   cd ..
+   ```
+
+1. Use `annex direct`:
+
+   ```bash
+   cd repository
+   git annex direct
+   ```
+
+   The output should be similar to this:
+
+   ```bash
+   commit
+   On branch master
+   Your branch is up-to-date with 'origin/master'.
+   nothing to commit, working tree clean
+   ok
+   direct debian.iso ok
+   direct  ok
+   ```
+
+1. Disable Git Annex with [`annex uninit`][uninit]:
+
+   ```bash
+   git annex uninit
+   ```
+
+   The output should be similar to this:
+
+   ```bash
+   unannex debian.iso ok
+   Deleted branch git-annex (was 2534d2c).
+   ```
+
+   This will `unannex` every file in the repository, leaving the original files.
+
+1. Switch back to `indirect` mode:
+
+   ```bash
+   git annex indirect
+   ```
+
+   The output should be similar to this:
+
+   ```bash
+   (merging origin/git-annex into git-annex...)
+   (recording state in git...)
+   commit  (recording state in git...)
+
+   ok
+   (recording state in git...)
+   [master fac3194] commit before switching to indirect mode
+    1 file changed, 1 deletion(-)
+    delete mode 120000 alpine-virt-3.4.4-x86_64.iso
+   ok
+   indirect  ok
+   ok
+   ```
+
+---
+
+At this point, you have two options. Either add, commit and push the files
+directly back to GitLab or switch to Git LFS. We will tackle the LFS switch in
+the next section.
+
+### Enabling Git LFS in your repo
+
+Git LFS is enabled by default on all GitLab products (GitLab CE, GitLab EE,
+GitLab.com), therefore, you don't need to do anything server-side.
+
+1. First, make sure you have `git-lfs` installed locally:
+
+   ```bash
+   git lfs help
+   ```
+
+   If the terminal doesn't prompt you with a full response on `git-lfs` commands,
+   [install the Git LFS client][install-lfs] first.
+
+1. Inside the repo, run the following command to initiate LFS:
+
+   ```bash
+   git lfs install
+   ```
+
+1. Enable `git-lfs` for the group of files you want to track. You
+   can track specific files, all files containing the same extension, or an
+   entire directory:
+
+   ```bash
+   git lfs track images/01.png   # per file
+   git lfs track **/*.png        # per extension
+   git lfs track images/         # per directory
+   ```
+
+   Once you do that, run `git status` and you'll see `.gitattributes` added
+   to your repo. It collects all file patterns that you chose to track via
+   `git-lfs`.
+
+1. Add the files, commit and push them to GitLab:
+
+   ```bash
+   git add .
+   git commit -m "commit message"
+   git push
+   ```
+
+   If your remote is set up with HTTP, you will be asked to enter your login
+   credentials. If you have [2FA enabled](../../user/profile/account/two_factor_authentication.md), make sure to use a
+   [personal access token](../../user/profile/account/two_factor_authentication.md#personal-access-tokens)
+   instead of your password.
+
+## Removing the Git Annex branches
+
+After the migration finishes successfully, you can remove all `git-annex`
+related branches from your repository.
+
+On GitLab, navigate to your project's **Repository ➔ Branches** and delete all
+branches created by Git Annex: `git-annex`, and all under `synced/`.
+
+![repository branches](img/git-annex-branches.png)
+
+You can also do this on the command line with:
+
+```bash
+git branch -d synced/master
+git branch -d synced/git-annex
+git push origin :synced/master
+git push origin :synced/git-annex
+git push origin :git-annex
+git remote prune origin
+```
+
+If there are still some Annex objects inside your repository (`.git/annex/`)
+or references inside `.git/config`, run `annex uninit` again:
+
+```bash
+git annex uninit
+```
+
+## Further Reading
+
+- (Blog Post) [Getting Started with Git FLS][post-1]
+- (Blog Post) [Announcing LFS Support in GitLab][post-2]
+- (Blog Post) [GitLab Annex Solves the Problem of Versioning Large Binaries with Git][post-3]
+- (GitLab Docs) [Git Annex](../git_annex.md)
+- (GitLab Docs) [Git LFS](manage_large_binaries_with_git_lfs.md)
+
+[annex-direct]: https://git-annex.branchable.com/direct_mode/
+[bkp-ext-drive]: https://www.thomas-krenn.com/en/wiki/Git-annex_Repository_on_an_External_Hard_Drive
+[Git Annex]: http://git-annex.branchable.com/
+[Git LFS]: https://git-lfs.github.com/
+[install-lfs]: https://git-lfs.github.com/
+[issue-remove-annex]: https://gitlab.com/gitlab-org/gitlab/issues/1648
+[lfs-track]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/#tracking-files-with-lfs
+[post-1]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/
+[post-2]: https://about.gitlab.com/blog/2015/11/23/announcing-git-lfs-support-in-gitlab/
+[post-3]: https://about.gitlab.com/blog/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/
+[uninit]: https://git-annex.branchable.com/git-annex-uninit/

doc/administration/repository_storage_types.md

@@ -183,7 +183,8 @@ CI Artifacts are S3 compatible since **9.4** (GitLab Premium), and available in
 ##### LFS Objects
-LFS Objects implements a similar storage pattern using 2 chars, 2 level folders, following Git own implementation:
+[LFS Objects in GitLab](lfs/manage_large_binaries_with_git_lfs.md) implement a similar
+storage pattern using 2 chars, 2 level folders, following Git's own implementation:
 ```ruby
 "shared/lfs-objects/#{oid[0..1}/#{oid[2..3]}/#{oid[4..-1]}"
@@ -192,7 +193,7 @@ LFS Objects implements a similar storage pattern using 2 chars, 2 level folders,
 "shared/lfs-objects/89/09/029eb962194cfb326259411b22ae3f4a814b5be4f80651735aeef9f3229c"
 ```
-They are also S3 compatible since **10.0** (GitLab Premium), and available in GitLab Core since **10.7**.
+LFS objects are also [S3 compatible](lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage).
 [ce-2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821
 [ce-28283]: https://gitlab.com/gitlab-org/gitlab-foss/issues/28283

doc/ci/ci_cd_for_external_repos/bitbucket_integration.md

@@ -151,7 +151,7 @@ To use GitLab CI/CD with a Bitbucket Cloud repository:
 GitLab is now configured to mirror changes from Bitbucket, run CI/CD pipelines
 configured in `.gitlab-ci.yml` and push the status to Bitbucket.
-[pull-mirroring]: ../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter
+[pull-mirroring]: ../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter
 <!-- ## Troubleshooting

doc/ci/ci_cd_for_external_repos/github_integration.md

@@ -46,7 +46,7 @@ repositories:
 GitLab will:
 1. Import the project.
-1. Enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter)
+1. Enable [Pull Mirroring](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter)
 1. Enable [GitHub project integration](../../user/project/integrations/github.md)
 1. Create a web hook on GitHub to notify GitLab of new commits.

doc/ci/ci_cd_for_external_repos/index.md

@@ -101,5 +101,5 @@ requests and not on branches you can add `except: [branches]` to the job specs.
 [ee-4642]: https://gitlab.com/gitlab-org/gitlab/merge_requests/4642
 [eep]: https://about.gitlab.com/pricing/
-[mirroring]: ../../workflow/repository_mirroring.md
+[mirroring]: ../../user/project/repository/repository_mirroring.md
 [settings]: ../../user/project/settings/index.md#sharing-and-permissions

doc/ci/environments.md

@@ -292,7 +292,7 @@ For the value of:
   the web server to serve these requests is based on your setup.
   We have used `$CI_ENVIRONMENT_SLUG` here because it is guaranteed to be unique. If
-  you're using a workflow like [GitLab Flow](../workflow/gitlab_flow.md), collisions
+  you're using a workflow like [GitLab Flow](../topics/gitlab_flow.md), collisions
   are unlikely and you may prefer environment names to be more closely based on the
   branch name. In that case, you could use `$CI_COMMIT_REF_SLUG` in `environment:url` in
   the example above: `https://$CI_COMMIT_REF_SLUG.example.com`, which would give a URL

doc/ci/examples/laravel_with_gitlab_and_envoy/index.md

@@ -379,7 +379,7 @@ These are persistent data and will be shared to every new release.
 Now, we would need to deploy our app by running `envoy run deploy`, but it won't be necessary since GitLab can handle that for us with CI's [environments](../../environments.md), which will be described [later](#setting-up-gitlab-cicd) in this tutorial.
 Now it's time to commit [Envoy.blade.php](https://gitlab.com/mehranrasulian/laravel-sample/blob/master/Envoy.blade.php) and push it to the `master` branch.
-To keep things simple, we commit directly to `master`, without using [feature-branches](../../../workflow/gitlab_flow.md#github-flow-as-a-simpler-alternative) since collaboration is beyond the scope of this tutorial.
+To keep things simple, we commit directly to `master`, without using [feature-branches](../../../topics/gitlab_flow.md#github-flow-as-a-simpler-alternative) since collaboration is beyond the scope of this tutorial.
 In a real world project, teams may use [Issue Tracker](../../../user/project/issues/index.md) and [Merge Requests](../../../user/project/merge_requests/index.md) to move their code across branches:
 ```bash

doc/ci/pipelines.md

@@ -28,7 +28,7 @@ If all the jobs in a stage:
 - Fail, the next stage is not (usually) executed and the pipeline ends early.
 NOTE: **Note:**
-If you have a [mirrored repository that GitLab pulls from](../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter),
+If you have a [mirrored repository that GitLab pulls from](../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter),
 you may need to enable pipeline triggering in your project's
 **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**.

doc/ci/quick_start/README.md

@@ -143,7 +143,7 @@ Now if you go to the **Pipelines** page you will see that the pipeline is
 pending.
 NOTE: **Note:**
-If you have a [mirrored repository where GitLab pulls from](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter),
+If you have a [mirrored repository where GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter),
 you may need to enable pipeline triggering in your project's
 **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**.

doc/ci/yaml/README.md

@@ -23,7 +23,7 @@ We have complete examples of configuring pipelines:
 - To see a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab`](https://gitlab.com/gitlab-org/gitlab/blob/master/.gitlab-ci.yml).
 NOTE: **Note:**
-If you have a [mirrored repository where GitLab pulls from](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter),
+If you have a [mirrored repository where GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository-starter),
 you may need to enable pipeline triggering in your project's
 **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**.

doc/development/contributing/merge_request_workflow.md

@@ -36,7 +36,7 @@ include a regression test are merged quickly, while new features without proper
 tests might be slower to receive feedback. The workflow to make a merge
 request is as follows:
-1. [Fork](../../workflow/forking_workflow.md#creating-a-fork) the project into
+1. [Fork](../../user/project/repository/forking_workflow.md) the project into
    your personal namespace (or group) on GitLab.com.
 1. Create a feature branch in your fork (don't work off `master`).
 1. Write [tests](../rake_tasks.md#run-tests) and code.

doc/development/git_object_deduplication.md

 # How Git object deduplication works in GitLab
-When a GitLab user [forks a project](../workflow/forking_workflow.md),
+When a GitLab user [forks a project](../user/project/repository/forking_workflow.md),
 GitLab creates a new Project with an associated Git repository that is a
 copy of the original project at the time of the fork. If a large project
 gets forked often, this can lead to a quick increase in Git repository