Merge remote-tracking branch 'ce/master' into ce-to-ee

changelogs/unreleased/remove-heading-space-from-diff-content.yml

+---
+title: Remove an extra leading space from diff paste data
+merge_request: 7133
+author: Hiroyuki Sato

changelogs/unreleased/upgrade-timeago.yml

+---
+title: Replace jQuery.timeago with timeago.js
+merge_request: 6274
+author: ClemMakesApps

doc/README.md

@@ -25,6 +25,7 @@
 - [Workflow](workflow/README.md) Using GitLab functionality and importing projects from GitHub and SVN.
 - [University](university/README.md) Learn Git and GitLab through videos and courses.
 - [Git Attributes](user/project/git_attributes.md) Managing Git attributes using a `.gitattributes` file.
+- [Git cheatsheet](https://gitlab.com/gitlab-com/marketing/raw/master/design/print/git-cheatsheet/print-pdf/git-cheatsheet.pdf) Download a PDF describing the most used Git operations.
 ## Administrator documentation

doc/administration/auth/ldap.md

@@ -40,6 +40,10 @@ of one hour.
 To enable LDAP integration you need to add your LDAP server settings in
 `/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml`.
+There is a Rake task to check LDAP configuration. After configuring LDAP
+using the documentation below, see [LDAP check Rake task](../raketasks/check.md#ldap-check)
+for information on the LDAP check Rake task.
+
 >**Note**: In GitLab EE, you can configure multiple LDAP servers to connect to
 one GitLab server.

doc/administration/raketasks/check.md

+# Check Rake Tasks
+
+## Repository Integrity
+
+Even though Git is very resilient and tries to prevent data integrity issues,
+there are times when things go wrong. The following Rake tasks intend to
+help GitLab administrators diagnose problem repositories so they can be fixed.
+
+There are 3 things that are checked to determine integrity.
+
+1. Git repository file system check ([git fsck](https://git-scm.com/docs/git-fsck)).
+   This step verifies the connectivity and validity of objects in the repository.
+1. Check for `config.lock` in the repository directory.
+1. Check for any branch/references lock files in `refs/heads`.
+
+It's important to note that the existence of `config.lock` or reference locks
+alone do not necessarily indicate a problem. Lock files are routinely created
+and removed as Git and GitLab perform operations on the repository. They serve
+to prevent data integrity issues. However, if a Git operation is interrupted these
+locks may not be cleaned up properly.
+
+The following symptoms may indicate a problem with repository integrity. If users
+experience these symptoms you may use the rake tasks described below to determine
+exactly which repositories are causing the trouble.
+
+- Receiving an error when trying to push code - `remote: error: cannot lock ref`
+- A 500 error when viewing the GitLab dashboard or when accessing a specific project.
+
+### Check all GitLab repositories
+
+This task loops through all repositories on the GitLab server and runs the
+3 integrity checks described previously.
+
+**Omnibus Installation**
+
+```
+sudo gitlab-rake gitlab:repo:check
+```
+
+**Source Installation**
+
+```bash
+sudo -u git -H bundle exec rake gitlab:repo:check RAILS_ENV=production
+```
+
+### Check repositories for a specific user
+
+This task checks all repositories that a specific user has access to. This is important
+because sometimes you know which user is experiencing trouble but you don't know
+which project might be the cause.
+
+If the rake task is executed without brackets at the end, you will be prompted
+to enter a username.
+
+**Omnibus Installation**
+
+```bash
+sudo gitlab-rake gitlab:user:check_repos
+sudo gitlab-rake gitlab:user:check_repos[<username>]
+```
+
+**Source Installation**
+
+```bash
+sudo -u git -H bundle exec rake gitlab:user:check_repos RAILS_ENV=production
+sudo -u git -H bundle exec rake gitlab:user:check_repos[<username>] RAILS_ENV=production
+```
+
+Example output:
+
+![gitlab:user:check_repos output](../img/raketasks/check_repos_output.png)
+
+## LDAP Check
+
+The LDAP check Rake task will test the bind_dn and password credentials
+(if configured) and will list a sample of LDAP users. This task is also
+executed as part of the `gitlab:check` task, but can run independently
+using the command below.
+
+**Omnibus Installation**
+
+```
+sudo gitlab-rake gitlab:ldap:check
+```
+
+**Source Installation**
+
+```bash
+sudo -u git -H bundle exec rake gitlab:ldap:check RAILS_ENV=production
+```
+
+By default, the task will return a sample of 100 LDAP users. Change this
+limit by passing a number to the check task:
+
+```bash
+rake gitlab:ldap:check[50]
+```

doc/api/notification_settings.md

@@ -4,7 +4,7 @@
 **Valid notification levels**
-The notification levels are defined in the `NotificationSetting::level` model enumeration. Currently, these levels are recognized:
+The notification levels are defined in the `NotificationSetting.level` model enumeration. Currently, these levels are recognized:
 ```
 disabled
@@ -28,6 +28,8 @@ reopen_merge_request
 close_merge_request
 reassign_merge_request
 merge_merge_request
+failed_pipeline
+success_pipeline
 ```
 ## Global notification settings
@@ -77,6 +79,8 @@ curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab
 | `close_merge_request` | boolean | no | Enable/disable this notification |
 | `reassign_merge_request` | boolean | no | Enable/disable this notification |
 | `merge_merge_request` | boolean | no | Enable/disable this notification |
+| `failed_pipeline` | boolean | no | Enable/disable this notification |
+| `success_pipeline` | boolean | no | Enable/disable this notification |
 Example response:
@@ -141,6 +145,8 @@ curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab
 | `close_merge_request` | boolean | no | Enable/disable this notification |
 | `reassign_merge_request` | boolean | no | Enable/disable this notification |
 | `merge_merge_request` | boolean | no | Enable/disable this notification |
+| `failed_pipeline` | boolean | no | Enable/disable this notification |
+| `success_pipeline` | boolean | no | Enable/disable this notification |
 Example responses:
@@ -161,7 +167,9 @@ Example responses:
     "reopen_merge_request": false,
     "close_merge_request": false,
     "reassign_merge_request": false,
-    "merge_merge_request": false
+    "merge_merge_request": false,
+    "failed_pipeline": false,
+    "success_pipeline": false
   }
 }
 ```

doc/api/users.md

@@ -35,6 +35,18 @@ GET /users
 ]
 ```
+In addition, you can filter users based on states eg. `blocked`, `active`
+This works only to filter users who are `blocked` or `active`.
+It does not support `active=false` or `blocked=false`.
+
+```
+GET /users?active=true
+```
+
+```
+GET /users?blocked=true
+```
+
 ### For admins
 ```
@@ -122,6 +134,8 @@ For example:
 GET /users?username=jack_smith
 ```
+You can search for users who are external with: `/users?external=true`
+
 ## Single user
 Get a single user.

doc/ci/docker/using_docker_build.md

@@ -44,7 +44,8 @@ GitLab Runner then executes build scripts as the `gitlab-runner` user.
 2. Install Docker Engine on server.
-    For more information how to install Docker Engine on different systems checkout the [Supported installations](https://docs.docker.com/engine/installation/).
+    For more information how to install Docker Engine on different systems
+    checkout the [Supported installations](https://docs.docker.com/engine/installation/).
 3. Add `gitlab-runner` user to `docker` group:
@@ -122,11 +123,17 @@ In order to do that, follow the steps:
         Insecure = false
     ```
-1. You can now use `docker` in the build script (note the inclusion of the `docker:dind` service):
+1. You can now use `docker` in the build script (note the inclusion of the
+   `docker:dind` service):
     ```yaml
     image: docker:latest
+    # When using dind, it's wise to use the overlayfs driver for
+    # improved performance.
+    variables:
+      DOCKER_DRIVER: overlay
+
     services:
     - docker:dind
@@ -140,15 +147,21 @@ In order to do that, follow the steps:
       - docker run my-docker-image /script/to/run/tests
     ```
-Docker-in-Docker works well, and is the recommended configuration, but it is not without its own challenges:
-* By enabling `--docker-privileged`, you are effectively disabling all of
-the security mechanisms of containers and exposing your host to privilege
-escalation which can lead to container breakout. For more information, check out the official Docker documentation on
-[Runtime privilege and Linux capabilities][docker-cap].
-* Using docker-in-docker, each build is in a clean environment without the past
-history. Concurrent builds work fine because every build gets it's own instance of docker engine so they won't conflict with each other. But this also means builds can be slower because there's no caching of layers.
-* By default, `docker:dind` uses `--storage-driver vfs` which is the slowest form
-offered.
+Docker-in-Docker works well, and is the recommended configuration, but it is
+not without its own challenges:
+
+- By enabling `--docker-privileged`, you are effectively disabling all of
+  the security mechanisms of containers and exposing your host to privilege
+  escalation which can lead to container breakout. For more information, check
+  out the official Docker documentation on
+  [Runtime privilege and Linux capabilities][docker-cap].
+- Using docker-in-docker, each build is in a clean environment without the past
+  history. Concurrent builds work fine because every build gets it's own
+  instance of Docker engine so they won't conflict with each other. But this
+  also means builds can be slower because there's no caching of layers.
+- By default, `docker:dind` uses `--storage-driver vfs` which is the slowest
+  form offered. To use a different driver, see
+  [Using the overlayfs driver](#using-the-overlayfs-driver).
 An example project using this approach can be found here: https://gitlab.com/gitlab-examples/docker.
@@ -221,6 +234,40 @@ work as expected since volume mounting is done in the context of the host
 machine, not the build container.
 e.g. `docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests`
+## Using the OverlayFS driver
+
+By default, when using `docker:dind`, Docker uses the `vfs` storage driver which
+copies the filesystem on every run. This is a very disk-intensive operation
+which can be avoided if a different driver is used, for example `overlay`.
+
+1. Make sure a recent kernel is used, preferably `>= 4.2`.
+1. Check whether the `overlay` module is loaded:
+
+    ```
+    sudo lsmod | grep overlay
+    ```
+
+    If you see no result, then it isn't loaded. To load it use:
+
+    ```
+    sudo modprobe overlay
+    ```
+
+    If everything went fine, you need to make sure module is loaded on reboot.
+    On Ubuntu systems, this is done by editing `/etc/modules`. Just add the
+    following line into it:
+
+    ```
+    overlay
+    ```
+
+1. Use the driver by defining a variable at the top of your `.gitlab-ci.yml`:
+
+    ```
+    variables:
+      DOCKER_DRIVER: overlay
+    ```
+
 ## Using the GitLab Container Registry
 > **Note:**

doc/development/instrumentation.md

@@ -24,7 +24,7 @@ namespace you can use the `configure` class method. This method simply yields
 the supplied block while passing `Gitlab::Metrics::Instrumentation` as its
 argument. An example:
-```
+```ruby
 Gitlab::Metrics::Instrumentation.configure do |conf|
   conf.instrument_method(Foo, :bar)
   conf.instrument_method(Foo, :baz)
@@ -41,7 +41,7 @@ Method instrumentation should be added in the initializer
 Instrumenting a single method:
-```
+```ruby
 Gitlab::Metrics::Instrumentation.configure do |conf|
   conf.instrument_method(User, :find_by)
 end
@@ -49,7 +49,7 @@ end
 Instrumenting an entire class hierarchy:
-```
+```ruby
 Gitlab::Metrics::Instrumentation.configure do |conf|
   conf.instrument_class_hierarchy(ActiveRecord::Base)
 end
@@ -57,7 +57,7 @@ end
 Instrumenting all public class methods:
-```
+```ruby
 Gitlab::Metrics::Instrumentation.configure do |conf|
   conf.instrument_methods(User)
 end
@@ -68,7 +68,7 @@ end
 The easiest way to check if a method has been instrumented is to check its
 source location. For example:
-```
+```ruby
 method = Rugged::TagCollection.instance_method(:[])
 method.source_location

doc/development/migration_style_guide.md

@@ -60,7 +60,7 @@ migration was tested.
 If you need to remove index, please add a condition like in following example:
-```
+```ruby
 remove_index :namespaces, column: :name if index_exists?(:namespaces, :name)
 ```
@@ -75,7 +75,7 @@ need for downtime. To use this method you must disable transactions by calling
 the method `disable_ddl_transaction!` in the body of your migration class like
 so:
-```
+```ruby
 class MyMigration < ActiveRecord::Migration
   include Gitlab::Database::MigrationHelpers
   disable_ddl_transaction!
@@ -96,7 +96,7 @@ the `up` and `down` methods in your migration class.
 For example, to add the column `foo` to the `projects` table with a default
 value of `10` you'd write the following:
-```
+```ruby
 class MyMigration < ActiveRecord::Migration
   include Gitlab::Database::MigrationHelpers
   disable_ddl_transaction!
@@ -125,7 +125,7 @@ set the limit to 8-bytes. This will allow the column to hold a value up to
 Rails migration example:
-```
+```ruby
 add_column_with_default(:projects, :foo, :integer, default: 10, limit: 8)
 # or
@@ -145,7 +145,7 @@ Please prefer Arel and plain SQL over usual ActiveRecord syntax. In case of usin
 Example with Arel:
-```
+```ruby
 users = Arel::Table.new(:users)
 users.group(users[:user_id]).having(users[:id].count.gt(5))
@@ -154,7 +154,7 @@ users.group(users[:user_id]).having(users[:id].count.gt(5))
 Example with plain SQL and `quote_string` helper:
-```
+```ruby
 select_all("SELECT name, COUNT(id) as cnt FROM tags GROUP BY name HAVING COUNT(id) > 1").each do |tag|
   tag_name = quote_string(tag["name"])
   duplicate_ids = select_all("SELECT id FROM tags WHERE name = '#{tag_name}'").map{|tag| tag["id"]}

doc/development/shell_commands.md

@@ -129,7 +129,7 @@ Various methods for opening and reading files in Ruby can be used to read the
 standard output of a process instead of a file.  The following two commands do
 roughly the same:
-```
+```ruby
 `touch /tmp/pawned-by-backticks`
 File.read('|touch /tmp/pawned-by-file-read')
 ```
@@ -142,7 +142,7 @@ attacker cannot control the start of the filename string you are opening.  For
 instance, the following is sufficient to protect against accidentally starting
 a shell command with `|`:
-```
+```ruby
 # we assume repo_path is not controlled by the attacker (user)
 path = File.join(repo_path, user_input)
 # path cannot start with '|' now.
@@ -160,7 +160,7 @@ Path traversal is a security where the program (GitLab) tries to restrict user
 access to a certain directory on disk, but the user manages to open a file
 outside that directory by taking advantage of the `../` path notation.
-```
+```ruby
 # Suppose the user gave us a path and they are trying to trick us
 user_input = '../other-repo.git/other-file'
@@ -177,7 +177,7 @@ File.open(full_path) do # Oops!
 A good way to protect against this is to compare the full path with its
 'absolute path' according to Ruby's `File.absolute_path`.
-```
+```ruby
 full_path = File.join(repo_path, user_input)
 if full_path != File.absolute_path(full_path)
   raise "Invalid path: #{full_path.inspect}"

doc/raketasks/check.md

 # Check Rake Tasks
-## Repository Integrity
-
-Even though Git is very resilient and tries to prevent data integrity issues,
-there are times when things go wrong. The following Rake tasks intend to
-help GitLab administrators diagnose problem repositories so they can be fixed.
-
-There are 3 things that are checked to determine integrity.
-
-1. Git repository file system check ([git fsck](https://git-scm.com/docs/git-fsck)).
-   This step verifies the connectivity and validity of objects in the repository.
-1. Check for `config.lock` in the repository directory.
-1. Check for any branch/references lock files in `refs/heads`.
-
-It's important to note that the existence of `config.lock` or reference locks
-alone do not necessarily indicate a problem. Lock files are routinely created
-and removed as Git and GitLab perform operations on the repository. They serve
-to prevent data integrity issues. However, if a Git operation is interrupted these
-locks may not be cleaned up properly.
-
-The following symptoms may indicate a problem with repository integrity. If users
-experience these symptoms you may use the rake tasks described below to determine
-exactly which repositories are causing the trouble.
-
-- Receiving an error when trying to push code - `remote: error: cannot lock ref`
-- A 500 error when viewing the GitLab dashboard or when accessing a specific project.
-
-### Check all GitLab repositories
-
-This task loops through all repositories on the GitLab server and runs the
-3 integrity checks described previously.
-
-```
-# omnibus-gitlab
-sudo gitlab-rake gitlab:repo:check
-
-# installation from source
-bundle exec rake gitlab:repo:check RAILS_ENV=production
-```
-
-### Check repositories for a specific user
-
-This task checks all repositories that a specific user has access to. This is important
-because sometimes you know which user is experiencing trouble but you don't know
-which project might be the cause.
-
-If the rake task is executed without brackets at the end, you will be prompted
-to enter a username.
-
-```bash
-# omnibus-gitlab
-sudo gitlab-rake gitlab:user:check_repos
-sudo gitlab-rake gitlab:user:check_repos[<username>]
-
-# installation from source
-bundle exec rake gitlab:user:check_repos RAILS_ENV=production
-bundle exec rake gitlab:user:check_repos[<username>] RAILS_ENV=production
-```
-
-Example output:
-
-![gitlab:user:check_repos output](check_repos_output.png)
+This document was moved to [administration/raketasks/check](../administration/raketasks/check.md).

doc/workflow/notifications.md

@@ -66,6 +66,7 @@ Below is the table of events users can be notified of:
 In all of the below cases, the notification will be sent to:
 - Participants:
   - the author and assignee of the issue/merge request
+  - the author of the pipeline
   - authors of comments on the issue/merge request
   - anyone mentioned by `@username` in the issue/merge request title or description
   - anyone mentioned by `@username` in any of the comments on the issue/merge request
@@ -88,6 +89,8 @@ In all of the below cases, the notification will be sent to:
 | Reopen merge request   | |
 | Merge merge request    | |
 | New comment            | The above, plus anyone mentioned by `@username` in the comment, with notification level "Mention" or higher |
+| Failed pipeline        | The above, plus the author of the pipeline |
+| Successful pipeline    | The above, plus the author of the pipeline |
 In addition, if the title or description of an Issue or Merge Request is

lib/api/users.rb

@@ -10,6 +10,9 @@ module API
       #  GET /users
       #  GET /users?search=Admin
       #  GET /users?username=root
+      #  GET /users?active=true
+      #  GET /users?external=true
+      #  GET /users?blocked=true
       get do
         unless can?(current_user, :read_users_list, nil)
           render_api_error!("Not authorized.", 403)
@@ -20,9 +23,15 @@ module API
         else
           skip_ldap = params[:skip_ldap].present? && params[:skip_ldap] == 'true'
           @users = User.all
+<<<<<<< HEAD
           @users = @users.active if params[:active].present?
           @users = @users.non_ldap if skip_ldap
+=======
+          @users = @users.active if to_boolean(params[:active])
+>>>>>>> ce/master
           @users = @users.search(params[:search]) if params[:search].present?
+          @users = @users.blocked if to_boolean(params[:blocked])
+          @users = @users.external if to_boolean(params[:external]) && current_user.is_admin?
           @users = paginate @users
         end

lib/banzai/filter/autolink_filter.rb

@@ -71,6 +71,14 @@ module Banzai
         @doc = parse_html(rinku)
       end
+      # Return true if any of the UNSAFE_PROTOCOLS strings are included in the URI scheme
+      def contains_unsafe?(scheme)
+        return false unless scheme
+
+        scheme = scheme.strip.downcase
+        Banzai::Filter::SanitizationFilter::UNSAFE_PROTOCOLS.any? { |protocol| scheme.include?(protocol) }
+      end
+
       # Autolinks any text matching LINK_PATTERN that Rinku didn't already
       # replace
       def text_parse
@@ -89,17 +97,27 @@ module Banzai
         doc
       end
-      def autolink_filter(text)
-        text.gsub(LINK_PATTERN) do |match|
-          # Remove any trailing HTML entities and store them for appending
-          # outside the link element. The entity must be marked HTML safe in
-          # order to be output literally rather than escaped.
-          match.gsub!(/((?:&[\w#]+;)+)\z/, '')
-          dropped = ($1 || '').html_safe
-
-          options = link_options.merge(href: match)
-          content_tag(:a, match, options) + dropped
+      def autolink_match(match)
+        # start by stripping out dangerous links
+        begin
+          uri = Addressable::URI.parse(match)
+          return match if contains_unsafe?(uri.scheme)
+        rescue Addressable::URI::InvalidURIError
+          return match
         end
+
+        # Remove any trailing HTML entities and store them for appending
+        # outside the link element. The entity must be marked HTML safe in
+        # order to be output literally rather than escaped.
+        match.gsub!(/((?:&[\w#]+;)+)\z/, '')
+        dropped = ($1 || '').html_safe
+
+        options = link_options.merge(href: match)
+        content_tag(:a, match, options) + dropped
+      end
+
+      def autolink_filter(text)
+        text.gsub(LINK_PATTERN) { |match| autolink_match(match) }
       end
       def link_options

lib/banzai/reference_parser/base_parser.rb

@@ -63,12 +63,7 @@ module Banzai
         nodes.select do |node|
           if node.has_attribute?(project_attr)
             node_id = node.attr(project_attr).to_i
-
-            if project && project.id == node_id
-              true
-            else
-              can?(user, :read_project, projects[node_id])
-            end
+            can_read_reference?(user, projects[node_id])
           else
             true
           end
@@ -226,6 +221,15 @@ module Banzai
       attr_reader :current_user, :project
+      # When a feature is disabled or visible only for
+      # team members we should not allow team members
+      # see reference comments.
+      # Override this method on subclasses
+      # to check if user can read resource
+      def can_read_reference?(user, ref_project)
+        raise NotImplementedError
+      end
+
       def lazy(&block)
         Gitlab::Lazy.new(&block)
       end

lib/banzai/reference_parser/commit_parser.rb

@@ -29,6 +29,12 @@ module Banzai
         commits
       end
+
+      private
+
+      def can_read_reference?(user, ref_project)
+        can?(user, :download_code, ref_project)
+      end
     end
   end
 end

lib/banzai/reference_parser/commit_range_parser.rb

@@ -33,6 +33,12 @@ module Banzai
         range.valid_commits? ? range : nil
       end
+
+      private
+
+      def can_read_reference?(user, ref_project)
+        can?(user, :download_code, ref_project)
+      end
     end
   end
 end

lib/banzai/reference_parser/external_issue_parser.rb

@@ -20,6 +20,12 @@ module Banzai
       def issue_ids_per_project(nodes)
         gather_attributes_per_project(nodes, self.class.data_attribute)
       end
+
+      private
+
+      def can_read_reference?(user, ref_project)
+        can?(user, :read_issue, ref_project)
+      end
     end
   end
 end