Merge branch 'master' into 43404-pipelines-commit

* master: (124 commits)
  Double-check next value for internal ids.
  Resolve "skeleton placeholder on diff has white background"
  Replace the `project/commits/comments.feature` spinach test with an rspec analog
  Set ENV['IN_MEMORY_APPLICATION_SETTINGS'] to 'true in spec/db/production/settings_spec.rb
  Update docs on `.gitlab-ci.yml` and variables policy
  Add note about support of variables policy to docs
  Revert and fix
  corrected git cheat sheet url
  Bump parser gem to 2.5.1.0 to remove warning in rspec
  Update invalid test application_settings_controller_spec.rb
  Use stable docker image instead of latest
  Handle cancelled request
  Stop polling and fetch the table
  [ci skip] Updates table when stage is clicked
  Backport GitLab domain helper methods
  use jquery object in togglePopover
  Fix other listeners
  Fix the metrics link being not rendering the html
  Fix feature_highlight listener
  Update custom issue tracker docs
  ...

doc/README.md

@@ -183,7 +183,7 @@ instant how code changes impact your production environment.
 ### Git and GitLab
 - [Git](topics/git/index.md): Getting started with Git, branching strategies, Git LFS, advanced use.
-- [Git cheatsheet](https://gitlab.com/gitlab-com/marketing/raw/master/design/print/git-cheatsheet/print-pdf/git-cheatsheet.pdf): Download a PDF describing the most used Git operations.
+- [Git cheatsheet](https://about.gitlab.com/images/press/git-cheat-sheet.pdf): Download a PDF describing the most used Git operations.
 - [GitLab Flow](workflow/gitlab_flow.md): explore the best of Git with the GitLab Flow strategy.
 ## Administrator documentation

doc/api/todos.md

@@ -15,7 +15,7 @@ Parameters:
 | Attribute | Type | Required | Description |
 | --------- | ---- | -------- | ----------- |
-| `action` | string | no | The action to be filtered. Can be `assigned`, `mentioned`, `build_failed`, `marked`, or `approval_required`. |
+| `action` | string | no | The action to be filtered. Can be `assigned`, `mentioned`, `build_failed`, `marked`, `approval_required`, `unmergeable` or `directly_addressed`. |
 | `author_id` | integer | no | The ID of an author |
 | `project_id` | integer | no | The ID of a project |
 | `state` | string | no | The state of the todo. Can be either `pending` or `done` |

doc/ci/docker/using_docker_build.md

@@ -101,12 +101,12 @@ In order to do that, follow the steps:
       --registration-token REGISTRATION_TOKEN \
       --executor docker \
       --description "My Docker Runner" \
-      --docker-image "docker:latest" \
+      --docker-image "docker:stable" \
       --docker-privileged
     ```
     The above command will register a new Runner to use the special
-    `docker:latest` image which is provided by Docker. **Notice that it's using
+    `docker:stable` image which is provided by Docker. **Notice that it's using
     the `privileged` mode to start the build and service containers.** If you
     want to use [docker-in-docker] mode, you always have to use `privileged = true`
     in your Docker containers.
@@ -120,7 +120,7 @@ In order to do that, follow the steps:
       executor = "docker"
       [runners.docker]
         tls_verify = false
-        image = "docker:latest"
+        image = "docker:stable"
         privileged = true
         disable_cache = false
         volumes = ["/cache"]
@@ -132,7 +132,7 @@ In order to do that, follow the steps:
    `docker:dind` service):
     ```yaml
-    image: docker:latest
+    image: docker:stable
     # When using dind, it's wise to use the overlayfs driver for
     # improved performance.
@@ -201,12 +201,12 @@ In order to do that, follow the steps:
       --registration-token REGISTRATION_TOKEN \
       --executor docker \
       --description "My Docker Runner" \
-      --docker-image "docker:latest" \
+      --docker-image "docker:stable" \
       --docker-volumes /var/run/docker.sock:/var/run/docker.sock
     ```
     The above command will register a new Runner to use the special
-    `docker:latest` image which is provided by Docker. **Notice that it's using
+    `docker:stable` image which is provided by Docker. **Notice that it's using
     the Docker daemon of the Runner itself, and any containers spawned by docker
     commands will be siblings of the Runner rather than children of the runner.**
     This may have complications and limitations that are unsuitable for your workflow.
@@ -220,7 +220,7 @@ In order to do that, follow the steps:
       executor = "docker"
       [runners.docker]
         tls_verify = false
-        image = "docker:latest"
+        image = "docker:stable"
         privileged = false
         disable_cache = false
         volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
@@ -232,7 +232,7 @@ In order to do that, follow the steps:
    include the `docker:dind` service as when using the Docker in Docker executor):
     ```yaml
-    image: docker:latest
+    image: docker:stable
     before_script:
     - docker info
@@ -286,7 +286,7 @@ any image that's used with the `--cache-from` argument must first be pulled
 Here's a simple `.gitlab-ci.yml` file showing how Docker caching can be utilized:
 ```yaml
-image: docker:latest
+image: docker:stable
 services:
   - docker:dind
@@ -388,7 +388,7 @@ could look like:
 ```yaml
  build:
-   image: docker:latest
+   image: docker:stable
    services:
    - docker:dind
    stage: build
@@ -434,7 +434,7 @@ when needed. Changes to `master` also get tagged as `latest` and deployed using
 an application-specific deploy script:
 ```yaml
-image: docker:latest
+image: docker:stable
 services:
 - docker:dind

doc/ci/docker/using_docker_images.md

@@ -86,7 +86,7 @@ services](#accessing-the-services).
 ### How the health check of services works
 Services are designed to provide additional functionality which is **network accessible**.
-It may be a database like MySQL, or Redis, and even `docker:dind` which
+It may be a database like MySQL, or Redis, and even `docker:stable-dind` which
 allows you to use Docker in Docker. It can be practically anything that is
 required for the CI/CD job to proceed and is accessed by network.

doc/ci/examples/browser_performance.md

@@ -17,7 +17,7 @@ performance:
   variables:
     URL: https://example.com
   services:
-    - docker:dind
+    - docker:stable-dind
   script:
     - mkdir gitlab-exporter
     - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/master/index.js
@@ -94,7 +94,7 @@ performance:
   stage: performance
   image: docker:git
   services:
-    - docker:dind
+    - docker:stable-dind
   dependencies:
     - review
   script:

doc/ci/yaml/README.md

@@ -308,7 +308,9 @@ except master.
 ## `only` and `except` (complex)
-> Introduced in GitLab 10.0
+> `refs` and `kubernetes` policies introduced in GitLab 10.0
+
+> `variables` policy introduced in 10.7
 CAUTION: **Warning:**
 This an _alpha_ feature, and it it subject to change at any time without

doc/install/kubernetes/gitlab_runner_chart.md

@@ -80,7 +80,7 @@ runners:
   image: ubuntu:16.04
   ## Run all containers with the privileged flag enabled
-  ## This will allow the docker:dind image to run if you need to run Docker
+  ## This will allow the docker:stable-dind image to run if you need to run Docker
   ## commands. Please read the docs before turning this on:
   ## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-dind
   ##
@@ -147,7 +147,7 @@ enable privileged mode in `values.yaml`:
 ```yaml
 runners:
   ## Run all containers with the privileged flag enabled
-  ## This will allow the docker:dind image to run if you need to run Docker
+  ## This will allow the docker:stable-dind image to run if you need to run Docker
   ## commands. Please read the docs before turning this on:
   ## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-dind
   ##

doc/user/project/integrations/custom_issue_tracker.md

@@ -15,8 +15,8 @@ in the table below.
 Once you have configured and enabled Custom Issue Tracker Service you'll see a link on the GitLab project pages that takes you to that custom issue tracker.
-
 ## Referencing issues
-Issues are referenced with `#<ID>`, where `<ID>` is a number (example `#143`). 
-So with the example above, `#143` would refer to `https://customissuetracker.com/project-name/143`.
\ No newline at end of file
+- Issues are referenced with `ANYTHING-<ID>`, where `ANYTHING` can be any string and `<ID>` is a number used in the target project of the custom integration (example `PROJECT-143`). 
+- `ANYTHING` is a placeholder to differentiate against GitLab issues, which are referenced with `#<ID>`. You can use a project name or project key to replace it for example.
+- So with the example above, `PROJECT-143` would refer to `https://customissuetracker.com/project-name/143`.
\ No newline at end of file

features/project/commits/comments.feature

-@project_commits
-Feature: Project Commits Comments
-  Background:
-    Given I sign in as a user
-    And I own project "Shop"
-    And I visit project commit page
-
-  @javascript
-  Scenario: I can comment on a commit
-    Given I leave a comment like "XML attached"
-    Then I should see a comment saying "XML attached"
-
-  @javascript
-  Scenario: I can't cancel the main form
-    Then I should not see the cancel comment button
-
-  @javascript
-  Scenario: I can preview with text
-    Given I write a comment like ":+1: Nice"
-    Then The comment preview tab should be display rendered Markdown
-
-  @javascript
-  Scenario: I preview a comment
-    Given I preview a comment text like "Bug fixed :smile:"
-    Then I should see the comment preview
-    And I should not see the comment text field
-
-  @javascript
-  Scenario: I can edit after preview
-    Given I preview a comment text like "Bug fixed :smile:"
-    Then I should see the comment write tab
-
-  @javascript
-  Scenario: I have a reset form after posting from preview
-    Given I preview a comment text like "Bug fixed :smile:"
-    And I submit the comment
-    Then I should see an empty comment text field
-    And I should not see the comment preview
-
-  @javascript
-  Scenario: I can delete a comment
-    Given I leave a comment like "XML attached"
-    Then I should see a comment saying "XML attached"
-    And I delete a comment
-    Then I should not see a comment saying "XML attached"
-
-  @javascript
-  Scenario: I can edit a comment with +1
-    Given I leave a comment like "XML attached"
-    And I edit the last comment with a +1
-    Then I should see +1 in the description

features/steps/shared/note.rb

@@ -6,70 +6,12 @@ module SharedNote
     wait_for_requests if javascript_test?
   end
-  step 'I delete a comment' do
-    page.within('.main-notes-list') do
-      note = find('.note')
-      note.hover
-
-      find('.more-actions').click
-      find('.more-actions .dropdown-menu li', match: :first)
-
-      accept_confirm { find(".js-note-delete").click }
-    end
-  end
-
   step 'I haven\'t written any comment text' do
     page.within(".js-main-target-form") do
       fill_in "note[note]", with: ""
     end
   end
-  step 'I leave a comment like "XML attached"' do
-    page.within(".js-main-target-form") do
-      fill_in "note[note]", with: "XML attached"
-      click_button "Comment"
-    end
-
-    wait_for_requests
-  end
-
-  step 'I preview a comment text like "Bug fixed :smile:"' do
-    page.within(".js-main-target-form") do
-      fill_in "note[note]", with: "Bug fixed :smile:"
-      find('.js-md-preview-button').click
-    end
-  end
-
-  step 'I submit the comment' do
-    page.within(".js-main-target-form") do
-      click_button "Comment"
-    end
-
-    wait_for_requests
-  end
-
-  step 'I write a comment like ":+1: Nice"' do
-    page.within(".js-main-target-form") do
-      fill_in 'note[note]', with: ':+1: Nice'
-    end
-  end
-
-  step 'I should not see a comment saying "XML attached"' do
-    expect(page).not_to have_css(".note")
-  end
-
-  step 'I should not see the cancel comment button' do
-    page.within(".js-main-target-form") do
-      should_not have_link("Cancel")
-    end
-  end
-
-  step 'I should not see the comment preview' do
-    page.within(".js-main-target-form") do
-      expect(find('.js-md-preview')).not_to be_visible
-    end
-  end
-
   step 'The comment preview tab should say there is nothing to do' do
     page.within(".js-main-target-form") do
       find('.js-md-preview-button').click
@@ -77,71 +19,7 @@ module SharedNote
     end
   end
-  step 'I should not see the comment text field' do
-    page.within(".js-main-target-form") do
-      expect(find('.js-note-text')).not_to be_visible
-    end
-  end
-
-  step 'I should see a comment saying "XML attached"' do
-    page.within(".note") do
-      expect(page).to have_content("XML attached")
-    end
-  end
-
-  step 'I should see an empty comment text field' do
-    page.within(".js-main-target-form") do
-      expect(page).to have_field("note[note]", with: "")
-    end
-  end
-
-  step 'I should see the comment write tab' do
-    page.within(".js-main-target-form") do
-      expect(page).to have_css('.js-md-write-button', visible: true)
-    end
-  end
-
-  step 'The comment preview tab should be display rendered Markdown' do
-    page.within(".js-main-target-form") do
-      find('.js-md-preview-button').click
-      expect(find('.js-md-preview')).to have_css('gl-emoji', visible: true)
-    end
-  end
-
-  step 'I should see the comment preview' do
-    page.within(".js-main-target-form") do
-      expect(page).to have_css('.js-md-preview', visible: true)
-    end
-  end
-
   step 'I should see no notes at all' do
     expect(page).not_to have_css('.note')
   end
-
-  # Markdown
-
-  step 'I edit the last comment with a +1' do
-    page.within(".main-notes-list") do
-      note = find('.note')
-      note.hover
-
-      note.find('.js-note-edit').click
-    end
-
-    page.find('.current-note-edit-form textarea')
-
-    page.within(".current-note-edit-form") do
-      fill_in 'note[note]', with: '+1 Awesome!'
-      click_button 'Save comment'
-    end
-    wait_for_requests
-  end
-
-  step 'I should see +1 in the description' do
-    page.within(".note") do
-      expect(page).to have_content("+1 Awesome!")
-    end
-
-    wait_for_requests
-  end
 end

features/support/capybara.rb

@@ -21,13 +21,7 @@ Capybara.register_driver :chrome do |app|
   options.add_argument("no-sandbox")
   # Run headless by default unless CHROME_HEADLESS specified
-  unless ENV['CHROME_HEADLESS'] =~ /^(false|no|0)$/i
-    options.add_argument("headless")
-
-    # Chrome documentation says this flag is needed for now
-    # https://developers.google.com/web/updates/2017/04/headless-chrome#cli
-    options.add_argument("disable-gpu")
-  end
+  options.add_argument("headless") unless ENV['CHROME_HEADLESS'] =~ /^(false|no|0)$/i
   # Disable /dev/shm use in CI. See https://gitlab.com/gitlab-org/gitlab-ee/issues/4252
   options.add_argument("disable-dev-shm-usage") if ENV['CI'] || ENV['CI_SERVER']

lib/api/helpers.rb

@@ -103,9 +103,9 @@ module API
     end
     def find_project(id)
-      if id =~ /^\d+$/
+      if id.is_a?(Integer) || id =~ /^\d+$/
         Project.find_by(id: id)
-      else
+      elsif id.include?("/")
         Project.find_by_full_path(id)
       end
     end

lib/gitlab.rb

@@ -3,21 +3,18 @@ require_dependency 'gitlab/git'
 module Gitlab
   COM_URL = 'https://gitlab.com'.freeze
   APP_DIRS_PATTERN = %r{^/?(app|config|ee|lib|spec|\(\w*\))}
+  SUBDOMAIN_REGEX = %r{\Ahttps://[a-z0-9]+\.gitlab\.com\z}
   def self.com?
-    # Check `staging?` as well to keep parity with gitlab.com
-    Gitlab.config.gitlab.url == COM_URL || staging?
+    # Check `gl_subdomain?` as well to keep parity with gitlab.com
+    Gitlab.config.gitlab.url == COM_URL || gl_subdomain?
   end
-  def self.staging?
-    Gitlab.config.gitlab.url == 'https://staging.gitlab.com'
+  def self.gl_subdomain?
+    SUBDOMAIN_REGEX === Gitlab.config.gitlab.url
   end
-  def self.dev?
-    Gitlab.config.gitlab.url == 'https://dev.gitlab.org'
-  end
-
-  def self.inc_controlled?
-    dev? || staging? || com?
+  def self.dev_env_or_com?
+    Rails.env.test? || Rails.env.development? || com?
   end
 end

lib/gitlab/cache/ci/project_pipeline_status.rb

@@ -40,7 +40,7 @@ module Gitlab
         end
         def self.cache_key_for_project(project)
-          "projects/#{project.id}/pipeline_status"
+          "#{Gitlab::Redis::Cache::CACHE_NAMESPACE}:projects/#{project.id}/pipeline_status"
         end
         def self.update_for_pipeline(pipeline)

lib/tasks/cache.rake

@@ -6,17 +6,22 @@ namespace :cache do
     desc "GitLab | Clear redis cache"
     task redis: :environment do
       Gitlab::Redis::Cache.with do |redis|
-        cursor = REDIS_SCAN_START_STOP
-        loop do
-          cursor, keys = redis.scan(
-            cursor,
-            match: "#{Gitlab::Redis::Cache::CACHE_NAMESPACE}*",
-            count: REDIS_CLEAR_BATCH_SIZE
-          )
-          redis.del(*keys) if keys.any?
-          break if cursor == REDIS_SCAN_START_STOP
+        cache_key_pattern = %W[#{Gitlab::Redis::Cache::CACHE_NAMESPACE}*
+                               projects/*/pipeline_status]
+        cache_key_pattern.each do |match|
+          cursor = REDIS_SCAN_START_STOP
+          loop do
+            cursor, keys = redis.scan(
+              cursor,
+              match: match,
+              count: REDIS_CLEAR_BATCH_SIZE
+            )
+            redis.del(*keys) if keys.any?
+
+            break if cursor == REDIS_SCAN_START_STOP
+          end
         end
       end
     end

package.json

@@ -16,7 +16,7 @@
     "webpack-prod": "NODE_ENV=production webpack --config config/webpack.config.js"
   },
   "dependencies": {
-    "@gitlab-org/gitlab-svgs": "^1.17.0",
+    "@gitlab-org/gitlab-svgs": "^1.18.0",
     "autosize": "^4.0.0",
     "axios": "^0.17.1",
     "babel-core": "^6.26.0",

spec/controllers/admin/application_settings_controller_spec.rb

@@ -72,11 +72,10 @@ describe Admin::ApplicationSettingsController do
       expect(ApplicationSetting.current.restricted_visibility_levels).to eq([10, 20])
     end
-    it 'falls back to defaults when settings are omitted' do
-      put :update, application_setting: {}
+    it 'updates the restricted_visibility_levels when empty array is passed' do
+      put :update, application_setting: { restricted_visibility_levels: [] }
       expect(response).to redirect_to(admin_application_settings_path)
-      expect(ApplicationSetting.current.default_project_visibility).to eq(Gitlab::VisibilityLevel::PRIVATE)
       expect(ApplicationSetting.current.restricted_visibility_levels).to be_empty
     end
   end

spec/db/production/settings_spec.rb

@@ -2,10 +2,15 @@ require 'spec_helper'
 require 'rainbow/ext/string'
 describe 'seed production settings' do
-  include StubENV
   let(:settings_file) { Rails.root.join('db/fixtures/production/010_settings.rb') }
   let(:settings) { Gitlab::CurrentSettings.current_application_settings }
+  before do
+    # It's important to set this variable so that we don't save a memoized
+    # (supposed to be) in-memory record in `Gitlab::CurrentSettings.in_memory_application_settings`
+    stub_env('IN_MEMORY_APPLICATION_SETTINGS', 'false')
+  end
+
   context 'GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN is set in the environment' do
     before do
       stub_env('GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN', '013456789')

spec/factories/ci/builds.rb

@@ -206,7 +206,7 @@ FactoryBot.define do
       options do
         {
             image: { name: 'ruby:2.1', entrypoint: '/bin/sh' },
-            services: ['postgres', { name: 'docker:dind', entrypoint: '/bin/sh', command: 'sleep 30', alias: 'docker' }],
+            services: ['postgres', { name: 'docker:stable-dind', entrypoint: '/bin/sh', command: 'sleep 30', alias: 'docker' }],
             after_script: %w(ls date),
             artifacts: {
                 name: 'artifacts_file',

spec/features/admin/admin_settings_spec.rb

@@ -32,6 +32,29 @@ feature 'Admin updates settings' do
     expect(find('#application_setting_visibility_level_20')).not_to be_checked
   end
+  scenario 'Modify import sources' do
+    expect(Gitlab::CurrentSettings.import_sources).not_to be_empty
+
+    page.within('.as-visibility-access') do
+      Gitlab::ImportSources.options.map do |name, _|
+        uncheck name
+      end
+
+      click_button 'Save changes'
+    end
+
+    expect(page).to have_content "Application settings saved successfully"
+    expect(Gitlab::CurrentSettings.import_sources).to be_empty
+
+    page.within('.as-visibility-access') do
+      check "Repo by URL"
+      click_button 'Save changes'
+    end
+
+    expect(page).to have_content "Application settings saved successfully"
+    expect(Gitlab::CurrentSettings.import_sources).to eq(['git'])
+  end
+
   scenario 'Change Visibility and Access Controls' do
     page.within('.as-visibility-access') do
       uncheck 'Project export enabled'
@@ -62,6 +85,26 @@ feature 'Admin updates settings' do
     expect(page).to have_content "Application settings saved successfully"
   end
+  scenario 'Modify oauth providers' do
+    expect(Gitlab::CurrentSettings.disabled_oauth_sign_in_sources).to be_empty
+
+    page.within('.as-signin') do
+      uncheck 'Google'
+      click_button 'Save changes'
+    end
+
+    expect(page).to have_content "Application settings saved successfully"
+    expect(Gitlab::CurrentSettings.disabled_oauth_sign_in_sources).to include('google_oauth2')
+
+    page.within('.as-signin') do
+      check "Google"
+      click_button 'Save changes'
+    end
+
+    expect(page).to have_content "Application settings saved successfully"
+    expect(Gitlab::CurrentSettings.disabled_oauth_sign_in_sources).not_to include('google_oauth2')
+  end
+
   scenario 'Change Help page' do
     page.within('.as-help-page') do
       fill_in 'Help page text', with: 'Example text'
@@ -211,16 +254,6 @@ feature 'Admin updates settings' do
     expect(find('#service_push_channel').value).to eq '#test_channel'
   end
-  context 'sign-in restrictions', :js do
-    it 'de-activates oauth sign-in source' do
-      page.within('.as-signin') do
-        find('input#application_setting_enabled_oauth_sign_in_sources_[value=gitlab]').send_keys(:return)
-
-        expect(find('.btn', text: 'GitLab.com')).not_to have_css('.active')
-      end
-    end
-  end
-
   scenario 'Change Keys settings' do
     page.within('.as-visibility-access') do
       select 'Are forbidden', from: 'RSA SSH keys'