Merge remote-tracking branch 'upstream/master' into pipeline-emails

* upstream/master: (64 commits)
  Update the 8.11 to 8.12 update documentation to use gitlab-shell 3.5.0
  Update GITLAB_SHELL_VERSION
  Move pushes_since_gc to Redis
  Fix Gitlab::Popen.popen thread-safety issue
  Allow trailing newline in secret base64 data
  Improve .haml-lint.yml, simplify the haml_lint task and remove CHANGELOG entry
  Fix a typo in documentation
  Restore SSH Key title auto-population behavior
  Add test with inactive error use case for 'POST /ci/api/v1/builds/register.json'
  Fix rubocop offences
  Update API documentation
  Update CHANGELOG
  Change response for /ci/api/v1/builds/register.json from 404 to 204
  Document the fact that merge requests from private forks can be checked out locally
  Change method name to #reopenable?
  Improve grammar
  Fix not working test with execute
  Add CHANGELOG
  Improve grammar
  Add #can_reopen? and tests
  ...

.gitlab-ci.yml

@@ -206,6 +206,7 @@ spinach 9 10 ruby21: *spinach-knapsack-ruby21
     - bundle exec $CI_BUILD_NAME
 rubocop: *exec
+rake haml_lint: *exec
 rake scss_lint: *exec
 rake brakeman: *exec
 rake flog: *exec

.haml-lint.yml

+# Whether to ignore frontmatter at the beginning of HAML documents for
+# frameworks such as Jekyll/Middleman
+skip_frontmatter: false
+exclude:
+  - 'vendor/**/*'
+  - 'spec/**/*'
+
+linters:
+  AltText:
+    enabled: false
+
+  ClassAttributeWithStaticValue:
+    enabled: false
+
+  ClassesBeforeIds:
+    enabled: false
+
+  ConsecutiveComments:
+    enabled: false
+
+  ConsecutiveSilentScripts:
+    enabled: false
+    max_consecutive: 2
+
+  EmptyObjectReference:
+    enabled: true
+
+  EmptyScript:
+    enabled: true
+
+  FinalNewline:
+    enabled: false
+    present: true
+
+  HtmlAttributes:
+    enabled: false
+
+  ImplicitDiv:
+    enabled: false
+
+  LeadingCommentSpace:
+    enabled: false
+
+  LineLength:
+    enabled: false
+    max: 80
+
+  MultilinePipe:
+    enabled: false
+
+  MultilineScript:
+    enabled: true
+
+  ObjectReferenceAttributes:
+    enabled: true
+
+  RuboCop:
+    enabled: false
+    # These cops are incredibly noisy when it comes to HAML templates, so we
+    # ignore them.
+    ignored_cops:
+      - Lint/BlockAlignment
+      - Lint/EndAlignment
+      - Lint/Void
+      - Metrics/LineLength
+      - Style/AlignParameters
+      - Style/BlockNesting
+      - Style/ElseAlignment
+      - Style/FileName
+      - Style/FinalNewline
+      - Style/FrozenStringLiteralComment
+      - Style/IfUnlessModifier
+      - Style/IndentationWidth
+      - Style/Next
+      - Style/TrailingBlankLines
+      - Style/TrailingWhitespace
+      - Style/WhileUntilModifier
+
+  RubyComments:
+    enabled: false
+
+  SpaceBeforeScript:
+    enabled: false
+
+  SpaceInsideHashAttributes:
+    enabled: false
+    style: space
+
+  Indentation:
+    enabled: true
+    character: space # or tab
+
+  TagName:
+    enabled: true
+
+  TrailingWhitespace:
+    enabled: false
+
+  UnnecessaryInterpolation:
+    enabled: false
+
+  UnnecessaryStringOutput:
+    enabled: false

CHANGELOG

@@ -8,19 +8,23 @@ v 8.12.0 (unreleased)
   - Prune events older than 12 months. (ritave)
   - Prepend blank line to `Closes` message on merge request linked to issue (lukehowell)
   - Filter tags by name !6121
+  - Update gitlab shell secret file also when it is empty. !3774 (glensc)
   - Give project selection dropdowns responsive width, make non-wrapping.
   - Make push events have equal vertical spacing.
   - Add two-factor recovery endpoint to internal API !5510
   - Pass the "Remember me" value to the U2F authentication form
   - Remove vendor prefixes for linear-gradient CSS (ClemMakesApps)
+  - Move pushes_since_gc from the database to Redis
   - Add font color contrast to external label in admin area (ClemMakesApps)
   - Change logo animation to CSS (ClemMakesApps)
   - Instructions for enabling Git packfile bitmaps !6104
   - Use Search::GlobalService.new in the `GET /projects/search/:query` endpoint
   - Fix pagination on user snippets page
   - Fix sorting of issues in API
+  - Ensure specs on sorting of issues in API are deterministic on MySQL
   - Escape search term before passing it to Regexp.new !6241 (winniehell)
   - Fix pinned sidebar behavior in smaller viewports !6169
+  - Fix file permissions change when updating a file on the Gitlab UI !5979
   - Change merge_error column from string to text type
   - Reduce contributions calendar data payload (ClemMakesApps)
   - Add `web_url` field to issue, merge request, and snippet API objects (Ben Boeckel)
@@ -77,6 +81,7 @@ v 8.12.0 (unreleased)
   - Remove inconsistent font weight for sidebar's labels (ClemMakesApps)
   - Align add button on repository view (ClemMakesApps)
   - Fix contributions calendar month label truncation (ClemMakesApps)
+  - Import release note descriptions from GitHub (EspadaV8)
   - Added tests for diff notes
   - Add pipeline events to Slack integration !5525
   - Add a button to download latest successful artifacts for branches and tags !5142
@@ -88,12 +93,14 @@ v 8.12.0 (unreleased)
   - Fix repo title alignment (ClemMakesApps)
   - Change update interval of contacted_at
   - Fix branch title trailing space on hover (ClemMakesApps)
+  - Don't include 'Created By' tag line when importing from GitHub if there is a linked GitLab account (EspadaV8)
   - Award emoji tooltips containing more than 10 usernames are now truncated !4780 (jlogandavison)
   - Fix duplicate "me" in award emoji tooltip !5218 (jlogandavison)
   - Order award emoji tooltips in order they were added (EspadaV8)
   - Fix spacing and vertical alignment on build status icon on commits page (ClemMakesApps)
   - Update merge_requests.md with a simpler way to check out a merge request. !5944
   - Fix button missing type (ClemMakesApps)
+  - Gitlab::Checks is now instrumented
   - Move to project dropdown with infinite scroll for better performance
   - Fix leaking of submit buttons outside the width of a main container !18731 (originally by @pavelloz)
   - Load branches asynchronously in Cherry Pick and Revert dialogs.
@@ -112,6 +119,8 @@ v 8.12.0 (unreleased)
   - Avoid conflict with admin labels when importing GitHub labels
   - User can edit closed MR with deleted fork (Katarzyna Kobierska Ula Budziszewska) !5496
   - Fix repository page ui issues
+  - Avoid protected branches checks when verifying access without branch name
+  - Add information about user and manual build start to runner as variables !6201 (Sergey Gnuskov)
   - Fixed invisible scroll controls on build page on iPhone
   - Fix error on raw build trace download for old builds stored in database !4822
   - Refactor the triggers page and documentation !6217
@@ -120,8 +129,14 @@ v 8.12.0 (unreleased)
   - API for Ci Lint !5953 (Katarzyna Kobierska Urszula Budziszewska)
   - Allow bulk update merge requests from merge requests index page
   - Add notification_settings API calls !5632 (mahcsig)
+  - Remove duplication between project builds and admin builds view !5680 (Katarzyna Kobierska Ula Budziszewska)
+  - Deleting source project with existing fork link will close all related merge requests !6177 (Katarzyna Kobierska Ula Budziszeska)
+  - Return 204 instead of 404 for /ci/api/v1/builds/register.json if no builds are scheduled for a runner !6225
+  - Fix Gitlab::Popen.popen thread-safety issue
 v 8.11.6 (unreleased)
+  - Fix an error where we were unable to create a CommitStatus for running state
+  - Restore SSH Key title auto-population behavior. !6186
 v 8.11.5
   - Optimize branch lookups and force a repository reload for Repository#find_branch. !6087

GITLAB_SHELL_VERSION

-3.4.0
+3.5.0

Gemfile

@@ -298,6 +298,7 @@ group :development, :test do
   gem 'rubocop', '~> 0.41.2', require: false
   gem 'rubocop-rspec', '~> 1.5.0', require: false
   gem 'scss_lint', '~> 0.47.0', require: false
+  gem 'haml_lint', '~> 0.18.2', require: false
   gem 'simplecov', '0.12.0', require: false
   gem 'flog', '~> 4.3.2', require: false
   gem 'flay', '~> 2.6.1', require: false

Gemfile.lock

@@ -322,6 +322,13 @@ GEM
     grape-entity (0.4.8)
       activesupport
       multi_json (>= 1.3.2)
+    haml (4.0.7)
+      tilt
+    haml_lint (0.18.2)
+      haml (~> 4.0)
+      rake (>= 10, < 12)
+      rubocop (>= 0.36.0)
+      sysexits (~> 1.1)
     hamlit (2.6.1)
       temple (~> 0.7.6)
       thor
@@ -723,6 +730,7 @@ GEM
     stringex (2.5.2)
     sys-filesystem (1.1.6)
       ffi
+    sysexits (1.2.0)
     systemu (2.6.5)
     task_list (1.0.2)
       html-pipeline
@@ -866,6 +874,7 @@ DEPENDENCIES
   gon (~> 6.1.0)
   grape (~> 0.15.0)
   grape-entity (~> 0.4.2)
+  haml_lint (~> 0.18.2)
   hamlit (~> 2.6.1)
   health_check (~> 2.1.0)
   hipchat (~> 1.5.0)

app/assets/stylesheets/framework/mobile.scss

@@ -79,10 +79,6 @@
     padding-left: 15px !important;
   }
-  .issue-info, .merge-request-info {
-    display: none;
-  }
-
   .nav-links, .nav-links {
     li a {
       font-size: 14px;

app/assets/stylesheets/pages/issuable.scss

@@ -206,7 +206,7 @@
     padding-top: 0;
     .block {
-      width: $sidebar_collapsed_width - 1px;
+      width: $sidebar_collapsed_width - 2px;
       margin-left: -19px;
       padding: 15px 0 0;
       border-bottom: none;

app/assets/stylesheets/pages/issues.scss

@@ -37,6 +37,15 @@ form.edit-issue {
   margin: 0;
 }
+ul.related-merge-requests > li {
+  display: -ms-flexbox;
+  display: -webkit-flex;
+  display: flex;
+  .merge-request-id {
+    flex-shrink: 0;
+  }
+}
+
 .merge-requests-title, .related-branches-title {
   font-size: 16px;
   font-weight: 600;

app/controllers/concerns/creates_commit.rb

@@ -7,8 +7,7 @@ module CreatesCommit
     commit_params = @commit_params.merge(
       source_project: @project,
       source_branch: @ref,
-      target_branch: @target_branch,
-      previous_path: @previous_path
+      target_branch: @target_branch
     )
     result = service.new(@tree_edit_project, current_user, commit_params).execute

app/controllers/projects/blob_controller.rb

@@ -38,12 +38,7 @@ class Projects::BlobController < Projects::ApplicationController
   end
   def update
-    if params[:file_path].present?
-      @previous_path = @path
-      @path = params[:file_path]
-      @commit_params[:file_path] = @path
-    end
-
+    @path = params[:file_path] if params[:file_path].present?
     after_edit_path =
       if from_merge_request && @target_branch == @ref
         diffs_namespace_project_merge_request_path(from_merge_request.target_project.namespace, from_merge_request.target_project, from_merge_request) +
@@ -143,6 +138,8 @@ class Projects::BlobController < Projects::ApplicationController
           params[:file_name] = params[:file].original_filename
         end
         File.join(@path, params[:file_name])
+      elsif params[:file_path].present?
+        params[:file_path]
       else
         @path
       end
@@ -155,6 +152,7 @@ class Projects::BlobController < Projects::ApplicationController
     @commit_params = {
       file_path: @file_path,
       commit_message: params[:commit_message],
+      previous_path: @path,
       file_content: params[:content],
       file_content_encoding: params[:encoding],
       last_commit_sha: params[:last_commit_sha]

app/controllers/projects/merge_requests_controller.rb

@@ -428,17 +428,11 @@ class Projects::MergeRequestsController < Projects::ApplicationController
   end
   def validates_merge_request
-    # If source project was removed (Ex. mr from fork to origin)
-    return invalid_mr unless @merge_request.source_project
-
     # Show git not found page
     # if there is no saved commits between source & target branch
     if @merge_request.commits.blank?
       # and if target branch doesn't exist
       return invalid_mr unless @merge_request.target_branch_exists?
-
-      # or if source branch doesn't exist
-      return invalid_mr unless @merge_request.source_branch_exists?
     end
   end

app/models/ci/build.rb

@@ -151,6 +151,7 @@ module Ci
       variables += runner.predefined_variables if runner
       variables += project.container_registry_variables
       variables += yaml_variables
+      variables += user_variables
       variables += project.secret_variables
       variables += trigger_request.user_variables if trigger_request
       variables
@@ -437,6 +438,15 @@ module Ci
       read_attribute(:yaml_variables) || build_attributes_from_config[:yaml_variables] || []
     end
+    def user_variables
+      return [] if user.blank?
+
+      [
+        { key: 'GITLAB_USER_ID', value: user.id.to_s, public: true },
+        { key: 'GITLAB_USER_EMAIL', value: user.email, public: true }
+      ]
+    end
+
     private
     def update_artifacts_size
@@ -472,6 +482,7 @@ module Ci
       ]
       variables << { key: 'CI_BUILD_TAG', value: ref, public: true } if tag?
       variables << { key: 'CI_BUILD_TRIGGERED', value: 'true', public: true } if trigger_request
+      variables << { key: 'CI_BUILD_MANUAL', value: 'true', public: true } if manual?
       variables
     end

app/models/merge_request.rb

@@ -316,6 +316,10 @@ class MergeRequest < ActiveRecord::Base
     closed? && forked_source_project_missing?
   end
+  def closed_without_source_project?
+    closed? && !source_project
+  end
+
   def forked_source_project_missing?
     return false unless for_fork?
     return true unless source_project
@@ -323,6 +327,12 @@ class MergeRequest < ActiveRecord::Base
     !source_project.forked_from?(target_project)
   end
+  def reopenable?
+    return false if closed_without_fork? || closed_without_source_project? || merged?
+
+    closed?
+  end
+
   def ensure_merge_request_diff
     merge_request_diff || create_merge_request_diff
   end

app/models/project.rb

@@ -58,7 +58,7 @@ class Project < ActiveRecord::Base
   # Relations
   belongs_to :creator, foreign_key: 'creator_id', class_name: 'User'
-  belongs_to :group, -> { where(type: Group) }, foreign_key: 'namespace_id'
+  belongs_to :group, -> { where(type: 'Group') }, foreign_key: 'namespace_id'
   belongs_to :namespace
   has_one :last_event, -> {order 'events.created_at DESC'}, class_name: 'Event', foreign_key: 'project_id'
@@ -1289,8 +1289,24 @@ class Project < ActiveRecord::Base
     end
   end
+  def pushes_since_gc
+    Gitlab::Redis.with { |redis| redis.get(pushes_since_gc_redis_key).to_i }
+  end
+
+  def increment_pushes_since_gc
+    Gitlab::Redis.with { |redis| redis.incr(pushes_since_gc_redis_key) }
+  end
+
+  def reset_pushes_since_gc
+    Gitlab::Redis.with { |redis| redis.del(pushes_since_gc_redis_key) }
+  end
+
   private
+  def pushes_since_gc_redis_key
+    "projects/#{id}/pushes_since_gc"
+  end
+
   # Prevents the creation of project_feature record for every project
   def setup_project_feature
     build_project_feature unless project_feature

app/models/repository.rb

@@ -813,7 +813,7 @@ class Repository
         update: true
       }
-      if previous_path
+      if previous_path && previous_path != path
         options[:file][:previous_path] = previous_path
         Gitlab::Git::Blob.rename(raw_repository, options)
       else

app/services/ci/process_pipeline_service.rb

@@ -31,13 +31,13 @@ module Ci
       current_status = status_for_prior_stages(index)
       created_builds_in_stage(index).select do |build|
-        process_build(build, current_status)
+        if HasStatus::COMPLETED_STATUSES.include?(current_status)
+          process_build(build, current_status)
+        end
       end
     end
     def process_build(build, current_status)
-      return false unless HasStatus::COMPLETED_STATUSES.include?(current_status)
-
       if valid_statuses_for_when(build.when).include?(current_status)
         build.enqueue
         true

app/services/projects/destroy_service.rb

@@ -27,6 +27,8 @@ module Projects
       # Git data (e.g. a list of branch names).
       flush_caches(project, wiki_path)
+      Projects::UnlinkForkService.new(project, current_user).execute
+
       Project.transaction do
         project.destroy!

app/services/projects/housekeeping_service.rb

@@ -30,10 +30,8 @@ module Projects
     end
     def increment!
-      if Gitlab::ExclusiveLease.new("project_housekeeping:increment!:#{@project.id}", timeout: 60).try_obtain
-        Gitlab::Metrics.measure(:increment_pushes_since_gc) do
-          update_pushes_since_gc(@project.pushes_since_gc + 1)
-        end
+      Gitlab::Metrics.measure(:increment_pushes_since_gc) do
+        @project.increment_pushes_since_gc
       end
     end
@@ -43,14 +41,10 @@ module Projects
       GitGarbageCollectWorker.perform_async(@project.id)
     ensure
       Gitlab::Metrics.measure(:reset_pushes_since_gc) do
-        update_pushes_since_gc(0)
+        @project.reset_pushes_since_gc
       end
     end
-    def update_pushes_since_gc(new_value)
-      @project.update_column(:pushes_since_gc, new_value)
-    end
-
     def try_obtain_lease
       Gitlab::Metrics.measure(:obtain_housekeeping_lease) do
         lease = ::Gitlab::ExclusiveLease.new("project_housekeeping:#{@project.id}", timeout: LEASE_TIMEOUT)

app/views/admin/builds/_build.html.haml

-- project = build.project
-%tr.build.commit
-  %td.status
-    = ci_status_with_icon(build.status)
-
-  %td
-    .branch-commit
-      - if can?(current_user, :read_build, build.project)
-        = link_to namespace_project_build_url(build.project.namespace, build.project, build) do
-          %span.build-link ##{build.id}
-      - else
-        %span.build-link ##{build.id}
-
-      - if build.ref
-        .icon-container
-          = build.tag? ? icon('tag') : icon('code-fork')
-        = link_to build.ref, namespace_project_commits_path(build.project.namespace, build.project, build.ref), class: "monospace branch-name"
-      - else
-        .light none
-      .icon-container
-        = custom_icon("icon_commit")
-
-      = link_to build.short_sha, namespace_project_commit_path(build.project.namespace, build.project, build.sha), class: "monospace commit-id"
-      - if build.stuck?
-        %i.fa.fa-warning.text-warning
-
-      .label-container
-        - if build.tags.any?
-          - build.tags.each do |tag|
-            %span.label.label-primary
-              = tag
-        - if build.try(:trigger_request)
-          %span.label.label-info triggered
-        - if build.try(:allow_failure)
-          %span.label.label-danger allowed to fail
-
-  %td
-    - if project
-      = link_to project.name_with_namespace, admin_namespace_project_path(project.namespace, project)
-
-  %td
-    - if build.try(:runner)
-      = runner_link(build.runner)
-    - else
-      .light none
-
-  %td
-    #{build.stage} / #{build.name}
-
-  %td
-    - if build.duration
-      %p.duration
-        = custom_icon("icon_timer")
-        = duration_in_numbers(build.duration)
-
-    - if build.finished_at
-      %p.finished-at
-        = icon("calendar")
-        %span #{time_ago_with_tooltip(build.finished_at)}
-
-  - if defined?(coverage) && coverage
-    %td.coverage
-      - if build.try(:coverage)
-        #{build.coverage}%
-
-  %td
-    .pull-right
-      - if can?(current_user, :read_build, project) && build.artifacts?
-        = link_to download_namespace_project_build_artifacts_path(build.project.namespace, build.project, build), title: 'Download artifacts', class: 'btn btn-build' do
-          %i.fa.fa-download
-      - if can?(current_user, :update_build, build.project)
-        - if build.active?
-          = link_to cancel_namespace_project_build_path(build.project.namespace, build.project, build, return_to: request.original_url), method: :post, title: 'Cancel', class: 'btn btn-build' do
-            %i.fa.fa-remove.cred
-        - elsif defined?(allow_retry) && allow_retry && build.retryable?
-          = link_to retry_namespace_project_build_path(build.project.namespace, build.project, build, return_to: request.original_url), method: :post, title: 'Retry', class: 'btn btn-build' do
-            %i.fa.fa-refresh