Add latest changes from gitlab-org/gitlab@master

.gitlab/issue_templates/Security developer workflow.md

 <!--
 # Read me first!
-Create this issue under https://dev.gitlab.org/gitlab/gitlabhq
+Create this issue under https://gitlab.com/gitlab-org/security
 Set the title to: `Description of the original issue`
 -->
-### Prior to starting the security release work
+## Prior to starting the security release work
 - [ ] Read the [security process for developers] if you are not familiar with it.
-- [ ] Link to the original issue adding it to the [links section](#links)
-- [ ] Run `scripts/security-harness` in the CE, EE, and/or Omnibus to prevent pushing to any remote besides `dev.gitlab.org`
-- [ ] Create a new branch prefixing it with `security-`
-- [ ] Create a MR targeting `dev.gitlab.org` `master`
-- [ ] Add a link to this issue in the original security issue on `gitlab.com`.
-#### Backports
-- [ ] Once the MR is ready to be merged, create MRs targeting the latest 3 stable branches
-    - [ ] At this point, it might be easy to squash the commits from the MR into one
-    - You can use the script `bin/secpick` instead of the following steps, to help you cherry-picking. See the [secpick documentation]
-    - [ ] Create each MR targeting the stable branch `X-Y-stable`, using the "Security Release" merge request template.
-    - Every merge request will have its own set of TODOs, so make sure to
-      complete those.
-- [ ] Make sure all MRs have a link in the [links section](#links)
-[secpick documentation]: https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md#secpick-script
-#### Documentation and final details
-- [ ] Check the topic on #releases to see when the next release is going to happen and add a link to the [links section](#links)
-- [ ] Add links to this issue and your MRs in the description of the security release issue
+- [ ] Link this issue in the Security Release issue on GitLab.com. You can find this issue in the topic of the `#releases` channel.
+- [ ] Add a link to the confidential `gitlab-org/gitlab` issue describing the vulnerability next to **Original issue** in the [links table](#links).
+- [ ] Add a link to the confidential `gitlab-org/gitlab` Security release issue next to **Security release issue** in the [links table](#links).
+- [ ] Run `scripts/security-harness` in your local repository to prevent accidentally pushing to any remote besides `gitlab.com/gitlab-org/security`.
+## Development
+- [ ] Create a new branch prefixing it with `security-`.
+- [ ] Create a merge request targeting `master` on `gitlab.com/gitlab-org/security` and use the [Security Release merge request template].
+- [ ] Follow the same [code review process]: Assign to a reviewer, then to a maintainer.
+After your merge request has being approved according to our [approval guidelines], you're ready to prepare the backports
+
+## Backports
+- [ ] Once the MR is ready to be merged, create MRs targeting the latest 3 stable branches
+   * At this point, it might be easy to squash the commits from the MR into one
+   * You can use the script `bin/secpick` instead of the following steps, to help you cherry-picking. See the [secpick documentation]
+- [ ] Create each MR targeting the stable branch `X-Y-stable`, using the [Security Release merge request template].
+   * Every merge request will have its own set of TODOs, so make sure to complete those.
+- [ ] Make sure all MRs are linked in the [Links section](#links)
+
+## Documentation and final details
+- [ ] Ensure the [Links section](#links) is completed.
 - [ ] Find out the versions affected (the Git history of the files affected may help you with this) and add them to the [details section](#details)
 - [ ] Fill in any upgrade notes that users may need to take into account in the [details section](#details)
 - [ ] Add Yes/No and further details if needed to the migration and settings columns in the [details section](#details)
 - [ ] Add the nickname of the external user who found the issue (and/or HackerOne profile) to the Thanks row in the [details section](#details)
 - [ ] Once your `master` MR is merged, comment on the original security issue with a link to that MR indicating the issue is fixed.
-### Summary
-#### Links
+## Summary
+### Links
 | Description | Link |
 | -------- | -------- |
 | Original issue   | #TODO  |
 | Security release issue   | #TODO  |
 | `master` MR | !TODO   |
-| `master` MR (EE) | !TODO   |
 | `Backport X.Y` MR | !TODO   |
 | `Backport X.Y` MR | !TODO   |
 | `Backport X.Y` MR | !TODO   |
-| `Backport X.Y` MR (EE) | !TODO   |
-| `Backport X.Y` MR (EE) | !TODO   |
-| `Backport X.Y` MR (EE) | !TODO   |
-#### Details
+### Details
 | Description | Details | Further details|
 | -------- | -------- | -------- |
@@ -65,6 +64,9 @@ Set the title to: `Description of the original issue`
 | Thanks | | |
 [security process for developers]: https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md
-[RM list]:  https://about.gitlab.com/release-managers/
+[secpick documentation]: https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md#secpick-script
+[security Release merge request template]: https://gitlab.com/gitlab-org/security/gitlab/blob/master/.gitlab/merge_request_templates/Security%20Release.md
+[code review process]: https://docs.gitlab.com/ee/development/code_review.html
+[approval guidelines]: https://docs.gitlab.com/ee/development/code_review.html#approval-guidelines
 /label ~security

.gitlab/merge_request_templates/Security Release.md

 <!--
 # README first!
-This MR should be created on `dev.gitlab.org`.
+This MR should be created on `gitlab.com/gitlab-org/security/gitlab`.
 See [the general developer security release guidelines](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md).
-This merge request _must not_ close the corresponding security issue _unless_ it
-targets master.
-
-When submitting a merge request for CE, a corresponding EE merge request is
-always required. This makes it easier to merge security merge requests, as
-manually merging CE into EE is no longer required.
-
 -->
+
 ## Related issues
 <!-- Mention the issue(s) this MR is related to -->
 ## Developer checklist
-- [ ] Link to the developer security workflow issue on `dev.gitlab.org`
-- [ ] MR targets `master`, or `X-Y-stable` for backports
-- [ ] Milestone is set for the version this MR applies to
-- [ ] Title of this MR is the same as for all backports
+- [ ] Link this MR in the `links` section of the related issue on [GitLab Security].
+- [ ] Merge request targets `master`, or `X-Y-stable` for backports.
+- [ ] Milestone is set for the version this merge request applies to.
+- [ ] Title of this merge request is the same as for all backports.
 - [ ] A [CHANGELOG entry](https://docs.gitlab.com/ee/development/changelog.html) is added without a `merge_request` value, with `type` set to `security`
-- [ ] Add a link to this MR in the `links` section of related issue
-- [ ] Set up an EE MR (always required for CE merge requests): EE_MR_LINK_HERE
-- [ ] Assign to a reviewer (that is not a release manager)
+- [ ] Assign to a reviewer and maintainer, per our [Code Review process].
+- [ ] If this merge request targets `master`, ensure it's approved according to our [Approval Guidelines].
+- [ ] Merge request _must not_ close the corresponding security issue, _unless_ it targets `master`.
+
+**Note:** Reviewer/maintainer should not be a Release Manager
 ## Reviewer checklist
@@ -33,3 +29,7 @@ manually merging CE into EE is no longer required.
 - [ ] Assigned to `@gitlab-release-tools-bot` with passing CI pipelines
 /label ~security
+
+[GitLab Security]: https://gitlab.com/gitlab-org/security/gitlab
+[approval guidelines]: https://docs.gitlab.com/ee/development/code_review.html#approval-guidelines
+[Code Review process]: https://docs.gitlab.com/ee/development/code_review.html

app/assets/javascripts/error_tracking_settings/store/actions.js

@@ -25,8 +25,8 @@ export const receiveProjectsError = ({ commit }) => {
 export const fetchProjects = ({ dispatch, state }) => {
   dispatch('requestProjects');
   return axios
-    .post(state.listProjectsEndpoint, {
-      error_tracking_setting: {
+    .get(state.listProjectsEndpoint, {
+      params: {
         api_host: state.apiHost,
         token: state.token,
       },

app/assets/stylesheets/framework/buttons.scss

@@ -29,7 +29,7 @@
   &:focus,
   &:active {
     background-color: $btn-active-gray;
-    box-shadow: $gl-btn-active-background;
+    box-shadow: none;
   }
 }

app/controllers/projects/blame_controller.rb

@@ -17,6 +17,7 @@ class Projects::BlameController < Projects::ApplicationController
     end
     environment_params = @repository.branch_exists?(@ref) ? { ref: @ref } : { commit: @commit }
+    environment_params[:find_latest] = true
     @environment = EnvironmentsFinder.new(@project, current_user, environment_params).execute.last
     @blame_groups = Gitlab::Blame.new(@blob, @commit).groups

app/controllers/projects/blob_controller.rb

@@ -205,6 +205,7 @@ class Projects::BlobController < Projects::ApplicationController
   def show_html
     environment_params = @repository.branch_exists?(@ref) ? { ref: @ref } : { commit: @commit }
+    environment_params[:find_latest] = true
     @environment = EnvironmentsFinder.new(@project, current_user, environment_params).execute.last
     @last_commit = @repository.last_commit_for_path(@commit.id, @blob.path)

app/controllers/projects/commit_controller.rb

@@ -151,7 +151,7 @@ class Projects::CommitController < Projects::ApplicationController
     @diffs = commit.diffs(opts)
     @notes_count = commit.notes.count
-    @environment = EnvironmentsFinder.new(@project, current_user, commit: @commit).execute.last
+    @environment = EnvironmentsFinder.new(@project, current_user, commit: @commit, find_latest: true).execute.last
   end
   # rubocop: disable CodeReuse/ActiveRecord

app/controllers/projects/compare_controller.rb

@@ -101,6 +101,7 @@ class Projects::CompareController < Projects::ApplicationController
   def define_environment
     if compare
       environment_params = @repository.branch_exists?(head_ref) ? { ref: head_ref } : { commit: compare.commit }
+      environment_params[:find_latest] = true
       @environment = EnvironmentsFinder.new(@project, current_user, environment_params).execute.last
     end
   end

app/controllers/projects/error_tracking/projects_controller.rb

+# frozen_string_literal: true
+
+module Projects
+  module ErrorTracking
+    class ProjectsController < Projects::ApplicationController
+      respond_to :json
+
+      before_action :authorize_read_sentry_issue!
+
+      def index
+        service = ::ErrorTracking::ListProjectsService.new(
+          project,
+          current_user,
+          list_projects_params
+        )
+        result = service.execute
+
+        if result[:status] == :success
+          render json: { projects: serialize_projects(result[:projects]) }
+        else
+          render(
+            status: result[:http_status] || :bad_request,
+            json: { message: result[:message] }
+          )
+        end
+      end
+
+      private
+
+      def list_projects_params
+        { api_host: params[:api_host], token: params[:token] }
+      end
+
+      def serialize_projects(projects)
+        ::ErrorTracking::ProjectSerializer
+          .new(project: project, user: current_user)
+          .represent(projects)
+      end
+    end
+  end
+end

app/controllers/projects/error_tracking_controller.rb

@@ -33,14 +33,6 @@ class Projects::ErrorTrackingController < Projects::ApplicationController
     end
   end
-  def list_projects
-    respond_to do |format|
-      format.json do
-        render_project_list_json
-      end
-    end
-  end
-
   private
   def render_index_json
@@ -84,28 +76,6 @@ class Projects::ErrorTrackingController < Projects::ApplicationController
     }
   end
-  def render_project_list_json
-    service = ErrorTracking::ListProjectsService.new(
-      project,
-      current_user,
-      list_projects_params
-    )
-    result = service.execute
-
-    if result[:status] == :success
-      render json: {
-        projects: serialize_projects(result[:projects])
-      }
-    else
-      return render(
-        status: result[:http_status] || :bad_request,
-        json: {
-          message: result[:message]
-        }
-      )
-    end
-  end
-
   def handle_errors(result)
     unless result[:status] == :success
       render json: { message: result[:message] },
@@ -117,10 +87,6 @@ class Projects::ErrorTrackingController < Projects::ApplicationController
     params.permit(:search_term, :sort, :cursor)
   end
-  def list_projects_params
-    params.require(:error_tracking_setting).permit([:api_host, :token])
-  end
-
   def issue_details_params
     params.permit(:issue_id)
   end
@@ -150,10 +116,4 @@ class Projects::ErrorTrackingController < Projects::ApplicationController
       .new(project: project, user: current_user)
       .represent(event)
   end
-
-  def serialize_projects(projects)
-    ErrorTracking::ProjectSerializer
-      .new(project: project, user: current_user)
-      .represent(projects)
-  end
 end

app/controllers/projects/merge_requests/creations_controller.rb

@@ -52,7 +52,7 @@ class Projects::MergeRequests::CreationsController < Projects::MergeRequests::Ap
     @diff_notes_disabled = true
-    @environment = @merge_request.environments_for(current_user).last
+    @environment = @merge_request.environments_for(current_user, latest: true).last
     render json: { html: view_to_html_string('projects/merge_requests/creations/_diffs', diffs: @diffs, environment: @environment) }
   end

app/controllers/projects/merge_requests/diffs_controller.rb

@@ -51,7 +51,7 @@ class Projects::MergeRequests::DiffsController < Projects::MergeRequests::Applic
   # Deprecated: https://gitlab.com/gitlab-org/gitlab/issues/37735
   def render_diffs
     diffs = @compare.diffs(diff_options)
-    @environment = @merge_request.environments_for(current_user).last
+    @environment = @merge_request.environments_for(current_user, latest: true).last
     diffs.unfold_diff_files(note_positions.unfoldable)
     diffs.write_cache

app/finders/environments_finder.rb

@@ -25,25 +25,13 @@ class EnvironmentsFinder
       .select(:environment_id)
     environments = project.environments.available
-      .where(id: environment_ids).order_by_last_deployed_at.to_a
-    environments.select! do |environment|
-      Ability.allowed?(current_user, :read_environment, environment)
-    end
-
-    if ref && commit
-      environments.select! do |environment|
-        environment.includes_commit?(commit)
-      end
-    end
-
-    if ref && params[:recently_updated]
-      environments.select! do |environment|
-        environment.recently_updated_on_branch?(ref)
-      end
+      .where(id: environment_ids)
+    if params[:find_latest]
+      find_one(environments.order_by_last_deployed_at_desc)
+    else
+      find_all(environments.order_by_last_deployed_at.to_a)
     end
-
-    environments
   end
   # rubocop: enable CodeReuse/ActiveRecord
@@ -62,6 +50,24 @@ class EnvironmentsFinder
   private
+  def find_one(environments)
+    [environments.find { |environment| valid_environment?(environment) }].compact
+  end
+
+  def find_all(environments)
+    environments.select { |environment| valid_environment?(environment) }
+  end
+
+  def valid_environment?(environment)
+    # Go in order of cost: SQL calls are cheaper than Gitaly calls
+    return false unless Ability.allowed?(current_user, :read_environment, environment)
+
+    return false if ref && params[:recently_updated] && !environment.recently_updated_on_branch?(ref)
+    return false if ref && commit && !environment.includes_commit?(commit)
+
+    true
+  end
+
   def ref
     params[:ref].try(:to_s)
   end

app/finders/events_finder.rb

@@ -6,7 +6,7 @@ class EventsFinder
   MAX_PER_PAGE = 100
-  attr_reader :source, :params, :current_user
+  attr_reader :source, :params, :current_user, :scope
   requires_cross_project_access unless: -> { source.is_a?(Project) }, model: Event
@@ -15,6 +15,7 @@ class EventsFinder
   # Arguments:
   #   source - which user or project to looks for events on
   #   current_user - only return events for projects visible to this user
+  #   scope - return all events across a user's projects
   #   params:
   #     action: string
   #     target_type: string
@@ -27,11 +28,12 @@ class EventsFinder
   def initialize(params = {})
     @source = params.delete(:source)
     @current_user = params.delete(:current_user)
+    @scope = params.delete(:scope)
     @params = params
   end
   def execute
-    events = source.events
+    events = get_events
     events = by_current_user_access(events)
     events = by_action(events)
@@ -47,6 +49,12 @@ class EventsFinder
   private
+  def get_events
+    return EventCollection.new(current_user.authorized_projects).all_project_events if scope == 'all'
+
+    source.events
+  end
+
   # rubocop: disable CodeReuse/ActiveRecord
   def by_current_user_access(events)
     events.merge(Project.public_or_visible_to_user(current_user))

app/helpers/dashboard_helper.rb

@@ -58,6 +58,10 @@ module DashboardHelper
       links += [:activity, :milestones]
     end
+    if can?(current_user, :read_instance_statistics)
+      links << :analytics
+    end
+
     links
   end
 end

app/models/environment.rb

@@ -48,13 +48,14 @@ class Environment < ApplicationRecord
   scope :available, -> { with_state(:available) }
   scope :stopped, -> { with_state(:stopped) }
+
   scope :order_by_last_deployed_at, -> do
-    max_deployment_id_sql =
-      Deployment.select(Deployment.arel_table[:id].maximum)
-      .where(Deployment.arel_table[:environment_id].eq(arel_table[:id]))
-      .to_sql
     order(Gitlab::Database.nulls_first_order("(#{max_deployment_id_sql})", 'ASC'))
   end
+  scope :order_by_last_deployed_at_desc, -> do
+    order(Gitlab::Database.nulls_last_order("(#{max_deployment_id_sql})", 'DESC'))
+  end
+
   scope :in_review_folder, -> { where(environment_type: "review") }
   scope :for_name, -> (name) { where(name: name) }
   scope :preload_cluster, -> { preload(last_deployment: :cluster) }
@@ -90,6 +91,12 @@ class Environment < ApplicationRecord
     end
   end
+  def self.max_deployment_id_sql
+    Deployment.select(Deployment.arel_table[:id].maximum)
+    .where(Deployment.arel_table[:environment_id].eq(arel_table[:id]))
+    .to_sql
+  end
+
   def self.pluck_names
     pluck(:name)
   end

app/models/event_collection.rb

@@ -30,17 +30,24 @@ class EventCollection
     relation = if groups
                  project_and_group_events
                else
-                 relation_with_join_lateral('project_id', projects)
+                 project_events
                end
     relation = paginate_events(relation)
     relation.with_associations.to_a
   end
+  def all_project_events
+    Event.from_union([project_events]).recent
+  end
+
   private
+  def project_events
+    relation_with_join_lateral('project_id', projects)
+  end
+
   def project_and_group_events
-    project_events = relation_with_join_lateral('project_id', projects)
     group_events = relation_with_join_lateral('group_id', groups)
     Event.from_union([project_events, group_events]).recent

app/models/merge_request.rb

@@ -1122,22 +1122,18 @@ class MergeRequest < ApplicationRecord
     actual_head_pipeline.success?
   end
-  def environments_for(current_user)
+  def environments_for(current_user, latest: false)
     return [] unless diff_head_commit
-    @environments ||= Hash.new do |h, current_user|
-      envs = EnvironmentsFinder.new(target_project, current_user,
-        ref: target_branch, commit: diff_head_commit, with_tags: true).execute
-      if source_project
-        envs.concat EnvironmentsFinder.new(source_project, current_user,
-          ref: source_branch, commit: diff_head_commit).execute
-      end
-
-      h[current_user] = envs.uniq
+    envs = EnvironmentsFinder.new(target_project, current_user,
+      ref: target_branch, commit: diff_head_commit, with_tags: true, find_latest: latest).execute
+    if source_project
+      envs.concat EnvironmentsFinder.new(source_project, current_user,
+        ref: source_branch, commit: diff_head_commit, find_latest: latest).execute
     end
-    @environments[current_user]
+    envs.uniq
   end
   ##

app/services/projects/operations/update_service.rb

@@ -30,7 +30,7 @@ module Projects
         settings = params[:error_tracking_setting_attributes]
         return {} if settings.blank?
-        api_url = ErrorTracking::ProjectErrorTrackingSetting.build_api_url_from(
+        api_url = ::ErrorTracking::ProjectErrorTrackingSetting.build_api_url_from(
           api_host: settings[:api_host],
           project_slug: settings.dig(:project, :slug),
           organization_slug: settings.dig(:project, :organization_slug)

app/views/layouts/instance_statistics.html.haml

-- page_title _('Instance Statistics')
-- header_title _('Instance Statistics'), instance_statistics_root_path
+- page_title _('Analytics')
+- header_title _('Analytics'), instance_statistics_root_path
 - nav 'instance_statistics'
 - @left_sidebar = true