Merge branch '33646-eliminate-cycle-analytics-total-stage' into 'master'

Remove Value Stream Total stage

Closes #33646

See merge request gitlab-org/gitlab!42345

app/assets/javascripts/cycle_analytics/cycle_analytics_store.js

@@ -23,9 +23,6 @@ const EMPTY_STAGE_TEXTS = {
   staging: __(
     'The staging stage shows the time between merging the MR and deploying code to the production environment. The data will be automatically added once you deploy to production for the first time.',
   ),
-  production: __(
-    'The total stage shows the time it takes between creating an issue and deploying the code to production. The data will be automatically added once you have completed the full idea to production cycle.',
-  ),
 };
 export default {

app/models/cycle_analytics/level_base.rb

@@ -2,7 +2,7 @@
 module CycleAnalytics
   module LevelBase
-    STAGES = %i[issue plan code test review staging production].freeze
+    STAGES = %i[issue plan code test review staging].freeze
     def all_medians_by_stage
       STAGES.each_with_object({}) do |stage_name, medians_per_stage|

changelogs/unreleased/33646-eliminate-cycle-analytics-total-stage.yml

+---
+title: Remove Value Stream Total stage
+merge_request: 42345
+author:
+type: removed

db/post_migrate/20200916081749_remove_cycle_analytics_total_stage_data.rb

+# frozen_string_literal: true
+
+class RemoveCycleAnalyticsTotalStageData < ActiveRecord::Migration[6.0]
+  DOWNTIME = false
+
+  def up
+    execute("DELETE FROM analytics_cycle_analytics_group_stages WHERE name='production'")
+    execute("DELETE FROM analytics_cycle_analytics_project_stages WHERE name='production'")
+  end
+
+  def down
+    # Migration is irreversible
+  end
+end

db/schema_migrations/20200916081749

+dde7a29268d925044d59455db87bfc1aa617eec6e30df1cc9dc531b52c909fe1
\ No newline at end of file

doc/user/analytics/value_stream_analytics.md

@@ -45,8 +45,6 @@ There are seven stages that are tracked as part of the Value Stream Analytics ca
   - Time spent on code review
 - **Staging** (Continuous Deployment)
   - Time between merging and deploying to production
-- **Total** (Total)
-  - Total lifecycle time. That is, the velocity of the project or team. [Previously known](https://gitlab.com/gitlab-org/gitlab/-/issues/38317) as **Production**.
 ## Filter the analytics data
@@ -95,7 +93,7 @@ Note: A commit is associated with an issue by [crosslinking](../project/issues/c
 ## How the stages are measured
 Value Stream Analytics records stage time and data based on the project issues with the
-exception of the staging and total stages, where only data deployed to
+exception of the staging stage, where only data deployed to
 production are measured.
 Specifically, if your CI is not set up and you have not defined a `production`
@@ -112,7 +110,6 @@ Each stage of Value Stream Analytics is further described in the table below.
 | Test      | Measures the median time to run the entire pipeline for that project. It's related to the time GitLab CI/CD takes to run every job for the commits pushed to that merge request defined in the previous stage. It is basically the start->finish time for all pipelines. |
 | Review    | Measures the median time taken to review the merge request that has a closing issue pattern, between its creation and until it's merged. |
 | Staging   | Measures the median time between merging the merge request with a closing issue pattern until the very first deployment to production. It's tracked by the environment set to `production` or matching `production/*` (case-sensitive, `Production` won't work) in your GitLab CI/CD configuration. If there isn't a production environment, this is not tracked. |
-| Total | The sum of all time (medians) taken to run the entire process, from issue creation to deploying the code to production. [Previously known](https://gitlab.com/gitlab-org/gitlab/-/issues/38317) as **Production**. |
 How this works, behind the scenes:
@@ -131,7 +128,7 @@ Value Stream Analytics dashboard will not present any data for:
 - Merge requests that do not close an issue.
 - Issues not labeled with a label present in the Issue Board or for issues not assigned a milestone.
-- Staging and production stages, if the project has no `production` or `production/*`
+- Staging stage, if the project has no `production` or `production/*`
   environment.
 ## Example workflow
@@ -158,9 +155,6 @@ environments is configured.
    request at 19:00. (stop of **Review** stage / start of **Staging** stage).
 1. Now that the merge request is merged, a deployment to the `production`
    environment starts and finishes at 19:30 (stop of **Staging** stage).
-1. The cycle completes and the sum of the median times of the previous stages
-   is recorded to the **Total** stage. That is the time between creating an
-   issue and deploying its relevant merge request to production.
 From the above example you can conclude the time it took each stage to complete
 as long as their total time:
@@ -171,10 +165,6 @@ as long as their total time:
 - **Test**: 5min
 - **Review**: 5h (19:00 - 14:00)
 - **Staging**: 30min (19:30 - 19:00)
-- **Total**: Since this stage measures the sum of median time of all
-  previous stages, we cannot calculate it if we don't know the status of the
-  stages before. In case this is the very first cycle that is run in the project,
-  then the **Total** time is 10h 30min (19:30 - 09:00)
 A few notes:

ee/app/assets/javascripts/analytics/cycle_analytics/constants.js

 import { gray10 } from '@gitlab/ui/scss_to_js/scss_variables';
 import { __ } from '~/locale';
-import { capitalizeFirstCharacter } from '~/lib/utils/text_utility';
 export const PROJECTS_PER_PAGE = 50;
@@ -27,12 +26,9 @@ export const EMPTY_STAGE_TEXT = {
   staging: __(
     'The staging stage shows the time between merging the MR and deploying code to the production environment. The data will be automatically added once you deploy to production for the first time.',
   ),
-  production: __(
-    'The production stage shows the total time it takes between creating an issue and deploying the code to production. The data will be automatically added once you have completed the full idea to production cycle.',
-  ),
 };
-export const DEFAULT_STAGE_NAMES = [...Object.keys(EMPTY_STAGE_TEXT), 'total'];
+export const DEFAULT_STAGE_NAMES = [...Object.keys(EMPTY_STAGE_TEXT)];
 export const TASKS_BY_TYPE_SUBJECT_ISSUE = 'Issue';
 export const TASKS_BY_TYPE_SUBJECT_MERGE_REQUEST = 'MergeRequest';
@@ -59,25 +55,6 @@ export const STAGE_ACTIONS = {
   ADD_STAGE: 'showAddStageForm',
 };
-export const STAGE_NAME = {
-  TOTAL: 'total',
-  PRODUCTION: 'production',
-  OVERVIEW: 'overview',
-};
-
-/**
- * An object containing capitalized stages names
- * i.e. { TOTAL: 'total' } => { TOTAL: 'Total' }
- */
-export const CAPITALIZED_STAGE_NAME = Object.keys(STAGE_NAME).reduce((acc, stage) => {
-  return {
-    ...acc,
-    [stage]: capitalizeFirstCharacter(STAGE_NAME[stage]),
-  };
-}, {});
-
-export const PATH_HOME_ICON = 'home';
-
 export const DEFAULT_VALUE_STREAM_ID = 'default';
 export const OVERVIEW_METRICS = {

ee/app/assets/javascripts/analytics/cycle_analytics/utils.js

-import { isNumber, sortBy } from 'lodash';
+import { isNumber } from 'lodash';
 import dateFormat from 'dateformat';
 import { s__, sprintf } from '~/locale';
 import { convertObjectPropsToCamelCase } from '~/lib/utils/common_utils';
@@ -13,7 +13,6 @@ import {
   parseSeconds,
 } from '~/lib/utils/datetime_utility';
 import { dateFormats } from '../shared/constants';
-import { STAGE_NAME, CAPITALIZED_STAGE_NAME, PATH_HOME_ICON } from './constants';
 import { toYmd } from '../shared/utils';
 const EVENT_TYPE_LABEL = 'label';
@@ -85,10 +84,8 @@ export const isPersistedStage = ({ custom, id }) => custom || isNumber(id);
  */
 const stageUrlSlug = ({ id, title, custom = false }) => {
   if (custom) return id;
-  // We still use 'production' as the id to access this stage, even though the title is 'Total'
-  return title.toLowerCase() === STAGE_NAME.TOTAL
-    ? STAGE_NAME.PRODUCTION
-    : convertToSnakeCase(title);
+
+  return convertToSnakeCase(title);
 };
 export const transformRawStages = (stages = []) =>
@@ -342,9 +339,6 @@ export const isStageNameExistsError = ({ status, errors }) =>
  * Takes the stages and median data, combined with the selected stage, to build an
  * array which is formatted to proivde the data required for the path navigation.
  *
- * The stage named 'Total' is renamed to 'Overview', it's configured to have
- * the 'home' icon - and is moved to the front of the array.
- *
  * @param {Array} stages - The stages available to the group / project
  * @param {Object} medians - The median values for the stages available to the group / project
  * @param {Object} selectedStage - The currently selected stage
@@ -357,20 +351,17 @@ export const transformStagesForPathNavigation = ({ stages, medians, selectedStag
       hoursPerDay: 24,
       limitToDays: true,
     });
-    const isTotalStage = stage.title === CAPITALIZED_STAGE_NAME.TOTAL;
     return {
       ...stage,
       metric: days ? sprintf(s__('ValueStreamAnalytics|%{days}d'), { days }) : null,
       selected: stage.title === selectedStage.title,
-      title: isTotalStage ? CAPITALIZED_STAGE_NAME.OVERVIEW : stage.title,
-      icon: isTotalStage ? PATH_HOME_ICON : null,
+      title: stage.title,
+      icon: null,
     };
   });
-  return sortBy(formattedStages, stage =>
-    stage.title === CAPITALIZED_STAGE_NAME.OVERVIEW ? 0 : 1,
-  );
+  return formattedStages;
 };
 /**

ee/spec/features/analytics/cycle_analytics/cycle_analytics_spec.rb

@@ -129,11 +129,11 @@ RSpec.describe 'Group Value Stream Analytics', :js do
         to_index: to)
     end
-    default_stage_order = %w[Issue Plan Code Test Review Staging Total].freeze
-    default_custom_stage_order = %w[Issue Plan Code Test Review Staging Total Cool\ beans].freeze
-    stages_near_middle_swapped = %w[Issue Plan Test Code Review Staging Total Cool\ beans].freeze
-    stage_dragged_to_top = %w[Review Issue Plan Code Test Staging Total Cool\ beans].freeze
-    stage_dragged_to_bottom = %w[Issue Plan Code Test Staging Total Cool\ beans Review].freeze
+    default_stage_order = %w[Issue Plan Code Test Review Staging].freeze
+    default_custom_stage_order = %w[Issue Plan Code Test Review Staging Cool\ beans].freeze
+    stages_near_middle_swapped = %w[Issue Plan Test Code Review Staging Cool\ beans].freeze
+    stage_dragged_to_top = %w[Review Issue Plan Code Test Staging Cool\ beans].freeze
+    stage_dragged_to_bottom = %w[Issue Plan Code Test Staging Cool\ beans Review].freeze
     shared_examples 'manual ordering disabled' do
       it 'does not allow stages to be draggable', :js do

ee/spec/features/groups/analytics/cycle_analytics/filters_and_data_spec.rb

@@ -142,7 +142,7 @@ RSpec.describe 'Group value stream analytics filters and data', :js do
       it 'displays the default list of stages' do
         stage_nav = page.find(stage_nav_selector)
-        %w[Issue Plan Code Test Review Staging Total].each do |item|
+        %w[Issue Plan Code Test Review Staging].each do |item|
           string_id = "CycleAnalytics|#{item}"
           expect(stage_nav).to have_content(s_(string_id))
         end
@@ -163,7 +163,7 @@ RSpec.describe 'Group value stream analytics filters and data', :js do
       it 'displays the default list of stages' do
         path_nav = page.find(path_nav_selector)
-        %w[Issue Plan Code Test Review Staging Overview].each do |item|
+        %w[Issue Plan Code Test Review Staging].each do |item|
           string_id = "CycleAnalytics|#{item}"
           expect(path_nav).to have_content(s_(string_id))
         end
@@ -303,8 +303,7 @@ RSpec.describe 'Group value stream analytics filters and data', :js do
       { title: 'Code', description: 'Time until first merge request', events_count: 1, median: 'about 5 hours' },
       { title: 'Test', description: 'Total test time for all commits/merges', events_count: 0, median: 'Not enough data' },
       { title: 'Review', description: 'Time between merge request creation and merge/close', events_count: 1, median: 'about 1 hour' },
-      { title: 'Staging', description: 'From merge request merge until deploy to production', events_count: 1, median: 'about 1 hour' },
-      { title: 'Total', description: 'From issue creation until deploy to production', events_count: 1, median: '5 days' }
+      { title: 'Staging', description: 'From merge request merge until deploy to production', events_count: 1, median: 'about 1 hour' }
     ]
     it 'each stage will have median values', :sidekiq_might_not_need_inline do

ee/spec/frontend/analytics/cycle_analytics/components/stage_event_list_spec.js

@@ -14,8 +14,6 @@ import {
   testEvents,
   stagingStage,
   stagingEvents,
-  totalStage,
-  totalEvents,
   codeStage,
   codeEvents,
 } from '../mock_data';
@@ -88,7 +86,6 @@ describe('Stage', () => {
       ${'Test'}    | ${testStage}
       ${'Code'}    | ${codeStage}
       ${'Staging'} | ${stagingStage}
-      ${'Total'}   | ${totalStage}
     `('$name stage will render the stage description', ({ stage }) => {
       wrapper = createComponent({ props: { stage, events: [] } });
       expect(wrapper.find($sel.description).text()).toEqual(stage.description);
@@ -100,7 +97,6 @@ describe('Stage', () => {
       ${'Plan'}   | ${planStage}   | ${planEvents}
       ${'Review'} | ${reviewStage} | ${reviewEvents}
       ${'Code'}   | ${codeStage}   | ${codeEvents}
-      ${'Total'}  | ${totalStage}  | ${totalEvents}
     `('$name stage will render the list of events', ({ stage, eventList }) => {
       // stages generated from fixtures may not have events
       const events = eventList.length ? eventList : generateEvents(5);
@@ -120,7 +116,6 @@ describe('Stage', () => {
       ${'Plan'}   | ${planStage}   | ${planEvents}
       ${'Review'} | ${reviewStage} | ${reviewEvents}
       ${'Code'}   | ${codeStage}   | ${codeEvents}
-      ${'Total'}  | ${totalStage}  | ${totalEvents}
     `('$name stage will render the items as StageEventItems', ({ stage, eventList }) => {
       wrapper = createComponent({ props: { events: eventList, stage }, stubs: mockStubs });
       expect(wrapper.find(StageEventItem).exists()).toBe(true);

ee/spec/frontend/analytics/cycle_analytics/mock_data.js

@@ -75,13 +75,12 @@ export const reviewStage = getStageByTitle(dummyState.stages, 'review');
 export const codeStage = getStageByTitle(dummyState.stages, 'code');
 export const testStage = getStageByTitle(dummyState.stages, 'test');
 export const stagingStage = getStageByTitle(dummyState.stages, 'staging');
-export const totalStage = getStageByTitle(dummyState.stages, 'total');
 export const allowedStages = [issueStage, planStage, codeStage];
 const deepCamelCase = obj => convertObjectPropsToCamelCase(obj, { deep: true });
-export const defaultStages = ['issue', 'plan', 'review', 'code', 'test', 'staging', 'production'];
+export const defaultStages = ['issue', 'plan', 'review', 'code', 'test', 'staging'];
 const stageFixtures = defaultStages.reduce((acc, stage) => {
   const events = getJSONFixture(fixtureEndpoints.stageEvents(stage));
@@ -111,14 +110,12 @@ export const stageMediansWithNumericIds = defaultStages.reduce((acc, stage) => {
 export const endDate = new Date(2019, 0, 14);
 export const startDate = getDateInPast(endDate, DEFAULT_DAYS_IN_PAST);
-export const rawIssueEvents = stageFixtures.issue;
 export const issueEvents = deepCamelCase(stageFixtures.issue);
 export const planEvents = deepCamelCase(stageFixtures.plan);
 export const reviewEvents = deepCamelCase(stageFixtures.review);
 export const codeEvents = deepCamelCase(stageFixtures.code);
 export const testEvents = deepCamelCase(stageFixtures.test);
 export const stagingEvents = deepCamelCase(stageFixtures.staging);
-export const totalEvents = deepCamelCase(stageFixtures.production);
 export const rawCustomStage = {
   title: 'Coolest beans stage',
   hidden: false,

ee/spec/frontend/analytics/cycle_analytics/store/mutations_spec.js

@@ -7,7 +7,6 @@ import {
   codeStage,
   stagingStage,
   reviewStage,
-  totalStage,
   startDate,
   endDate,
   selectedProjects,
@@ -130,7 +129,7 @@ describe('Cycle analytics mutations', () => {
       });
       it('will convert the stats object to stages', () => {
-        [issueStage, planStage, codeStage, stagingStage, reviewStage, totalStage].forEach(stage => {
+        [issueStage, planStage, codeStage, stagingStage, reviewStage].forEach(stage => {
           expect(state.stages).toContainEqual(stage);
         });
       });

ee/spec/frontend/analytics/cycle_analytics/utils_spec.js

@@ -18,7 +18,6 @@ import {
   prepareTimeMetricsData,
 } from 'ee/analytics/cycle_analytics/utils';
 import { toYmd } from 'ee/analytics/shared/utils';
-import { CAPITALIZED_STAGE_NAME, PATH_HOME_ICON } from 'ee/analytics/cycle_analytics/constants';
 import { getDatesInRange } from '~/lib/utils/datetime_utility';
 import { slugify } from '~/lib/utils/text_utility';
 import {
@@ -36,7 +35,6 @@ import {
   rawTasksByTypeData,
   allowedStages,
   stageMediansWithNumericIds,
-  totalStage,
   pathNavIssueMetric,
   timeMetricsData,
 } from './mock_data';
@@ -315,7 +313,7 @@ describe('Cycle analytics utils', () => {
   });
   describe('transformStagesForPathNavigation', () => {
-    const stages = [...allowedStages, totalStage];
+    const stages = allowedStages;
     const response = transformStagesForPathNavigation({
       stages,
       medians: stageMediansWithNumericIds,
@@ -339,22 +337,6 @@ describe('Cycle analytics utils', () => {
         expect(issue.metric).toEqual(pathNavIssueMetric);
       });
-
-      describe(`${CAPITALIZED_STAGE_NAME.OVERVIEW} stage specific changes`, () => {
-        const overview = response.filter(stage => stage.name === CAPITALIZED_STAGE_NAME.TOTAL)[0];
-
-        it(`renames '${CAPITALIZED_STAGE_NAME.TOTAL}' stage title to '${CAPITALIZED_STAGE_NAME.OVERVIEW}'`, () => {
-          expect(overview.title).toEqual(CAPITALIZED_STAGE_NAME.OVERVIEW);
-        });
-
-        it('includes the correct icon', () => {
-          expect(overview.icon).toEqual(PATH_HOME_ICON);
-        });
-
-        it(`moves the stage to the front`, () => {
-          expect(response[0]).toEqual(overview);
-        });
-      });
     });
   });

ee/spec/migrations/remove_cycle_analytics_total_stage_data_spec.rb

+# frozen_string_literal: true
+
+require 'spec_helper'
+require_migration!
+
+RSpec.describe RemoveCycleAnalyticsTotalStageData, :migration do
+  let(:group_stages_table) { table(:analytics_cycle_analytics_group_stages) }
+  let(:project_stages_table) { table(:analytics_cycle_analytics_project_stages) }
+  let(:namespaces) { table(:namespaces) }
+  let(:projects) { table(:projects) }
+  let(:group_value_streams_table) { table(:analytics_cycle_analytics_group_value_streams) }
+
+  let(:project) { projects.create!(id: 12058473, namespace_id: group.id, name: 'gitlab', path: 'gitlab') }
+  let(:group) { namespaces.create!(name: 'foo', path: 'foo') }
+  let(:group_value_stream) { group_value_streams_table.create!(group_id: group.id, name: 'default') }
+
+  before do
+    group_stages_table.create!(
+      id: 1,
+      name: 'production',
+      group_value_stream_id: group_value_stream.id,
+      group_id: group.id,
+      start_event_identifier: 1,
+      end_event_identifier: 2
+    )
+
+    group_stages_table.create!(
+      id: 2,
+      name: 'plan',
+      group_value_stream_id: group_value_stream.id,
+      group_id: group.id,
+      start_event_identifier: 1,
+      end_event_identifier: 2
+    )
+
+    project_stages_table.create!(
+      id: 1,
+      name: 'production',
+      project_id: project.id,
+      start_event_identifier: 1,
+      end_event_identifier: 2
+    )
+
+    project_stages_table.create!(
+      id: 2,
+      name: 'plan',
+      project_id: project.id,
+      start_event_identifier: 1,
+      end_event_identifier: 2
+    )
+  end
+
+  it 'removes "production" stage info and keeps other stages' do
+    migrate!
+
+    expect(group_stages_table.all.map(&:id)).to eq [2]
+    expect(project_stages_table.all.map(&:id)).to eq [2]
+  end
+end

lib/gitlab/analytics/cycle_analytics/default_stages.rb

@@ -18,8 +18,7 @@ module Gitlab
             params_for_code_stage,
             params_for_test_stage,
             params_for_review_stage,
-            params_for_staging_stage,
-            params_for_production_stage
+            params_for_staging_stage
           ]
         end
@@ -86,16 +85,6 @@ module Gitlab
             end_event_identifier: :merge_request_first_deployed_to_production
           }
         end
-
-        def self.params_for_production_stage
-          {
-            name: 'production',
-            custom: false,
-            relative_position: 7,
-            start_event_identifier: :issue_created,
-            end_event_identifier: :production_stage_end
-          }
-        end
       end
     end
   end

lib/gitlab/analytics/cycle_analytics/stage_events/production_stage_end.rb

@@ -6,7 +6,7 @@ module Gitlab
       module StageEvents
         class ProductionStageEnd < StageEvent
           def self.name
-            _("Issue first depoloyed to production")
+            _("Issue first deployed to production")
           end
           def self.identifier

lib/gitlab/cycle_analytics/production_stage.rb

-# frozen_string_literal: true
-
-module Gitlab
-  module CycleAnalytics
-    class ProductionStage < BaseStage
-      include ProductionHelper
-
-      def start_time_attrs
-        @start_time_attrs ||= issue_table[:created_at]
-      end
-
-      def end_time_attrs
-        @end_time_attrs ||= mr_metrics_table[:first_deployed_to_production_at]
-      end
-
-      def name
-        :production
-      end
-
-      def title
-        s_('CycleAnalyticsStage|Total')
-      end
-
-      def legend
-        _("Related Issues")
-      end
-
-      def description
-        _("From issue creation until deploy to production")
-      end
-
-      def query
-        # Limit to merge requests that have been deployed to production after `@from`
-        query.where(mr_metrics_table[:first_deployed_to_production_at].gteq(@from))
-      end
-    end
-  end
-end

locale/gitlab.pot

@@ -13905,7 +13905,7 @@ msgstr ""
 msgid "Issue events"
 msgstr ""
-msgid "Issue first depoloyed to production"
+msgid "Issue first deployed to production"
 msgstr ""
 msgid "Issue label"
@@ -25306,9 +25306,6 @@ msgstr ""
 msgid "The private key to use when a client certificate is provided. This value is encrypted at rest."
 msgstr ""
-msgid "The production stage shows the total time it takes between creating an issue and deploying the code to production. The data will be automatically added once you have completed the full idea to production cycle."
-msgstr ""
-
 msgid "The project can be accessed by any logged in user."
 msgstr ""
@@ -25399,9 +25396,6 @@ msgstr ""
 msgid "The time taken by each data entry gathered by that stage."
 msgstr ""
-msgid "The total stage shows the time it takes between creating an issue and deploying the code to production. The data will be automatically added once you have completed the full idea to production cycle."
-msgstr ""
-
 msgid "The update action will time out after %{number_of_minutes} minutes. For big repositories, use a clone/push combination."
 msgstr ""

spec/features/cycle_analytics_spec.rb

@@ -76,9 +76,6 @@ RSpec.describe 'Value Stream Analytics', :js do
         click_stage('Staging')
         expect_build_to_be_present
-
-        click_stage('Total')
-        expect_issue_to_be_present
       end
       context "when I change the time period observed" do