Skip to content

Change index on ci_builds to optimize Jobs Controller

Gregory Stark requested to merge add-ci-builds-index-for-jobscontroller into master

Change the index on project_id to be on project_id, id to enable paged listing of ci_builds for a given project.

Database Checklist

When adding migrations:

  • Updated db/schema.rb
  • Added a down method so the migration can be reverted
  • Added the output of the migration(s) to the MR body
  • Added the execution time of the migration(s) to the MR body
  • Added tests for the migration in spec/migrations if necessary (e.g. when migrating data)
  • Made sure the migration won't interfere with a running GitLab cluster, for example by disabling transactions for long running migrations

When adding indexes:

  • Described the need for these indexes in the MR body
  • Made sure existing indexes can not be reused instead

When removing columns, tables, indexes or other structures:

  • Removed these in a post-deployment migration
  • Made sure the application no longer uses (or ignores) these structures

General Checklist

stark@tweedle:~/gitlab/gdk/gitlab-development-kit/gitlab$ bundle exec rake db:migrate
== 20170927095921 AddCiBuildsIndexForJobscontroller: migrating ================
-- index_exists?(:ci_builds, [:project_id, :id])
   -> 0.0079s
-- transaction_open?()
   -> 0.0000s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- add_index(:ci_builds, [:project_id, :id], {:algorithm=>:concurrently})
   -> 0.0996s
-- index_exists?(:ci_builds, :project_id)
   -> 0.0089s
-- transaction_open?()
   -> 0.0000s
-- select_one("SELECT current_setting('server_version_num') AS v")
   -> 0.0004s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- remove_index(:ci_builds, {:algorithm=>:concurrently, :column=>:project_id})
   -> 0.0114s
== 20170927095921 AddCiBuildsIndexForJobscontroller: migrated (0.1294s) =======

The old query plan:

gitlabhq_production=# explain SELECT "ci_builds".* FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."project_id" = 7764 AND "ci_builds"."status" IN ('pending', 'running', 'success', 'failed', 'canceled', 'skipped', 'manual') ORDER BY id DESC LIMIT 30 OFFSET 0;
                                                                                  QUERY PLAN                                                                                  
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.56..2903.76 rows=30 width=1373)
   ->  Index Scan Backward using ci_builds_pkey on ci_builds  (cost=0.56..8183160.24 rows=84560 width=1373)
         Filter: (((type)::text = 'Ci::Build'::text) AND (project_id = 7764) AND ((status)::text = ANY ('{pending,running,success,failed,canceled,skipped,manual}'::text[])))
(3 rows)

The new query plan:

gitlabhq_development***=# explain SELECT "ci_builds".* FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."project_id" = 7764 AND "ci_builds"."status" IN ('pending', 'running', 'success', 'failed', 'canceled', 'skipped', 'manual') ORDER BY created_at DESC LIMIT 30 OFFSET 0;
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                      QUERY PLAN                                                                      │
├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Limit  (cost=0.28..8.30 rows=1 width=501)                                                                                                            │
│   ->  Index Scan Backward using tmp2 on ci_builds  (cost=0.28..8.30 rows=1 width=501)                                                                │
│         Index Cond: (project_id = 7764)                                                                                                              │
│         Filter: (((type)::text = 'Ci::Build'::text) AND ((status)::text = ANY ('{pending,running,success,failed,canceled,skipped,manual}'::text[]))) │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Edited by yorickpeterse-staging

Merge request reports