Skip to content

Finish migration to the new events setup

yorickpeterse-staging requested to merge events-migration-cleanup into master

This MR cleans up / finishes the process of migrating events to the new database setup, including the removal of the old migrations table. For GitLab.com we expect to save at least 165 GB of storage with these changes.

Related issues/MRs:

Migration Output

This excludes the steal migration since it doesn't produce any meaningful output.

==  SwapEventMigrationTables: migrating =======================================
-- rename_table(:events, :events_old)
   -> 0.1747s
-- rename_table(:events_for_migration, :events)
   -> 0.0257s
-- rename_table(:events_old, :events_for_migration)
   -> 0.0345s
==  SwapEventMigrationTables: migrated (0.2358s) ==============================

==  DropEventsForMigrationTable: migrating ====================================
-- transaction()
-- drop_table(:events_for_migration)
   -> 0.0812s
   -> 3.2884s
==  DropEventsForMigrationTable: migrated (3.3997s) ===========================

Migration Timings

Migration Time
SwapEventMigrationTables 0.2 seconds
DropEventsForMigrationTable 3.4 seconds

Production Impact

The process should have no impact visible to users. The migrations can execute in parallel to the running application. This was verified by running a simple loop that queries events every 1 second, which produced no errors.

Size Impact

This migration will save around 165 GB based on recent estimates. I re-verified this today to be certain using a replica and a separate database restored from production. The replica had 941 GB of data, while the restored host (after the migrations) has 759 GB of data; 182 GB less.

Database Checklist

When adding migrations:

  • Updated db/schema.rb
  • Added a down method so the migration can be reverted
  • Added the output of the migration(s) to the MR body
  • Added the execution time of the migration(s) to the MR body
  • Made sure the migration won't interfere with a running GitLab cluster, for example by disabling transactions for long running migrations

When removing columns, tables, indexes or other structures:

  • Removed these in a post-deployment migration
  • Made sure the application no longer uses (or ignores) these structures

General Checklist

Edited by username-removed-443319

Merge request reports