We had concerns about the cached values on Redis with the previous two
releases strategy:

First release (this commit):
  - Create new encrypted fields in the database.
  - Start populating new encrypted fields, read the encrypted fields or
    fallback to the plaintext fields.
  - Backfill the data removing the plaintext fields to the encrypted
    fields.
Second release:
  - Remove the virtual attribute (created in step 2).
  - Drop plaintext columns from the database (empty columns after
    step 3).

We end up with a better strategy only using migration scripts in one
release:
  - Pre-deployment migration: Add columns required for storing encrypted
    values.
  - Pre-deployment migration: Store the encrypted values in the new
    columns.
  - Post-deployment migration: Remove the old unencrypted columns

This is the plan to encrypt the plaintext tokens:

First release (this commit):
  1. Create new encrypted fields in the database.
  2. Start populating new encrypted fields, read the encrypted fields or
     fallback to the plaintext fields.
  3. Backfill the data removing the plaintext fields to the encrypted fields.

Second release:
  4. Remove the virtual attribute (created in step 2).
  5. Drop plaintext columns from the database (empty columns after step 3).

The Sidekiq job `RemoveExpiredMembersWorker` was failing to run in
production because it was hitting statement timeouts because it was
scanning all rows in order. On staging, where it used to scan 4 million
rows, adding an index brought this down to only a few hundred rows.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/67286

Jan Provaznik authored 5 years ago and Andreas Brandl committed 5 years ago

This fixes previously added migration which caused timeouts on
big events table.

It consists of two parts:

1. Redirecting users to the configured external storage
1. Allowing the external storage to request the static object(s)
   on behalf of the user by means of specific tokens

Part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6829

On our dev instance, /admin/applications as not loading because:

1. There was an unindexed query by `application_id`.
2. There was an expensive query that attempted to load 1 million
   unique entries via ActiveRecord just to find the unique count.

We fix the first issue by adding an index for that column.

We fix the second issue with a simple SELECT COUNT(DISTINCT
resource_owner_id) SQL query.

In addition, we add pagination to avoid loading more than 20
applications at once.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/67228

Detect if pipeline runs for a GitHub pull request

When using a mirror for CI/CD only we register a pull_request
webhook. When a pull_request webhook is received, if the
source branch SHA matches the actual head of the branch in the
repository we create immediately a new pipeline for the
external pull request. Otherwise we store the
pull request info for when the push webhook is received.

When using "only/except: external_pull_requests" we can detect
if the pipeline has a open pull request on GitHub and create or
not the job based on that.

Since it is not possible to dynamically detect if a job is automatically
cancellable or not, a this new attribute is necessary. Moreover, it let
the maintainer of the repo to adjust the behaviour of the auto cancellation
feature to match exactly what he needs.

Kerri Miller authored 5 years ago and Nick Thomas committed 5 years ago

These are the structural changes for supporting the EE feature of moving
"code_owner_approval_required" state from existing on a project to being
on the protected branches individually, allowing for CODEOWNER
validation on push events.

This speeds up the following query:

```sql
SELECT  users.* FROM users WHERE users.unconfirmed_email = ? ORDER BY
users.id ASC LIMIT 1
```

Presumably, this is a query coming from Devise.

Context is https://gitlab.com/gitlab-org/gitlab-ce/issues/66958.

This creates a partial index intended to speed up queries on
`ci_builds`. Particularly, `gitlab-monitor` has rather heavy queries.
Those have been changed to only look back 7 days and benefit from this
index tremendously.

Relates to
https://gitlab.com/gitlab-org/gitlab-exporter/merge_requests/101.

Jan Provaznik authored 5 years ago and Thong Kuah committed 5 years ago

Creates new event when an epic is created, closed, reopened or
commented.

Etienne Baqué authored 5 years ago and Andreas Brandl committed 5 years ago

Modified schema via migrations.
Added one-to-one relationship between the two models.
Added changelog file

This column is not present in `db/schema.rb` and hence needs to be
removed conditionally.

See https://gitlab.com/gitlab-org/gitlab-ce/issues/66901 for background

This adjusts the partial condition for an index. The index is intended
to be used when counting active users with `ghost IS NOT TRUE AND
bot_type IS NULL`.

With the current index, this wasn't working as the partial condition
didn't match the query: `ghost <> TRUE` is not semantically equivalent
to `ghost IS NOT TRUE` (null semantics).

The reason we add an index particularly intended for EE is that the EE
query is going to have the additional part `AND bot_type IS NULL`
whereas the CE query doesn't. Logically, it'd be enough to have an index
for `ghost IS NOT TRUE`. However, on GitLab.com, the query planner makes
poor choices when the additional `AND bot_type IS NULL` part is present:
It goes for the index on `bot_type` and doesn't use the partial index.

Note the existing index isn't being used at all according to GitLab.com
index statistics. Hence we can first remove it and don't have to worry
about the window of time without an index.

Relates to https://gitlab.com/gitlab-org/gitlab-ce/issues/66770

Felipe Artur authored 5 years ago and Andreas Brandl committed 5 years ago

Rename epic column state to state_id to be consistent with
issues and merge requests

Now we have cluster_id, state index, we don't the cluster_id index as
well.

For Environment.deployed_to_cluster

Persists if a board list is collapsed for each user.

A new param with_security_reports was added to
GET /groups/:id/projects API and the code to
support this logic in GroupProjectsFinder and
Project model. Also, a DB index was added to
ci_job_artifacts table to speed up the search
of security reports artifacts for projects

Adam Hegyi authored 5 years ago and Andreas Brandl committed 5 years ago

This change sets NOT NULL constraint to users.private profile.

closes https://gitlab.com/gitlab-org/gitlab-ce/issues/57538

Added migrations to make token column accepting null values and to add
encrypted token column.

Removed unused method for name setter method