We had concerns about the cached values on Redis with the previous two
releases strategy:

First release (this commit):
  - Create new encrypted fields in the database.
  - Start populating new encrypted fields, read the encrypted fields or
    fallback to the plaintext fields.
  - Backfill the data removing the plaintext fields to the encrypted
    fields.
Second release:
  - Remove the virtual attribute (created in step 2).
  - Drop plaintext columns from the database (empty columns after
    step 3).

We end up with a better strategy only using migration scripts in one
release:
  - Pre-deployment migration: Add columns required for storing encrypted
    values.
  - Pre-deployment migration: Store the encrypted values in the new
    columns.
  - Post-deployment migration: Remove the old unencrypted columns

This is the plan to encrypt the plaintext tokens:

First release (this commit):
  1. Create new encrypted fields in the database.
  2. Start populating new encrypted fields, read the encrypted fields or
     fallback to the plaintext fields.
  3. Backfill the data removing the plaintext fields to the encrypted fields.

Second release:
  4. Remove the virtual attribute (created in step 2).
  5. Drop plaintext columns from the database (empty columns after step 3).

Dmitriy Zaporozhets authored 5 years ago and Peter Leitzen committed 5 years ago

Signed-off-by: Dmitriy Zaporozhets <dmitriy.zaporozhets@gmail.com>

It consists of two parts:

1. Redirecting users to the configured external storage
1. Allowing the external storage to request the static object(s)
   on behalf of the user by means of specific tokens

Part of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6829

This is to accomodate prepended modules.

User images and videos will get proxied through
the Camo server in order to keep malicious
sites from collecting the IP address of users.

Jeremy Jackson authored 5 years ago and Mayra Cabrera committed 5 years ago

This introduces several changes, but these are all just ported from the
EE project.

- This will make it easy to identify the project even if admins change
the name of the project or move it.

Add method to store session ids by ip

Add new specs for storing session ids

Add cleaning up records after login

Add retrieving anonymous sessions

Add login recaptcha setting

Add new setting to sessions controller

Add conditions for showing captcha

Add sessions controller specs

Add admin settings specs for login protection

Add new settings to api

Add stub to devise spec

Add new translation key

Add cr remarks

Rename class call

Add cr remarks

Change if-clause for consistency

Add cr remarks

Add code review remarks

Refactor AnonymousSession class

Add changelog entry

Move AnonymousSession class to lib

Move store unauthenticated sessions to sessions controller

Move link to recaptcha info

Regenerate text file

Improve copy on the spam page

Change action filter for storing anonymous sessions

Fix rubocop offences

Add code review remarks

Reuben Pereira authored 5 years ago and James Lopez committed 5 years ago

- The most common use case for qualified_domain_validator currently is
to allow blank ([]) but not allow nil. Modify the
qualified_domain_validator to support this use case.

Reuben Pereira authored 5 years ago and Mayra Cabrera committed 5 years ago

Signed-off-by: Istvan szalai <istvan.szalai@savoirfairelinux.com>

Loading `ApplicationSetting` from Redis was responsible for at least 50%
of the CPU load of the Redis cluster on GitLab.com. Since these values
generally don't change very much, we can load this from the database and
cache it in memory, skipping Redis altogther. We use
`ActiveSupport::Cache::MemoryStore` as a drop-in replacement for
`RedisCacheStore` even though we probably don't need synchronized access
within `Thread.current`.

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/63977

The circuit breaker itself got removed a while ago, when that happened
some parts got left behind. Using grep old stale settings and
validations were found and are now removed.

Storing this key in secrets.yml was a bad idea,
it would require users using HA setups to manually
replicate secrets across nodes during update,
it also needed support from omnibus package

* Revert "Generate Let's Encrypt private key"
  This reverts commit 444959bf.

* Add Let's Encrypt private key to settings
  as encrypted attribute

* Generate Let's Encrypt private key
  in database migration

Store Let's Encrypt account email in application settings
Also add explicit terms of service consent

Martin Wortschack authored 5 years ago and Nick Thomas committed 5 years ago

- Update PO file

Thong Kuah authored 5 years ago and James Lopez committed 5 years ago

Renamed UrlValidator to AddressableUrlValidator to avoid 'url:' naming collision with ActiveModel::Validations::UrlValidator in 'validates' statement.
Make use of the options attribute of the parent class ActiveModel::EachValidator.
Add more options: allow_nil, allow_blank, message.
Renamed 'protocols' option to 'schemes' to match the option naming from UrlValidator.

It could happen that there's a cached (in Redis) ApplicationSetting
record, and calling
`Gitlab::CurrentSettings.current_application_settings` only returns it
instead of creating a new DB record, which makes the
`ApplicationSetting.current_without_cache.update!` call fail.

Signed-off-by: Rémy Coutable <remy@rymai.me>

Imre (Admin) authored 5 years ago and Andreas Brandl committed 5 years ago

Move Contribution Analytics related spec in spec/features/groups/group_page_with_external_authorization_service_spec to EE

This reverts merge request !26823

Imre (Admin) authored 5 years ago and Andreas Brandl committed 5 years ago

Move Contribution Analytics related spec in spec/features/groups/group_page_with_external_authorization_service_spec to EE

Signed-off-by: Rémy Coutable <remy@rymai.me>

So the fake can enjoy it, too. We don't use `prepend`
because that'll require we change `allow_any_instance_of` to
`expect_next_instance_of`, but that's not very easy to do.
We can do that later.

Kamil Trzcińśki authored 6 years ago and Grzegorz Bizon committed 6 years ago

This makes code to support encrypted runner tokens.
This code also finished previously started encryption
process.

Renamed EmailValidator to DeviseEmailValidator to avoid 'email:' naming collision with ActiveModel::Validations::EmailValidator in 'validates' statement.
Make use of the options attribute of the parent class ActiveModel::EachValidator.
Add more options: regex.

Cached markdown version is composed both from global and local
markdown version. This allows admins to bump version locally when
needed (e.g. when external URL is changed).

This replaces the use of fake_application_settings with
`::ApplicationSetting.build`_from_defaults. The reason is that
`fake_application_settings` doesn't have the custom accessors that
`ApplicationSetting` has, e.g. `#commit_email_hostname`, thus this
can lead to unexpected `nil` values which comes from the database
column instead of `.default_commit_email_hostname` returned by
`ApplicationSetting#commit_email_hostname`.

Using `::ApplicationSetting.build_from_defaults` should be safe as it
doesn't try to `INSERT` a DB record, in contrary to
`::ApplicationSetting.create_from_defaults` which we used to use, and
which created issues that the introduction of
`fake_application_settings` tried to resolve (575dced5).

Signed-off-by: Rémy Coutable <remy@rymai.me>