Ian Baum authored 4 years ago and Valery Sizov committed 4 years ago

Implementation of PITR recovery before promotion

Closes gitlab#225173

See merge request gitlab-org/omnibus-gitlab!4636

(cherry picked from commit 9ae9fcf7)

691a1fa5 Geo: Implement PITR recovery before promotion of secondary node
1171279d Apply 1 suggestion(s) to 1 file(s)
402e18c6 Apply suggestion. Better lsn file check
fe0c04a3 Merge branch 'master' of gitlab.com:gitlab-org/omnibus-gitlab into pitr
e3f99908 Apply 1 suggestion(s) to 1 file(s)
1d5781dc Merge branch 'pitr' of gitlab.com:gitlab-org/omnibus-gitlab into pitr
7a34d0da Apply 1 suggestion(s) to 1 file(s)
3e9c8592 Apply suggestions
dbbccdee Merge branch 'pitr' of gitlab.com:gitlab-org/omnibus-gitlab into pitr
2ff5b783 Apply suggestions
8b9163ac Add suggestions
565f96c6 Apply suggestions
39b2465d Apply suggestions
c8f5d0ab Merge remote-tracking branch 'origin/master' into pitr
0ba306f4 Fix part of Geo::PromoteToPrimaryNode specs
90c50385 Make recovery_to_point_in_time tio return true
8dd17632 Merge remote-tracking branch 'origin/master' into pitr
1af3bf5a Separate Promote DB task
8a521139 Update specs for the command split
f068dfc3 Merge remote-tracking branch 'origin/master' into pitr
175a4efd Apply 1 suggestion(s) to 1 file(s)
b1af4039 Remove io/console

This reverts commit 21ce8bc4, reversing
changes made to 4c9562b6.

Makes it possible to configure Praefect's reconciliation through
gitlab.rb.

Robert Marshall authored 4 years ago and Ian Baum committed 4 years ago

- PGBouncer commands check for if the service is running using a status
  check. This does not always work, as the service can be running
  without listening. Changed to run a command that should always work so
  that it verifies that PGBouncer is both running and ready.

Related https://gitlab.com/gitlab-org/gitlab-orchestrator/-/issues/195



Signed-off-by: Robert Marshall <rmarshall@gitlab.com>

In https://gitlab.com/gitlab-org/gitlab/-/merge_requests/42267, we are
disabling the Sidekiq Exporter logs by default since they are noisy and
don't add a lot of value.  This change allows admins to enable them if
needed.

Background in https://gitlab.com/gitlab-org/gitlab/-/issues/238556

this patch makes following changes:

1. Location block does not match a URL query string which results the
   .git/info/refs being never matched. To match the config of gitlab.com,
    we simply remove the pattern here.

2. Disable buffering for artifacts.

3. Escape the backslashes to make sure "." (dot) symbols is matched
   correctly during nginx location regex matching.

Unlike AWS and Google, Azure needs to use an Azure client inside
Workhorse to support direct uploads. Using standard HTTP transfers with
pre-signed URLs with the Azure Put Blob API
(https://docs.microsoft.com/en-us/rest/api/storageservices/put-blob)
doesn't work because Azure doesn't support chunked transfer encoding.
However, Azure does support uploading files in segments via the Put
Block and Put Block List API
(https://docs.microsoft.com/en-us/rest/api/storageservices/put-block),
but this requires an Azure client.

To support this, this commit extracts the Azure Fog credentials from the
Rails connection information and adds them to the Workhorse
configuration.

This changes requires two merge requests to work:

1. https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/555
2. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38882

Part of https://gitlab.com/gitlab-org/gitlab/-/issues/25877

Signed-off-by: Balasankar "Balu" C <balasankarc@autistici.org>

Signed-off-by: Balasankar "Balu" C <balasankarc@autistici.org>

Signed-off-by: Balasankar "Balu" C <balasankarc@autistici.org>

This is done following the same pattern of an `enable` and `disable`
recipe. Unless a service is explicitly enabled, we consider it to be disabled.

Test coverage for existing consul recipe was split into the corresponding
spec files at the correct folder location.

Mitchell Nielsen authored 4 years ago and Balasankar C committed 4 years ago

Updates the list of deprecated OSs to be displayed at the end of Omnibus
reconfigure.

This commit adds settings for the `storage_options` setting used to
configure AWS server side encryption. Note that only SSE-KMS
(https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html#sse)
is supported; SSE-C is not.

Related merge requests:

1. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38240
2. https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/537

Ian Baum authored 4 years ago and Balasankar C committed 4 years ago

* Matching behavior in rails codebase
* This will be default behavior in 4.0

'read_only_after_failover' configuration toggle has been removed from
Praefect. This commit removes the configuration toggle from Praefect's
configuration template.

Since Gitlab 13.2 Geo does not rely on the
Foreign Data Wrapper to replicate data.

Since Gitlab 13.2 Geo does not rely on the
Foreign Data Wrapper to replicate data.

Cron job introduced in
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38086

The query is missing a SELECT which causes the database to throw
invalid syntax errors when attempting to resume the replication.

Don't rely on complex ENV mocking.

Enabling PG to listen on a different address doesn't explicitly disable
the unix socket functionality, this would be the easy way to let the
postgres-exporter connect to the database using the service account,
preventing problems with authentication methods and trusting IPs