Ian Baum authored 4 years ago and Valery Sizov committed 4 years ago

Implementation of PITR recovery before promotion

Closes gitlab#225173

See merge request gitlab-org/omnibus-gitlab!4636

(cherry picked from commit 9ae9fcf7)

691a1fa5 Geo: Implement PITR recovery before promotion of secondary node
1171279d Apply 1 suggestion(s) to 1 file(s)
402e18c6 Apply suggestion. Better lsn file check
fe0c04a3 Merge branch 'master' of gitlab.com:gitlab-org/omnibus-gitlab into pitr
e3f99908 Apply 1 suggestion(s) to 1 file(s)
1d5781dc Merge branch 'pitr' of gitlab.com:gitlab-org/omnibus-gitlab into pitr
7a34d0da Apply 1 suggestion(s) to 1 file(s)
3e9c8592 Apply suggestions
dbbccdee Merge branch 'pitr' of gitlab.com:gitlab-org/omnibus-gitlab into pitr
2ff5b783 Apply suggestions
8b9163ac Add suggestions
565f96c6 Apply suggestions
39b2465d Apply suggestions
c8f5d0ab Merge remote-tracking branch 'origin/master' into pitr
0ba306f4 Fix part of Geo::PromoteToPrimaryNode specs
90c50385 Make recovery_to_point_in_time tio return true
8dd17632 Merge remote-tracking branch 'origin/master' into pitr
1af3bf5a Separate Promote DB task
8a521139 Update specs for the command split
f068dfc3 Merge remote-tracking branch 'origin/master' into pitr
175a4efd Apply 1 suggestion(s) to 1 file(s)
b1af4039 Remove io/console

This reverts commit 21ce8bc4, reversing
changes made to 4c9562b6.

Will Chandler authored 4 years ago and DJ Mountney committed 4 years ago

When both TLS and Prometheus are enabled in Praefect, the
'prometheus_listen_addr' setting is placed in the '[tls]' block.
This stops prometheus from being enabled.

This change move prometheus above the so this cannot occur.

Ian Baum authored 4 years ago and DJ Mountney committed 4 years ago

* When an upgrade times out, advise the user on how to proceed.

Makes it possible to configure Praefect's reconciliation through
gitlab.rb.

Robert Marshall authored 4 years ago and Ian Baum committed 4 years ago

- PGBouncer commands check for if the service is running using a status
  check. This does not always work, as the service can be running
  without listening. Changed to run a command that should always work so
  that it verifies that PGBouncer is both running and ready.

Related https://gitlab.com/gitlab-org/gitlab-orchestrator/-/issues/195



Signed-off-by: Robert Marshall <rmarshall@gitlab.com>

In https://gitlab.com/gitlab-org/gitlab/-/merge_requests/42267, we are
disabling the Sidekiq Exporter logs by default since they are noisy and
don't add a lot of value.  This change allows admins to enable them if
needed.

Gitaly produceds about 2500 `grpc_server_handled_total` metrics by
default, this results in the same amount of recorded metrics.
By inlining the `rate()`, we can reduce the number of metrics stored
by Prometheus in the default stand-alone setup by 15%.

This saves about 400MiB of the default 2500MiB of Prometheus metrics.

https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4166



Signed-off-by: Ben Kochie <bjk@gitlab.com>

Background in https://gitlab.com/gitlab-org/gitlab/-/issues/238556

Signed-off-by: Takuya Noguchi <takninnovationresearch@gmail.com>

this patch makes following changes:

1. Location block does not match a URL query string which results the
   .git/info/refs being never matched. To match the config of gitlab.com,
    we simply remove the pattern here.

2. Disable buffering for artifacts.

3. Escape the backslashes to make sure "." (dot) symbols is matched
   correctly during nginx location regex matching.

Balasankar C authored 4 years ago and DJ Mountney committed 4 years ago

Signed-off-by: Balasankar "Balu" C <balasankarc@autistici.org>

Without this there will be:

> There was an error running gitlab-ctl reconfigure:
> 
> /etc/gitlab/gitlab.rb:1639: syntax error, unexpected tSTRING_BEG, expecting '}'
>    'storage.path' => '/var/opt/gi...
>    ^
> /etc/gitlab/gitlab.rb:1639: syntax error, unexpected =>, expecting end-of-input
>    'storage.path' => '/var/opt/gitlab/alertmanage...
>                   ^~

Unlike AWS and Google, Azure needs to use an Azure client inside
Workhorse to support direct uploads. Using standard HTTP transfers with
pre-signed URLs with the Azure Put Blob API
(https://docs.microsoft.com/en-us/rest/api/storageservices/put-blob)
doesn't work because Azure doesn't support chunked transfer encoding.
However, Azure does support uploading files in segments via the Put
Block and Put Block List API
(https://docs.microsoft.com/en-us/rest/api/storageservices/put-block),
but this requires an Azure client.

To support this, this commit extracts the Azure Fog credentials from the
Rails connection information and adds them to the Workhorse
configuration.

This changes requires two merge requests to work:

1. https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/555
2. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38882

Part of https://gitlab.com/gitlab-org/gitlab/-/issues/25877

This is done following the same pattern of an `enable` and `disable`
recipe. Unless a service is explicitly enabled, we consider it to be disabled.

Test coverage for existing consul recipe was split into the corresponding
spec files at the correct folder location.

Mitchell Nielsen authored 4 years ago and Balasankar C committed 4 years ago

Updates the list of deprecated OSs to be displayed at the end of Omnibus
reconfigure.

Prometheus external labels are useful.  Epecially if one is, say, trying
to set up extended metrics retention via thanos and its sidecar :)

This change allows external labels to be specified and configured via
the omnibus control file, `gitlab.rb`.

This commit adds settings for the `storage_options` setting used to
configure AWS server side encryption. Note that only SSE-KMS
(https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html#sse)
is supported; SSE-C is not.

Related merge requests:

1. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38240
2. https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/537