Ian Baum authored 4 years ago and Valery Sizov committed 4 years ago

Implementation of PITR recovery before promotion

Closes gitlab#225173

See merge request gitlab-org/omnibus-gitlab!4636

(cherry picked from commit 9ae9fcf7)

691a1fa5 Geo: Implement PITR recovery before promotion of secondary node
1171279d Apply 1 suggestion(s) to 1 file(s)
402e18c6 Apply suggestion. Better lsn file check
fe0c04a3 Merge branch 'master' of gitlab.com:gitlab-org/omnibus-gitlab into pitr
e3f99908 Apply 1 suggestion(s) to 1 file(s)
1d5781dc Merge branch 'pitr' of gitlab.com:gitlab-org/omnibus-gitlab into pitr
7a34d0da Apply 1 suggestion(s) to 1 file(s)
3e9c8592 Apply suggestions
dbbccdee Merge branch 'pitr' of gitlab.com:gitlab-org/omnibus-gitlab into pitr
2ff5b783 Apply suggestions
8b9163ac Add suggestions
565f96c6 Apply suggestions
39b2465d Apply suggestions
c8f5d0ab Merge remote-tracking branch 'origin/master' into pitr
0ba306f4 Fix part of Geo::PromoteToPrimaryNode specs
90c50385 Make recovery_to_point_in_time tio return true
8dd17632 Merge remote-tracking branch 'origin/master' into pitr
1af3bf5a Separate Promote DB task
8a521139 Update specs for the command split
f068dfc3 Merge remote-tracking branch 'origin/master' into pitr
175a4efd Apply 1 suggestion(s) to 1 file(s)
b1af4039 Remove io/console

Robert Marshall authored 4 years ago and Ian Baum committed 4 years ago

- PGBouncer commands check for if the service is running using a status
  check. This does not always work, as the service can be running
  without listening. Changed to run a command that should always work so
  that it verifies that PGBouncer is both running and ready.

Related https://gitlab.com/gitlab-org/gitlab-orchestrator/-/issues/195



Signed-off-by: Robert Marshall <rmarshall@gitlab.com>

The query is missing a SELECT which causes the database to throw
invalid syntax errors when attempting to resume the replication.

task was relocated to gitlab:geo namespace

It will allow promotion even if preflight checks
fail.

preflight checks

`get-postgresql-primary` command prints out the connection information
(IP address and port) of primary PostgreSQL node. This is useful to
identify the primary node in a clustered deployment, for example with
repmgr or Patroni.

This command requires a Consul agent on the same node that it is
running. It uses Consul DNS interface to lookup the SRV record for
PostgreSQL service.

- Update POC code to also call rake task that calls primary api to pause
- Add changelog

Relates to https://gitlab.com/gitlab-org/omnibus-gitlab/issues/35913

It will run preflight checks for promoting to
a primary node. `promote-to-primary-node` command
can include or optionally skip these checks.

Hossein Pursultani authored 4 years ago and DJ Mountney committed 4 years ago

This also includes Psycopg2 which is required by both patroni service
and patronictl commands.

Signed-off-by: Balasankar "Balu" C <balasankarc@autistici.org>

Ian Baum authored 5 years ago and DJ Mountney committed 5 years ago

* Error in GEO due to this. Possible that upgrading the db before
migrations will clear the issue

Robert Marshall authored 5 years ago and DJ Mountney committed 5 years ago

- Regenerate the database.ini file if it does not exist. This addresses
  failures when the consul databases.ini file is not yet created and a
  restart gets issued to `pgbouncer`.

Resolves https://gitlab.com/gitlab-org/distribution/team-tasks/issues/229

Closes https://gitlab.com/gitlab-org/distribution/team-tasks/issues/229



Signed-off-by: Robert Marshall <rmarshall@gitlab.com>

Ian Baum authored 5 years ago and DJ Mountney committed 5 years ago

* Add rubocop-rails and rubocop-performance gems
* Update renamed gems in .rubocop.yml
* Auto-correct where possible
* Manually fix remaining gems

Signed-off-by: Balasankar "Balu" C <balasankar@gitlab.com>

Signed-off-by: Balasankar "Balu" C <balasankar@gitlab.com>

Signed-off-by: Balasankar "Balu" C <balasankar@gitlab.com>

Signed-off-by: Balasankar "Balu" C <balasankar@gitlab.com>

For peer authentication to work, the user running the command should be
same as that running the psql query. This worked so far because the user
mentioned in query is gitlab-consul and we are running these queries as
gitlab-consul, from recipes.

Defaults to gitlabhq_production (backwards-compatible).

It seems people regularly run into the problem where they forget a
step in the setup of Geo, so they get the following error:

> ERROR:  replication slots can only be used if wal_level >= replica

This error seems to happen when on the primary to role was not
configured correctly. The admin should have added
`roles ['geo_primary_role']` or `geo_primary_role['enable'] = true` to
`gitlab.rb` on the primary node.

Since this is a _common_ error, add this as a tip to the
troubleshooting section of the error message.