Ian Baum authored 4 years ago and Balasankar C committed 4 years ago

* Adds postgresql['cert_auth_addresses'] attribute
* Updates pg_hba.conf template to add entries in that attribute

This setting needs to be enabled for pg_rewind to work during a Patroni
failover.

Relates to https://gitlab.com/gitlab-org/gitlab/-/issues/267144

Ian Baum authored 4 years ago and Balasankar C committed 4 years ago

* Matching behavior in rails codebase
* This will be default behavior in 4.0

Robert Marshall authored 4 years ago and Balasankar C committed 4 years ago

- Adds postgresql['hostssl']; defaults to `false`. If set to `true`,
  then `host` type connections will be set as `hostssl` for
  `trust_auth_cidr_addresses` and `md5_auth_cidr_addresses` entries.

Signed-off-by: Robert Marshall <rmarshall@gitlab.com>

* It should be renamed
* I'm going to be making some behavioral changes in a related issue, so best to
  have it renamed ahead of time

Robert Marshall authored 4 years ago and DJ Mountney committed 4 years ago

- Enable the btree_gist postgres extension that is now required because
  of https://gitlab.com/gitlab-org/gitlab/-/merge_requests/32335

Closes https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5392



Signed-off-by: Robert Marshall <rmarshall@gitlab.com>

As discussed in
https://gitlab.com/gitlab-com/gl-infra/production/issues/1704, many of
the logging parameters can be changed at runtime. Those that have a
context of `sighup` can be reloaded with a SIGHUP and not require a full
database restart.

```
gitlabhq_development=# select name, context from pg_settings where name like 'log_%' order by context;
            name             |      context
-----------------------------+-------------------
 logging_collector           | postmaster
 log_checkpoints             | sighup
 log_destination             | sighup
 log_directory               | sighup
 log_autovacuum_min_duration | sighup
 log_file_mode               | sighup
 log_filename                | sighup
 log_hostname                | sighup
 log_line_prefix             | sighup
 log_rotation_age            | sighup
 log_rotation_size           | sighup
 log_timezone                | sighup
 log_truncate_on_rotation    | sighup
 log_lock_waits              | superuser
 log_min_duration_statement  | superuser
 log_min_error_statement     | superuser
 log_min_messages            | superuser
 log_parser_stats            | superuser
 log_planner_stats           | superuser
 log_replication_commands    | superuser
 log_statement               | superuser
 log_statement_stats         | superuser
 log_duration                | superuser
 log_error_verbosity         | superuser
 log_executor_stats          | superuser
 log_temp_files              | superuser
 log_disconnections          | superuser-backend
 log_connections             | superuser-backend
(28 rows)
```

PostgreSQL supports CSV logging out of the box, and we should enabled
this form of structured logging to make it easier to parse and ingest
these logs into our Elasticsearch cluster.

This came out of an investigation in
https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/9254 and
pertains to
https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4973.

Signed-off-by: Balasankar "Balu" C <balasankar@gitlab.com>

* Rename enabled? to service_dir_enabled?
* Add changelog
* Remove dev cruft

* Switch to using the postgresql_config matchers

Print a warning out if is set to an invalid value

For users who have disabled the bundled postgresql daemon, this can be used to link the appropriate binaries accordingly.

Useful if they want to use our bundled backup rake tasks

Starting with Chef 14, sysctl resource is now part of Chef. This will
collide with out custom resource named sysctl.

1. Step into runit_service resource
2. Check for reload_log_dir block to be notified
3. Pass options as needed by runit_service

Fix errors with regex espression

Update tests for the new bin behaviour

Update pg-upgrade with the correct dir information for the move to the
major dir

Add tests for the pg_version library

Stub out the running db version in the db tests

Fix rubocop error

Symlink in the old postgres install directory in order to keep upgrade
working

Use relative symlink in order to survive packaging process

Move the old postgres dir symlink into postinst

Use the proper directory test for symlink creation

Avoid automatic restarts of the postgres runit service

Update the geo-postgres spec with some missing PGVersion syntax

Fix duplicate method running_version

In base_pg_helper

A previous merge conflict must have been resolved incorrectly

Remove extra pgversion work that is no longer needed in this branch

Stick with the old postgres version for now, we will bump the verison in
a new MR

This is better while we are still using the full path the postgresql
installs

This currently should change any behaviour, but is prep for two upcoming
changes.

One where move the install directory for postgres to only use the major
postgres version in the filepath.

And two, the introduction of Postgres 10, where it will matter that the
major version we are using of postgres is 9.6, rather than just 9,
during the transition. And that the version scheme changes a bit
starting in 10. https://www.postgresql.org/support/versioning/

changes

Runs a query against the db to check the currently running version, and
compares that with the binary version. Instruct users to restart
postgres in that case.

This allows us to set/manage this setting via Chef, instead of having to
hack it into the runtime.conf file directly on the primary.

This makes it easier to see what's really happening under the hood
in a development environment.

* Add spec for gitlab-psql-rc
* Move runit templates from gitlab to postgresql cookbook

cookbook

* This is just to get the specs passing

* Upgrades from chef 12.21.31 to 13.6.4, including dependent gems
* Updates chefspec to 7.1.1
* Fixes usage of node.default.gitaly in gitaly recipe
* Fixes logging configuration under chef 13
* Updates CHANGELOG
* Multiple rspec fixes
** Changes uses of `old_run_action` to a more compatible call
** More complete Kernel.load mocking
** Globally mocks `#freeze` on helper instances
** Reset the Gitlab singleton in global `before`
** Fixes bad cache interactions in `services_spec`
** Remove pending from Chef 13 dependent example

= Chef 13.6.4

  $EDITOR Gemfile # pin Chef to 13.6.4
  bundle upgrade chef
  git add --patch Gemfile Gemfile.lock
  $EDITOR config/software/*.rb # reflect changes in Gemfile.lock
  git add config/software

13.6.4 is the most-recent-but-one release in the `stable` series of chef
releases. https://downloads.chef.io/chef/stable

We had tried with the latest `stable` release of chef 13, 13.7.16, but
hit issues with the defaulting of array properties:

https://github.com/chef/chef/issues/6784
https://github.com/chef/chef/issues/6792

Care should be taken to upgrade over 13.7.16 to the next stable release,
though we do have examples that will fail if these bugs are not fixed.

= Chefspec 7.1.1

It's necessary to upgrade to chefspec >= 7 to support chef 13.  We take
the opportunity to go to 7.1.1 which is the latest stable version.  7.1
auto-generates matchers, so we are able to remove
`spec/support/matchers.rb` and `package/libraries/matchers.rb`

== Fixes usage of node.default.gitaly in gitaly recipe

Chef 13 no longer auto-generates accessors on the `attributes` Mashes,
and instead expects you to access them using the `#[]=` method.

We had one use in the gitaly recipe, which has been corrected to follow
common style.

= Be more explicit about the run mode for chef-client

The run-mode of the chef-client was not configured explcitly, and
instead relies on `interval` not being specified to mean 'run once and
exit', rather than using the `once` configuration option.

Additionally, when specifying `once` it also makes sense to specify
`client_fork false` as it avoids a needless fork.

= Fixes logging configuration under chef 13

Due to some refactorings in the development of chef 13, it is no longer
possible to have just a logfile and logging formatter configured
*without* an additional STDOUT logger.

This issue has been raised upstream as
https://github.com/chef/chef/issues/6889 in the interim we monkey-patch
the application class to surpess the creation of the STDOUT logger.

= rspec fixups

== Fix uses of `ruby_block('example').old_run_action(:run)`

We were using the (internal to chefspec) `old_run_action` method to test
the behaviour of the wrapped ruby blocks in `ruby_block` resources.

Due to internal refactorings in chefspec 7.1 `old_run_action` is no
longer available to us.

Here we change uses of the `ruby_block('example').old_run_action(:run)`
pattern to the more compatible `ruby_block('example').block.call`.

== More complete Kernel.load mocking

Chef 13 freezes modules as it loads them.  This prevents accidental
redefinition of the methods, but was initially clashing with itself as
during a chefspec run the cookbook compiler was attempting to load all
libraries after we had already loaded them in the chef_helper for ease
of mocking.

We extended our existing mocking of Kernel.load to be consistent with
the loads of libraries from cookbooks other than the gitlab one.

== Globally mocks `#freeze` on helper instances

Related to the changes to mocking Kernel.load, once this was implemented
we are bitten by default values in LWRPs being frozen in the attribute
validator.  In order to keep on being able to mock all instances of
`PgHelper` and related classes we add a mock of `#freeze` to several
helper classes.

== Reset the Gitlab singleton in global `before`

The Gitlab global object was carrying state from one example group to
another, this was causing subtle issues when running example groups that
mutated the global Gitlab configuration in incompatible ways.

Here we save the empty state of the Gitlab configuration singleton at
the start of the test run, and then reset back to that state in the
global `before`.

== Fixes bad `cached` interaction in `services_spec`

As a knock-on effect of resetting the Gitlab singleton for every
example, we hit problems with how the `services_spec` was making use of
the `cached` rspec helper.

== Remove pending from Chef 13 dependent example

An example that had initially raised awareness of the need for a chef13
upgrade started working.  As this was a pending rspec example this made
the tests fail.  Here we remove the pending qualifier.

Closes #3176

When gitlab_rails['enable'] = false, the database won't be created for
the load to happen, causing an error from the
postgresql_extension[pg_trgm] resource:

  postgresql_extension[pg_trgm] (gitlab::postgresql line 213) had an error: Mixlib::ShellOut::ShellCommandFailed: postgresql_query[enable pg_trgm extension] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/resources/extension.rb line 6) had an error: Mixlib::ShellOut::ShellCommandFailed: execute[enable pg_trgm extension (gitlab-psql)] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/postgresql/resources/query.rb line 11) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '2'
  ---- Begin output of /opt/gitlab/bin/gitlab-psql -d gitlabhq_production -c "CREATE EXTENSION IF NOT EXISTS pg_trgm" ----
  STDOUT:
  STDERR: psql: FATAL:  database "gitlabhq_production" does not exist
  ---- End output of /opt/gitlab/bin/gitlab-psql -d gitlabhq_production -c "CREATE EXTENSION IF NOT EXISTS pg_trgm" ----
  Ran /opt/gitlab/bin/gitlab-psql -d gitlabhq_production -c "CREATE EXTENSION IF NOT EXISTS pg_trgm" returned 2

Looking at the postgresql_extension LWRP we see that the logic around
whether to enable the extension is a quite hairy not_if expression,
which is hard to follow and debug.  To improve this here we extract the
existing BasePgHelper#extension_can_be_enabled? and use that to simplify
the flow of the `postgresql_extension` LWRP and `recipes/postgresql_spec`

Ian Baum authored 7 years ago and Marin Jankovski committed 7 years ago

Resolve "Follow-up from "Resolve "`PG::InsufficientPrivilege: ERROR` when starting GitLab / Build Failed #15823054"""