database.md 45.5 KB
Newer Older
1
2
3
4
5
6
---
stage: Enablement
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
---

7
8
# Database settings

9
GitLab supports only PostgreSQL database management system.
10

11
Thus you have two options for database servers to use with Omnibus GitLab:
12

Craig Norris's avatar
Craig Norris committed
13
14
15
- Use the packaged PostgreSQL server included with Omnibus GitLab (no
  configuration required, recommended).
- Use an [external PostgreSQL server](#using-a-non-packaged-postgresql-database-management-server).
16

17
18
19
## Using the PostgreSQL Database Service shipped with Omnibus GitLab

### Configuring SSL
20

21
22
Omnibus automatically enables SSL on the PostgreSQL server, but it will accept
both encrypted and unencrypted connections by default. Enforcing SSL requires
Craig Norris's avatar
Craig Norris committed
23
24
using the `hostssl` configuration in `pg_hba.conf`. For more details, see the
[`pg_hba.conf` documentation](https://www.postgresql.org/docs/11/auth-pg-hba-conf.html).
25

Craig Norris's avatar
Craig Norris committed
26
SSL support depends on the following files:
27

Craig Norris's avatar
Craig Norris committed
28
29
30
31
32
33
- The public SSL certificate for the database (`server.crt`).
- The corresponding private key for the SSL certificate (`server.key`).
- A root certificate bundle that validates the server's certificate (`root.crt`).
  By default, Omnibus GitLab uses the embedded certificate bundle in
  `/opt/gitlab/embedded/ssl/certs/cacert.pem`. This isn't required for
  self-signed certificates.
34

Craig Norris's avatar
Craig Norris committed
35
36
A self-signed certificate and private key will be generated for use. If you'd
prefer to use a CA-signed certificate, use the following steps.
37
38

Note that the location of these files can be configurable, but the private key
Craig Norris's avatar
Craig Norris committed
39
40
41
_must_ be readable by the `gitlab-psql` user. Omnibus manages the permissions of
the files for you, but if the paths are customized, you *must* ensure that the
`gitlab-psql` can access the directory the files are placed in.
42

43
For more details, see the [PostgreSQL documentation](https://www.postgresql.org/docs/11/ssl-tcp.html).
44
45
46
47
48
49
50

Note that `server.crt` and `server.key` may be different from the default SSL
certificates used to access GitLab. For example, suppose the external hostname
of your database is `database.example.com`, and your external GitLab hostname
is `gitlab.example.com`. You will either need a wildcard certificate for
`*.example.com` or two different SSL certificates.

Craig Norris's avatar
Craig Norris committed
51
After you have these files, enable SSL:
52
53
54

1. Edit `/etc/gitlab/gitlab.rb`:

55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
   ```ruby
   postgresql['ssl_cert_file'] = '/custom/path/to/server.crt'
   postgresql['ssl_key_file'] = '/custom/path/to/server.key'
   postgresql['ssl_ca_file'] = '/custom/path/to/bundle.pem'
   postgresql['internal_certificate'] = "-----BEGIN CERTIFICATE-----
   ...base64-encoded certificate...
   -----END CERTIFICATE-----
   "
   postgresql['internal_key'] = "-----BEGIN RSA PRIVATE KEY-----
   ...base64-encoded private key...
   -----END RSA PRIVATE KEY-----
   "
   ```

   Relative paths will be rooted from the PostgreSQL data directory
   (`/var/opt/gitlab/postgresql/data` by default).
71

Evan Read's avatar
Evan Read committed
72
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) to apply the configuration changes.
73

74
75
1. Restart PostgreSQL for the changes to take effect:

Evan Read's avatar
Evan Read committed
76
   ```shell
77
78
79
   gitlab-ctl restart postgresql
   ```

Craig Norris's avatar
Craig Norris committed
80
81
   If PostgreSQL fails to start, check the logs (for example,
   `/var/log/gitlab/postgresql/current`) for more details.
82
83
84
85
86
87
88
89
90

#### Require SSL

1. Add the following to `/etc/gitlab/gitlab.rb`:

    ```ruby
    postgresql['db_sslmode'] = 'require'
    ```

Evan Read's avatar
Evan Read committed
91
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) to apply the configuration changes.
92

93
94
1. Restart PostgreSQL for the changes to take effect:

Evan Read's avatar
Evan Read committed
95
   ```shell
96
97
   gitlab-ctl restart postgresql
   ```
98

Craig Norris's avatar
Craig Norris committed
99
100
   If PostgreSQL fails to start, check the logs (for example,
   `/var/log/gitlab/postgresql/current`) for more details.
101

102
#### Disabling SSL
103

104
1. Add the following to `/etc/gitlab/gitlab.rb`:
105

Ben Bodenmiller's avatar
Ben Bodenmiller committed
106
107
108
   ```ruby
   postgresql['ssl'] = 'off'
   ```
109

Evan Read's avatar
Evan Read committed
110
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) to apply the configuration changes.
111
112
113

1. Restart PostgreSQL for the changes to take effect:

Evan Read's avatar
Evan Read committed
114
   ```shell
115
116
   gitlab-ctl restart postgresql
   ```
117

Craig Norris's avatar
Craig Norris committed
118
119
   If PostgreSQL fails to start, check the logs (for example,
   `/var/log/gitlab/postgresql/current`) for more details.
120

121
#### Verifying that SSL is being used
122

Craig Norris's avatar
Craig Norris committed
123
To determine whether SSL is being used by clients, you can run:
124

Evan Read's avatar
Evan Read committed
125
```shell
126
127
128
129
130
gitlab-rails dbconsole
```

At startup, you should see a banner as the following:

131
```plaintext
132
133
134
135
136
psql (9.6.5)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: on)
Type "help" for help.
```

Craig Norris's avatar
Craig Norris committed
137
To determine if clients are using SSL, issue this SQL query:
138
139
140
141
142
143
144

```sql
SELECT * FROM pg_stat_ssl;
```

For example:

145
```plaintext
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
gitlabhq_production=> SELECT * FROM pg_stat_ssl;
  pid  | ssl | version |           cipher            | bits | compression | clientdn
-------+-----+---------+-----------------------------+------+-------------+----------
 47506 | t   | TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 |  256 | t           |
 47509 | t   | TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 |  256 | t           |
 47510 | t   | TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 |  256 | t           |
 47527 | t   | TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 |  256 | t           |
 47528 | f   |         |                             |      |             |
 47537 | t   | TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 |  256 | t           |
 47560 | f   |         |                             |      |             |
 47561 | f   |         |                             |      |             |
 47563 | t   | TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 |  256 | t           |
 47564 | t   | TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 |  256 | t           |
 47565 | f   |         |                             |      |             |
 47569 | f   |         |                             |      |             |
 47570 | t   | TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 |  256 | t           |
 47573 | f   |         |                             |      |             |
 47585 | f   |         |                             |      |             |
 47586 | t   | TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 |  256 | t           |
 47618 | t   | TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 |  256 | t           |
 47628 | t   | TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 |  256 | t           |
 55812 | t   | TLSv1.2 | ECDHE-RSA-AES256-GCM-SHA384 |  256 | t           |
(19 rows)
```

171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
1. Rows that have `t` listed under the `ssl` column are enabled.
1. Rows that have a value in the `clientdn` are using the `cert` authentication method

#### Configure SSL client authentication

Client SSL certificates can be used to authenticate to the database server. Creating the certificates
is beyond the scope of `omnibus-gitlab`. But users who have an existing SSL certificate management solution
can use this.

##### Configure the database server

1. Create a certificate and key for the server, the common name should equal the DNS name of the server
1. Copy the server certificate, key, and CA file to the PostgreSQL server, and ensure the permissions are correct
   1. The certificate should be owned by the database user (default: `gitlab-psql`)
   1. The key file should be owned by the database user, and its permissions should be `0400`
   1. The CA file should be owned by the database user, and its permissions should be `0400`

   NOTE: **Note:**
   Do not use the filenames `server.crt` or `server.key` for these files. These are reserved for internal use of `omnibus-gitlab`.

1. Ensure the following is set in `gitlab.rb`.

   ```ruby
   postgresql['ssl_cert_file'] = 'PATH_TO_CERTIFICATE'
   postgresql['ssl_key_file'] = 'PATH_TO_KEY_FILE'
   postgresql['ssl_ca_file'] = 'PATH_TO_CA_FILE'
   postgresql['listen_address'] = 'IP_ADDRESS'
   postgresql['cert_auth_addresses'] = {
   'IP_ADDRESS' => {
     'database' => 'gitlabhq_production',
     'user' => 'gitlab'
   }
   ```

   `listen_address` should be set to an IP address of the server that the clients will use to connect to the database.
   `cert_auth_addresses` should contain a list of IP addresses, and the databases and users that are allowed to connect to the database.
1. Run `gitlab-ctl reconfigure` then `gitlab-ctl restart postgresql` in order for the new settings to take effect

#### Configure the Rails client

In order for the rails client to connect to the server, you will need a certficate and key with the `commonName` set to `gitlab`, that is signed by a certificate authority trusted in the CA file specified in `ssl_ca_file` on the database server.

1. Configure `gitlab.rb`

   ```ruby
   gitlab_rails['db_host'] = 'IP_ADDRESS_OR_HOSTNAME_OF_DATABASE_SERVER'
   gitlab_rails['db_sslcert'] = 'PATH_TO_CERTIFICATE_FILE'
   gitlab_rails['db_sslkey'] = 'PATH_TO_KEY_FILE'
   gitlab_rails['db_rootcert'] = 'PATH_TO_CA_FILE'
   ```

1. Run `gitlab-ctl reconfigure` for the rails client to use the new settings
1. Follow the steps in [Verifying that SSL is being used](#verifying-that-ssl-is-being-used) to ensure the authentication is working.
224

225
### Configure packaged PostgreSQL server to listen on TCP/IP
226
227
228
229

The packaged PostgreSQL server can be configured to listen for TCP/IP connections,
with the caveat that some non-critical scripts expect UNIX sockets and may misbehave.

Craig Norris's avatar
Craig Norris committed
230
231
To configure the use of TCP/IP for the database service, make changes to both
the `postgresql` and `gitlab_rails` sections of `gitlab.rb`.
232
233
234
235
236

#### Configure PostgreSQL block

The following settings are affected in the `postgresql` block:

Craig Norris's avatar
Craig Norris committed
237
238
239
240
241
242
- `listen_address`: Controls the address on which PostgreSQL will listen.
- `port`: Controls the port on which PostgreSQL will listen, and _must be set_
  if `listen_address` is.
- `md5_auth_cidr_addresses`: A list of CIDR address blocks which are allowed to
  connect to the server, after authentication with a password.
- `trust_auth_cidr_addresses`: A list of CIDR address blocks which are allowed
243
  to connect to the server, without authentication of any kind. _Be very careful
Craig Norris's avatar
Craig Norris committed
244
245
246
247
248
249
250
  with this setting._ It's suggested that this be limited to the loopback
  address of `127.0.0.1/24` or even `127.0.0.1/32`.
- `sql_user`: Controls the expected username for MD5 authentication. This
  defaults to `gitlab`, and isn't a required setting.
- `sql_user_password`: Sets the password that PostgreSQL will accept for MD5
  authentication. Replace `securesqlpassword` in the following example with an
  acceptable password.
251
252
253
254
255
256
257

```ruby
postgresql['listen_address'] = '0.0.0.0'
postgresql['port'] = 5432
postgresql['md5_auth_cidr_addresses'] = %w()
postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/24)
postgresql['sql_user'] = "gitlab"
258
259
260
261

##! SQL_USER_PASSWORD_HASH can be generated using the command `gitlab-ctl pg-password-md5 gitlab`,
##! where `gitlab` is the name of the SQL user that connects to GitLab.
postgresql['sql_user_password'] = "SQL_USER_PASSWORD_HASH"
262
263
264

# force ssl on all connections defined in trust_auth_cidr_addresses and md5_auth_cidr_addresses
postgresql['hostssl'] = true
265
266
267
268
```

Any client or GitLab service which will connect over the network will need to
provide the values of `sql_user` for the username, and password provided to the
Craig Norris's avatar
Craig Norris committed
269
270
configuration when connecting to the PostgreSQL server. They must also be in the
network block provided to `md5_auth_cidr_addresses`
271
272
273
274

#### Configure GitLab Rails block

To configure the `gitlab-rails` application to connect to the PostgreSQL database
Craig Norris's avatar
Craig Norris committed
275
over the network, several settings must be configured:
276

Craig Norris's avatar
Craig Norris committed
277
- `db_host`: Needs to be set to the IP address of the database sever. If this is
Katrin Leinweber's avatar
Katrin Leinweber committed
278
  on the same instance as the PostgreSQL service, this can be `127.0.0.1` and _will
279
  not require_ password authentication.
Craig Norris's avatar
Craig Norris committed
280
- `db_port`: Sets the port on the PostgreSQL server to connect to, and _must be set_
281
  if `db_host` is set.
Craig Norris's avatar
Craig Norris committed
282
- `db_username`: Configures the username with which to connect to PostgreSQL. This
283
  defaults to `gitlab`.
Craig Norris's avatar
Craig Norris committed
284
- `db_password`: Must be provided if connecting to PostgreSQL over TCP/IP, and from
285
286
287
288
  an instance in the `postgresql['md5_auth_cidr_addresses']` block from settings
  above. This is not required if you are connecting to `127.0.0.1` and have configured
  `postgresql['trust_auth_cidr_addresses']` to include it.

289
```ruby
290
291
292
293
294
295
296
297
gitlab_rails['db_host'] = '127.0.0.1'
gitlab_rails['db_port'] = 5432
gitlab_rails['db_username'] = "gitlab"
gitlab_rails['db_password'] = "securesqlpassword"
```

#### Apply and restart services

Craig Norris's avatar
Craig Norris committed
298
After making the previous changes, an administrator should run `gitlab-ctl reconfigure`.
299
300
301
If you experience any issues in regards to the service not listening on TCP, try
directly restarting the service with `gitlab-ctl restart postgresql`.

Craig Norris's avatar
Craig Norris committed
302
Some included scripts of the Omnibus package (such as `gitlab-psql`) expect the
303
connections to PostgreSQL to be handled over the UNIX socket, and may not function
304
305
306
properly. You can enable TCP/IP without disabling UNIX sockets.

### Enabling PostgreSQL WAL (Write Ahead Log) Archiving
307

Craig Norris's avatar
Craig Norris committed
308
309
By default, WAL archiving of the packaged PostgreSQL isn't enabled. Consider the
following when seeking to enable WAL archiving:
310

Craig Norris's avatar
Craig Norris committed
311
312
313
314
- The WAL level needs to be 'replica' or higher (9.6+ options are `minimal`,
  `replica`, or `logical`)
- Increasing the WAL level will increase the amount of storage consumed in
  regular operations
315
316
317
318

To enable WAL Archiving:

1. Edit `/etc/gitlab/gitlab.rb`:
319

320
321
322
323
324
325
326
327
328
329
330
   ```ruby
   # Replication settings
   postgresql['sql_replication_user'] = "gitlab_replicator"
   postgresql['wal_level'] = "replica"
       ...
       ...
   # Backup/Archive settings
   postgresql['archive_mode'] = "on"
   postgresql['archive_command'] = "/your/wal/archiver/here"
   postgresql['archive_timeout'] = "60"
   ```
331

Evan Read's avatar
Evan Read committed
332
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) for the changes to take effect. This will result in a database restart.
333

334
335
### Store PostgreSQL data in a different directory

Craig Norris's avatar
Craig Norris committed
336
337
By default, everything is stored under `/var/opt/gitlab/postgresql`, controlled
by the `postgresql['dir']` attribute.
338
339
340

This consists of:

Craig Norris's avatar
Craig Norris committed
341
342
343
344
345
- The database socket will be `/var/opt/gitlab/postgresql/.s.PGSQL.5432`. This
  is controlled by `postgresql['unix_socket_directory']`.
- The `gitlab-psql` system user will have its `HOME` directory set to this. This
  is controlled by `postgresql['home']`.
- The actual data will be stored in `/var/opt/gitlab/postgresql/data`.
346
347
348

To change the location of the PostgreSQL data

349
CAUTION: **Caution:**
Craig Norris's avatar
Craig Norris committed
350
351
If you have an existing database, you need to move the data to the new location
first.
352

353
CAUTION: **Caution:**
Craig Norris's avatar
Craig Norris committed
354
355
This is an intrusive operation. It cannot be done without downtime on an
existing installation
356

Craig Norris's avatar
Craig Norris committed
357
1. If this is an existing installation, stop GitLab: `gitlab-ctl stop`.
358
359
360
361
362
363
1. Update `postgresql['dir']` to the desired location.
1. Run `gitlab-ctl reconfigure`.
1. Start GitLab `gitlab-ctl start`.

### Upgrade packaged PostgreSQL server

Craig Norris's avatar
Craig Norris committed
364
365
366
Omnibus GitLab provides the `gitlab-ctl pg-upgrade` command to update the
packaged PostgreSQL server to a later version (if one is included in the
package). This updates PostgreSQL to the [default shipped version](../package-information/postgresql_versions.md)
367
during package upgrades, unless specifically [opted out](#opt-out-of-automatic-postgresql-upgrades).
368

369
CAUTION: **Before upgrading:**
Craig Norris's avatar
Craig Norris committed
370
371
372
373
374
It's important that you fully read this section before running any commands. For
single node installations, this upgrade needs downtime, as the database must be
down while the upgrade is being performed. The length of time depends on the
size of your database. To avoid downtime, it's possible to upgrade to a new
database [using Slony](https://docs.gitlab.com/ee/update/upgrading_postgresql_using_slony.html).
375

376
TIP: **Tip:**
Craig Norris's avatar
Craig Norris committed
377
378
If you encounter any problems during upgrade, raise an issue with a full
description at the [Omnibus GitLab issue tracker](https://gitlab.com/gitlab-org/omnibus-gitlab).
379

Craig Norris's avatar
Craig Norris committed
380
To upgrade the PostgreSQL version, be sure that:
381

Craig Norris's avatar
Craig Norris committed
382
383
384
385
386
- You're running the latest version of GitLab, and it's working.
- If you recently upgraded, you ran `sudo gitlab-ctl reconfigure` successfully
  before you proceed.
- You have sufficient disk space for two copies of your database. _Do not attempt
  to upgrade unless you have enough free space available._
387

Craig Norris's avatar
Craig Norris committed
388
389
390
391
392
393
  - Check your database size using `sudo du -sh /var/opt/gitlab/postgresql/data`
    (or update to your database path).
  - Check the space available using `sudo df -h`. If the partition where the
    database resides doesn't have enough space, pass the argument `--tmp-dir $DIR`
    to the command. GitLab 13.3 includes an available disk space check and aborts
    the upgrade if the requirements aren't met.
Marcin Sedlak-Jakubowski's avatar
Marcin Sedlak-Jakubowski committed
394

Craig Norris's avatar
Craig Norris committed
395
396
After you confirm that the above checklist is satisfied, you can proceed with
the upgrade:
397

398
```shell
399
400
401
sudo gitlab-ctl pg-upgrade
```

402
TIP: **Tip:**
Craig Norris's avatar
Craig Norris committed
403
404
405
`pg-upgrade` can take arguments; for example, you can set the timeout for the
execution of the underlying commands (`--timeout=1d2h3m4s5ms`). Run `gitlab-ctl pg-upgrade -h`
to see the full list.
406

407
`gitlab-ctl pg-upgrade` performs the following steps:
408

409
410
1. Checks to ensure the database is in a known good state.
1. Checks if there's enough free disk space and aborts otherwise. You can skip this by appending the `--skip-disk-check` flag.
411
1. Shuts down the existing database, any unnecessary services, and enables the GitLab deploy page.
412
413
414
415
416
417
418
419
1. Changes the symlinks in `/opt/gitlab/embedded/bin/` for PostgreSQL to point to the newer version of the database.
1. Creates a new directory containing a new, empty database with a locale matching the existing database.
1. Uses the `pg_upgrade` tool to copy the data from the old database to the new database.
1. Moves the old database out of the way.
1. Moves the new database to the expected location.
1. Calls `sudo gitlab-ctl reconfigure` to do the required configuration changes and starts the new database server.
1. Starts the remaining services and removes the deploy page.
1. If any errors are detected during this process, it reverts to the old version of the database.
420

Marcin Sedlak-Jakubowski's avatar
Marcin Sedlak-Jakubowski committed
421
After the upgrade is complete, verify that everything is working as expected.
422

Craig Norris's avatar
Craig Norris committed
423
424
_After you have verified that your GitLab instance is running correctly_, you
can clean up the old database files:
425

426
```shell
427
428
sudo rm -rf /var/opt/gitlab/postgresql/data.<old_version>
sudo rm -f /var/opt/gitlab/postgresql-version.old
429
430
```

Craig Norris's avatar
Craig Norris committed
431
432
You can find details of PostgreSQL versions shipped with various GitLab versions
in [PostgreSQL versions shipped with Omnibus GitLab](../package-information/postgresql_versions.md).
433

434
#### Opt out of automatic PostgreSQL upgrades
435

436
To opt out of automatic PostgreSQL upgrade during GitLab package upgrades, run:
437

438
439
440
```shell
sudo touch /etc/gitlab/disable-postgresql-upgrade
```
441

442
#### GitLab 13.3 and later
DJ Mountney's avatar
DJ Mountney committed
443

444
As of GitLab 13.3, PostgreSQL 11.7 and 12.3 are both shipped with Omnibus.
Craig Norris's avatar
Craig Norris committed
445
446
During a package upgrade, the database isn't upgraded to PostgreSQL 12. If you
want to upgrade to PostgreSQL 12, you must do it manually:
447

448
449
450
```shell
sudo gitlab-ctl pg-upgrade -V 12
```
451

Craig Norris's avatar
Craig Norris committed
452
PostgreSQL 12 isn't supported on Geo deployments and is [planned](https://gitlab.com/groups/gitlab-org/-/epics/2374)
453
for the future releases. The [fault-tolerant PostgreSQL 12 deployment](https://docs.gitlab.com/ee/administration/postgresql/replication_and_failover.html)
Craig Norris's avatar
Craig Norris committed
454
455
is possible only by using Patroni. Repmgr is no longer supported for
PostgreSQL 12.
456

457
458
#### GitLab 12.10 and later

Craig Norris's avatar
Craig Norris committed
459
460
The default PostgreSQL version is set to 11.x, and an upgrade of the database is
done on package upgrades for installs that are not using repmgr or Geo.
461

Craig Norris's avatar
Craig Norris committed
462
The upgrade is skipped in any of the following cases:
463

Craig Norris's avatar
Craig Norris committed
464
465
466
- You're running the database in high availability using repmgr.
- Your database nodes are part of GitLab Geo configuration.
- You have specifically [opted out](#opt-out-of-automatic-postgresql-upgrades).
467
468
469

To upgrade PostgreSQL on installs with HA or Geo, see [Packaged PostgreSQL deployed in an HA/Geo Cluster](#packaged-postgresql-deployed-in-an-hageo-cluster).

470
471
472
473
474
#### GitLab 12.8 and later

**As of GitLab 12.8, PostgreSQL 9.6.17, 10.12, and 11.7 are shipped with
Omnibus GitLab.**

Craig Norris's avatar
Craig Norris committed
475
476
477
478
During package upgrades (unless opted out) and when user manually runs
`gitlab-ctl pg-upgrade`, `omnibus-gitlab` will still be attempting to upgrade
the database only to 10.x, while 11.x will be available for users to manually
upgrade to. To manually update PostgreSQL to version 11.x , the `pg-upgrade`
479
command has to be passed with a version argument (`-V` or `--target-version`):
480
481
482
483
484
485
486
487
488
489

```shell
sudo gitlab-ctl pg-upgrade -V 11
```

#### GitLab 12.0 and later

**As of GitLab 12.0, PostgreSQL 9.6.11 and 10.7 are shipped with Omnibus
GitLab.**

Craig Norris's avatar
Craig Norris committed
490
491
On upgrades, we will be upgrading the database to 10.7 unless specifically opted
out, as previously described.
492
493
494
495
496
497
498
499

#### GitLab 11.11 and later

**As of GitLab 11.11, PostgreSQL 9.6.X and 10.7 are shipped with Omnibus
GitLab.**

Fresh installs will be getting PostgreSQL 10.7 while GitLab package upgrades
will retain the existing version of PostgreSQL. Users can manually upgrade to
Craig Norris's avatar
Craig Norris committed
500
the 10.7 using the `pg-upgrade` command as previously mentioned.
501

502
### Revert packaged PostgreSQL server to previous version
503

504
DANGER: **Danger:**
Craig Norris's avatar
Craig Norris committed
505
This operation will revert your current database, _including its data_, to its state
506
507
508
before your last upgrade. Be sure to create a backup before attempting to downgrade
your packaged PostgreSQL database.

509
510
511
512
513
514
On GitLab versions which ship multiple PostgreSQL versions, users can downgrade
an already upgraded PostgreSQL version to the earlier version using the `gitlab-ctl
revert-pg-upgrade` command. This command also supports the `-V` flag to specify
a target version for scenarios where more than two PostgreSQL versions are shipped in
the package (for example: GitLab 12.8 where PostgreSQL 9.6.x, 10.x, and 11.x are
shipped).
515

516
If the target version is not specified, it will use the version in `/var/opt/gitlab/postgresql-version.old`
Craig Norris's avatar
Craig Norris committed
517
if available. Otherwise, it falls back to the default version shipped with GitLab.
518

Larissa Lane's avatar
Larissa Lane committed
519
On other GitLab versions which ship only one PostgreSQL version, you can't
520
521
downgrade your PostgreSQL version. You must downgrade GitLab to an older version for
this.
522
523
524

### Connecting to the bundled PostgreSQL database

Craig Norris's avatar
Craig Norris committed
525
526
527
If you need to connect to the bundled PostgreSQL database and are using the
default Omnibus GitLab database configuration, you can connect as the
application user:
528

Evan Read's avatar
Evan Read committed
529
```shell
530
531
532
sudo gitlab-rails dbconsole
```

533
or as a PostgreSQL superuser:
534

Evan Read's avatar
Evan Read committed
535
```shell
536
537
538
sudo gitlab-psql -d gitlabhq_production
```

539
540
## Using a non-packaged PostgreSQL database management server

Craig Norris's avatar
Craig Norris committed
541
By default, GitLab is configured to use the PostgreSQL server that's included
542
in Omnibus GitLab. You can also reconfigure it to use an external instance of
543
PostgreSQL.
544

545
546
CAUTION: **Caution:**
If you are using non-packaged PostgreSQL server, you need to make
Evan Read's avatar
Evan Read committed
547
sure that PostgreSQL is set up according to the [database requirements document](https://docs.gitlab.com/ee/install/requirements.html#database).
548

Evan Read's avatar
Evan Read committed
549
1. Edit `/etc/gitlab/gitlab.rb`:
550

551
552
553
554
555
556
557
558
559
560
561
562
   ```ruby
   # Disable the built-in Postgres
   postgresql['enable'] = false

   # Fill in the connection details for database.yml
   gitlab_rails['db_adapter'] = 'postgresql'
   gitlab_rails['db_encoding'] = 'utf8'
   gitlab_rails['db_host'] = '127.0.0.1'
   gitlab_rails['db_port'] = 5432
   gitlab_rails['db_username'] = 'USERNAME'
   gitlab_rails['db_password'] = 'PASSWORD'
   ```
563

564
565
   Don't forget to remove the `#` comment characters at the beginning of these
   lines.
566

567
   Note that:
568

569
570
   - `/etc/gitlab/gitlab.rb` should have file permissions `0600` because it contains
     plain-text passwords.
571
   - PostgreSQL allows to listen on [multiple addresses](https://www.postgresql.org/docs/11/runtime-config-connection.html)
572

573
     If you use multiple addresses in `gitlab_rails['db_host']`, comma-separated, the first address in the list will be used for connection.
574

Evan Read's avatar
Evan Read committed
575
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) for the changes to take effect.
576

Evan Read's avatar
Evan Read committed
577
1. [Seed the database](#seed-the-database-fresh-installs-only).
578

579
580
### UNIX socket configuration for non-packaged PostgreSQL

Craig Norris's avatar
Craig Norris committed
581
582
583
If you want to use your system's PostgreSQL server (installed on the same system
as GitLab) instead of the one bundled with GitLab, you can do so by using a UNIX
socket:
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603

1. Edit `/etc/gitlab/gitlab.rb`:

   ```ruby
   # Disable the built-in Postgres
   postgresql['enable'] = false

   # Fill in the connection details for database.yml
   gitlab_rails['db_adapter'] = 'postgresql'
   gitlab_rails['db_encoding'] = 'utf8'
   # The path where the socket lives
   gitlab_rails['db_host'] = '/var/run/postgresql/'
   ```

1. Reconfigure GitLab for the changes to take effect:

   ```ruby
   sudo gitlab-ctl-reconfigure
   ```

604
605
606
607
608
609
### Configuring SSL

#### Require SSL

1. Add the following to `/etc/gitlab/gitlab.rb`:

Evan Read's avatar
Evan Read committed
610
611
612
   ```ruby
   postgresql['db_sslmode'] = 'require'
   ```
613

Evan Read's avatar
Evan Read committed
614
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) to apply the configuration changes.
615

616
617
1. Restart PostgreSQL for the changes to take effect:

Evan Read's avatar
Evan Read committed
618
   ```shell
619
620
621
   gitlab-ctl restart postgresql
   ```

Craig Norris's avatar
Craig Norris committed
622
623
   If PostgreSQL fails to start, check the logs (for example,
   `/var/log/gitlab/postgresql/current`) for more details.
624
625
626
627

#### Require SSL and verify server certificate against CA bundle

PostgreSQL can be configured to require SSL and verify the server certificate
Craig Norris's avatar
Craig Norris committed
628
629
against a CA bundle to prevent spoofing.
The CA bundle that's specified in `gitlab_rails['db_sslrootcert']` must contain
630
631
632
633
both the root and intermediate certificates.

1. Add the following to `/etc/gitlab/gitlab.rb`:

Evan Read's avatar
Evan Read committed
634
635
636
637
   ```ruby
   gitlab_rails['db_sslmode'] = "verify-full"
   gitlab_rails['db_sslrootcert'] = "your-full-ca-bundle.pem"
   ```
638

639
   If you are using Amazon RDS for your PostgreSQL server, ensure you
Evan Read's avatar
Evan Read committed
640
641
642
643
   download and use the [combined CA bundle](https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem)
   for `gitlab_rails['db_sslrootcert']`. More information on this can be found
   in the [using SSL/TLS to Encrypt a Connection to a DB Instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html)
   article on AWS.
644

Evan Read's avatar
Evan Read committed
645
1. [Reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#omnibus-gitlab-reconfigure) to apply the configuration changes.
646

647
648
1. Restart PostgreSQL for the changes to take effect:

Evan Read's avatar
Evan Read committed
649
   ```shell
650
651
652
   gitlab-ctl restart postgresql
   ```

Craig Norris's avatar
Craig Norris committed
653
654
   If PostgreSQL fails to start, check the logs (for example,
   `/var/log/gitlab/postgresql/current`) for more details.
655

656
### Backup and restore a non-packaged PostgreSQL database
657

Evan Read's avatar
Evan Read committed
658
When using the [Rake backup create and restore task](https://docs.gitlab.com/ee/raketasks/backup_restore.html#create-a-backup-of-the-gitlab-system), GitLab will
659
attempt to use the packaged `pg_dump` command to create a database backup file
660
and the packaged `psql` command to restore a backup. This will only work if
661
662
they are the correct versions. Check the versions of the packaged `pg_dump` and
`psql`:
663

Evan Read's avatar
Evan Read committed
664
```shell
Balasankar C's avatar
Balasankar C committed
665
666
/opt/gitlab/embedded/bin/pg_dump --version
/opt/gitlab/embedded/bin/psql --version
667
```
668

669
670
If these versions are different from your non-packaged external PostgreSQL, you
will need to install tools that match your database version and then follow the
671
steps below. There are multiple ways to install PostgreSQL client tools. See
Evan Read's avatar
Evan Read committed
672
<https://www.postgresql.org/download/> for options.
673

674
675
Once the correct `psql` and `pg_dump` tools are available on your system, follow
these steps, using the correct path to the location you installed the new tools:
676

Evan Read's avatar
Evan Read committed
677
678
1. Add symbolic links to the non-packaged versions:

Evan Read's avatar
Evan Read committed
679
   ```shell
680
681
   ln -s /path/to/new/pg_dump /path/to/new/psql /opt/gitlab/bin/
   ```
682

Evan Read's avatar
Evan Read committed
683
1. Check the versions:
684

685
   ```shell
686
687
688
   /opt/gitlab/bin/pg_dump --version
   /opt/gitlab/bin/psql --version
   ```
689

690
   They should now be the same as your non-packaged external PostgreSQL.
691

692
After this is done, ensure that the backup and restore tasks are using the
Evan Read's avatar
Evan Read committed
693
694
correct executables by running both the [backup](https://docs.gitlab.com/ee/raketasks/backup_restore.html#create-a-backup-of-the-gitlab-system) and
[restore](https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-a-previously-created-backup) tasks.
695

696
697
### Upgrade a non-packaged PostgreSQL database

698
Before proceeding with the upgrade, note the following:
699

Craig Norris's avatar
Craig Norris committed
700
701
702
703
704
705
706
707
708
709
710
711
712
713
- Before upgrading, review the [GitLab and PostgreSQL version compatibility table](../package-information/postgresql_versions.md)
  to determine your upgrade path. When using GitLab backup or restore, you
  _must_ keep the same version of GitLab; first upgrade PostgreSQL, and then GitLab.
- The [backup and restore Rake task](https://docs.gitlab.com/ee/raketasks/backup_restore.html#create-a-backup-of-the-gitlab-system)
  can be used to back up and restore the database to a later version of PostgreSQL.
- If configuring a version number whose binaries are unavailable on the file
  system, GitLab/Rails uses the default database's version binaries (default as
  per [GitLab and PostgreSQL version compatibility table](../package-information/postgresql_versions.md)).
- If you're using Amazon RDS and are seeing extremely high (near 100%) CPU
  utilization following a major version upgrade (for example, from `10.x` to
  `11.x`), running an `ANALYZE VERBOSE;` query may be necessary to recreate query
  plans and reduce CPU utilization on the database server(s).
  [Amazon recommends this](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.html)
  as part of a major version upgrade.
714

715
The following example demonstrates upgrading from a database host running PostgreSQL 10 to another database host running PostgreSQL 11 and incurs downtime:
716

Craig Norris's avatar
Craig Norris committed
717
1. Spin up a new PostgreSQL 11 database server that's set up according to the [database requirements](https://docs.gitlab.com/ee/install/requirements.html#database).
718

Craig Norris's avatar
Craig Norris committed
719
720
721
1. Ensure that the compatible versions of `pg_dump` and `pg_restore` are being
   used on the GitLab Rails instance. To amend GitLab configuration, edit
   `/etc/gitlab/gitlab.rb` and specify the value of `postgresql['version']`:
722
723
724
725
726

    ```ruby
    postgresql['version'] = 11
    ```

727
  NOTE: **Note:**
Craig Norris's avatar
Craig Norris committed
728
729
730
731
732
  Connecting to PostgreSQL 12 (alongside with amending `postgresql['version'] = 12`)
  breaks the [GitLab Backup/Restore](https://docs.gitlab.com/ee/raketasks/backup_restore.html) functionality unless the v12 client binaries are available on the file system. More on this topic can be found under [backup and restore a non-packaged database](#backup-and-restore-a-non-packaged-postgresql-database).
  This problem with missing 12 client binaries is partially resolved in GitLab 13.3,
  where PostgreSQL 12 is shipped with Omnibus GitLab, and it's being tackled in
  the [support for PostgreSQL 12](https://gitlab.com/groups/gitlab-org/-/epics/2374) epic.
733

734
735
736
737
738
739
740
741
1. Reconfigure GitLab:

   ```shell
   sudo gitlab-ctl reconfigure
   ```

1. Stop GitLab (note that this step will cause downtime):

742
   ```shell
743
744
745
   sudo gitlab-ctl stop
   ```

Craig Norris's avatar
Craig Norris committed
746
747
1. Run the backup Rake task using the SKIP options to back up only the database.
   Make a note of the backup file name; you'll use it later to restore.
748
749
750
751
752
753
754

   ```shell
   sudo gitlab-backup create SKIP=repositories,uploads,builds,artifacts,lfs,pages,registry
   ```

1. Shutdown the PostgreSQL 10 database host.

Craig Norris's avatar
Craig Norris committed
755
756
1. Edit `/etc/gitlab/gitlab.rb` and update the `gitlab_rails['db_host']` setting
   to point to the PostgreSQL database 11 host.
757
758
759
760
761
762
763

1. Reconfigure GitLab:

   ```shell
   sudo gitlab-ctl reconfigure
   ```

Craig Norris's avatar
Craig Norris committed
764
765
1. Restore the database using the database backup file created earlier, and be
   sure to answer **no** when asked "This task will now rebuild the authorized_keys file":
766

767
   ```shell
768
769
770
771
772
   sudo gitlab-backup restore BACKUP=<database-backup-filename>
   ```

1. Start GitLab:

773
   ```shell
774
775
776
   sudo gitlab-ctl start
   ```

777
### Seed the database (fresh installs only)
778

779
CAUTION: **Caution:**
Craig Norris's avatar
Craig Norris committed
780
This is a destructive command; do not run it on an existing database.
781

Craig Norris's avatar
Craig Norris committed
782
783
Omnibus GitLab will not seed your external database. Run the following command
to import the schema and create the first administration user:
784
785

```shell
786
# Remove 'sudo' if you are the 'git' user
787
788
789
sudo gitlab-rake gitlab:setup
```

790
791
792
If you want to specify a password for the default `root` user, specify the
`initial_root_password` setting in `/etc/gitlab/gitlab.rb` before running the
`gitlab:setup` command above:
793
794

```ruby
795
gitlab_rails['initial_root_password'] = 'nonstandardpassword'
796
797
```

798
799
800
801
802
803
804
805
If you want to specify the initial registration token for shared GitLab Runners,
specify the `initial_shared_runners_registration_token` setting in `/etc/gitlab/gitlab.rb`
before running the `gitlab:setup` command:

```ruby
gitlab_rails['initial_shared_runners_registration_token'] = 'token'
```

806
807
808
### Pin the packaged PostgreSQL version (fresh installs only)

NOTE: **Note:**
809
810
811
812
GitLab 13.0 only ships with PostgreSQL 11. PostgreSQL 12 is only available from GitLab 13.3 onward.
PostgreSQL 12 is not supported on Geo deployments and is [planned](https://gitlab.com/groups/gitlab-org/-/epics/2374)
for the future releases. The [fault-tolerant PostgreSQL 12 deployment](https://docs.gitlab.com/ee/administration/postgresql/replication_and_failover.html)
is only possible using Patroni. Repmgr is no longer supported for PostgreSQL 12.
813
814
815

Omnibus GitLab will initialize PostgreSQL with the [default version](../package-information/postgresql_versions.md).

816
817
To initialize PostgreSQL with a non-default version, you can set `postgresql['version']` to the major version one of
the [packaged PostgreSQL versions](../package-information/postgresql_versions.md) prior to the initial reconfigure.
818
For example, starting GitLab 13.3 you can use `postgresql['version'] = 12` to opt in to use PostgreSQL 12.
819
820
821
822
823
824

CAUTION: **Caution:**
Setting `postgresql['version']` while using the Omnibus packaged PostgreSQL after the initial reconfigure will
throw errors about the data directory being initialized on a different version of PostgreSQL. If this is encountered,
see [Revert packaged PostgreSQL server to previous version](#revert-packaged-postgresql-server-to-previous-version).

825
826
827
828
### Troubleshooting

#### Set `default_transaction_isolation` into `read committed`

829
If you see errors similar to the following in your `production/sidekiq` log:
830

831
```plaintext
832
833
834
835
836
837
838
839
840
841
842
ActiveRecord::StatementInvalid PG::TRSerializationFailure: ERROR:  could not serialize access due to concurrent update
```

Chances are your database's `default_transaction_isolation` configuration is not
in line with GitLab application requirement. You can check this configuration by
connecting to your PostgreSQL database and run `SHOW default_transaction_isolation;`.
GitLab application expects `read committed` to be configured.

This `default_transaction_isolation` configuration is set in your
`postgresql.conf` file. You will need to restart/reload the database once you
changed the configuration. This configuration comes by default in the packaged
Evan Read's avatar
Evan Read committed
843
PostgreSQL server included with Omnibus GitLab.
844
845
846
847

## Application Settings for the Database

### Disabling automatic database migration
848

849
If you have multiple GitLab servers sharing a database, you will want to limit the
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
number of nodes that are performing the migration steps during reconfiguration.

Edit `/etc/gitlab/gitlab.rb`:

```ruby
# Enable or disable automatic database migrations
gitlab_rails['auto_migrate'] = false
```

Don't forget to remove the `#` comment characters at the beginning of this
line.

`/etc/gitlab/gitlab.rb` should have file permissions `0600` because it contains
plain-text passwords.

The next time a reconfigure is triggered, the migration steps will not be performed.

867
868
### Setting client statement_timeout

Andreas Brandl's avatar
Andreas Brandl committed
869
The amount of time that Rails will wait for a database transaction to complete
870
before timing out can now be adjusted with the `gitlab_rails['db_statement_timeout']`
871
setting. By default, this setting is not used.
872
873
874
875
876
877
878
879
880
881

Edit `/etc/gitlab/gitlab.rb`:

```ruby
gitlab_rails['db_statement_timeout'] = 45000
```

In this case the client `statement_timeout` is set to 45 seconds. The value
is specified in milliseconds.

Andreas Brandl's avatar
Andreas Brandl committed
882
883
884
885
### Setting connection timeout

The amount of time that Rails will wait for a PostgreSQL connection attempt to succeed
before timing out can be adjusted with the `gitlab_rails['db_connect_timeout']`
886
setting. By default, this setting is not used:
Andreas Brandl's avatar
Andreas Brandl committed
887

888
1. Edit `/etc/gitlab/gitlab.rb`:
Andreas Brandl's avatar
Andreas Brandl committed
889

890
891
892
893
894
895
896
897
898
   ```ruby
   gitlab_rails['db_connect_timeout'] = 5
   ```

1. Reconfigure GitLab:

   ```shell
   sudo gitlab-ctl reconfigure
   ```
Andreas Brandl's avatar
Andreas Brandl committed
899
900
901
902
903

In this case the client `connect_timeout` is set to 5 seconds. The value
is specified in seconds. A minimum value of 2 seconds applies. Setting this to `<= 0`
or not specifying the setting at all disables the timeout.

904
905
## Automatic database reindexing

906
907
908
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/249662) in GitLab 13.5.

CAUTION: **Warning:**
Craig Norris's avatar
Craig Norris committed
909
This is an experimental feature that isn't enabled by default.
910
911
912
913
914

Recreates database indexes in the background (called "reindexing"). This can
be used to remove bloated space that has accumulated in indexes and helps to maintain healthy and
efficient indexes.

Craig Norris's avatar
Craig Norris committed
915
The reindexing task can be started regularly through a cronjob. To configure the cronjob,
916
917
918
919
920
921
922
`gitlab_rails['database_reindexing']['enable']` should be set to `true`.

In a multi-node environment, this feature should only be enabled on an application host.
The reindexing process cannot go through PgBouncer, it has to have a direct database connection.

By default, this starts the cronjob every hour during weekends (likely a low-traffic time) only.

923
You can change the schedule by refining the following settings:
924

925
1. Edit `/etc/gitlab/gitlab.rb`:
926

927
928
929
930
931
932
933
934
935
936
937
938
939
   ```shell
   gitlab_rails['database_reindexing']['hour'] = '*'
   gitlab_rails['database_reindexing']['minute'] = 0
   gitlab_rails['database_reindexing']['month'] = '*'
   gitlab_rails['database_reindexing']['day_of_month'] = '*'
   gitlab_rails['database_reindexing']['day_of_week'] = '0,6'
   ```

1. Reconfigure GitLab:

   ```shell
   sudo gitlab-ctl reconfigure
   ```
940

941
## Packaged PostgreSQL deployed in an HA/Geo Cluster
Ian Baum's avatar
Ian Baum committed
942

943
### Upgrading a GitLab HA cluster
Evan Read's avatar
Evan Read committed
944

945
NOTE: **Note:**
946
947
948
949
As of GitLab 13.3, PostgreSQL 12 is shipped with Omnibus GitLab. However, the current support for is limited to
single database node installation. [Fault-tolerant PostgreSQL deployments](https://docs.gitlab.com/ee/administration/postgresql/replication_and_failover.html),
and Geo installations are not supported, but [planned](https://gitlab.com/groups/gitlab-org/-/epics/2374) for the future releases.

950
951
952
The following instructions are valid for a fault tolerant setup with repmgr. To upgrade PostgreSQL version in a
Patroni cluster see [Upgrading PostgreSQL major version in a Patroni cluster](https://docs.gitlab.com/ee/administration/postgresql/replication_and_failover.html#upgrading-postgresql-major-version-in-a-patroni-cluster).

953
954
955
956
957
If [PostgreSQL is configured for high availability](https://docs.gitlab.com/ee/administration/postgresql/index.html),
`pg-upgrade` should be run on all the nodes running PostgreSQL. Other nodes can be
skipped, but must be running the same GitLab version as the database nodes.

Follow the steps below to upgrade the database nodes:
958
959

1. Secondary nodes must be upgraded before the primary node.
960
   1. On the secondary nodes, edit `/etc/gitlab/gitlab.rb` to include the following:
Aric Buerer's avatar
Aric Buerer committed
961

Evan Read's avatar
Evan Read committed
962
   ```shell
963
964
965
   # Replace X with value of number of db nodes + 1
   postgresql['max_replication_slots'] = X
    ```
Aric Buerer's avatar
Aric Buerer committed
966

Evan Read's avatar
Evan Read committed
967
   1. Run `gitlab-ctl reconfigure` to update the configuration.
968
   1. Run `sudo gitlab-ctl restart postgresql` to get PostgreSQL restarted with the new configuration.
969
   1. On running `pg-upgrade` on a PostgreSQL secondary node, the node will be removed
970
971
972
973
974
975
976
      from the cluster.
   1. Once all the secondary nodes are upgraded using `pg-upgrade`, the user
      will be left with a single-node cluster that has only the primary node.
   1. `pg-upgrade`, on secondary nodes will not update the existing data to
      match the new version, as that data will be replaced by the data from
      primary node. It will, however move the existing data to a backup
      location.
977
1. Once all secondary nodes are upgraded, run `pg-upgrade` on primary node.
Aric Buerer's avatar
Aric Buerer committed
978
   1. On the primary node, edit `/etc/gitlab/gitlab.rb` to include the following:
Aric Buerer's avatar
Aric Buerer committed
979

Evan Read's avatar
Evan Read committed
980
   ```shell
981
982
983
   # Replace X with value of number of db nodes + 1
   postgresql['max_replication_slots'] = X
    ```
Aric Buerer's avatar
Aric Buerer committed
984

Evan Read's avatar
Evan Read committed
985
   1. Run `gitlab-ctl reconfigure` to update the configuration.
986
   1. Run `sudo gitlab-ctl restart postgresql` to get PostgreSQL restarted with the new configuration.
987
   1. On a primary node, `pg-upgrade` will update the existing data to match
988
      the new PostgreSQL version.
989
1. Recreate the secondary nodes by running the following command on each of them
990

Evan Read's avatar
Evan Read committed
991
   ```shell
992
993
994
   gitlab-ctl repmgr standby setup MASTER_NODE_NAME
   ```

995
1. Check if the repmgr cluster is back to the original state
996

Evan Read's avatar
Evan Read committed
997
   ```shell
998
999
   gitlab-ctl repmgr cluster show
   ```
1000