Enabling Geo secondary starts the geo-postgresql database on the wrong port
Steps to reproduce:
- Download @brodock's latest image.
- Install it.
- Add
geo_secondary_role['enable'] = true
to/etc/gitlab/gitlab.rb
- Run
gitlab-ctl reconfigure
It appears that there is a race condition occurring here. The runit service definition creates a new service, which is automatically started. That service spins until a postgresql.conf
file is available, which happens with the initdb
call.
Attached are the logs:
stanhu@geo-secondary-test:/tmp$ sudo tail /var/log/gitlab/geo-postgresql/current
2017-04-08_14:05:38.29090 postgres: could not access the server configuration file "/var/opt/gitlab/geo-postgresql/data/postgresql.conf": No such file or directory
2017-04-08_14:05:39.30951 FATAL: lock file "postmaster.pid" already exists
2017-04-08_14:05:39.30952 HINT: Is another postgres (PID 10035) running in data directory "/var/opt/gitlab/geo-postgresql/data"?
2017-04-08_14:05:40.33141 FATAL: lock file "postmaster.pid" already exists
2017-04-08_14:05:40.33143 HINT: Is another postgres (PID 10040) running in data directory "/var/opt/gitlab/geo-postgresql/data"?
2017-04-08_14:05:41.92292 LOG: database system was shut down at 2017-04-08 14:05:41 UTC
2017-04-08_14:05:41.92999 LOG: MultiXact member wraparound protections are now enabled
2017-04-08_14:05:41.93070 LOG: database system is ready to accept connections
2017-04-08_14:05:41.93328 LOG: autovacuum launcher started
stanhu@geo-secondary-test:/tmp$ sudo ls -al /var/opt/gitlab/geo-postgresql/data/postgresql.conf
-rw-r--r-- 1 gitlab-psql root 20142 Apr 8 14:05 /var/opt/gitlab/geo-postgresql/data/postgresql.conf
stanhu@geo-secondary-test:/tmp$ sudo gitlab-ctl status
run: geo-postgresql: (pid 10044) 390s; run: log: (pid 10021) 393s
run: gitaly: (pid 10600) 310s; run: log: (pid 9860) 436s
run: gitlab-monitor: (pid 10645) 308s; run: log: (pid 9996) 399s
run: gitlab-workhorse: (pid 10611) 309s; run: log: (pid 9871) 430s
run: logrotate: (pid 9896) 421s; run: log: (pid 9895) 421s
run: nginx: (pid 9882) 427s; run: log: (pid 9881) 427s
run: node-exporter: (pid 9945) 413s; run: log: (pid 9944) 413s
run: postgres-exporter: (pid 10634) 308s; run: log: (pid 9969) 405s
run: postgresql: (pid 9740) 450s; run: log: (pid 9739) 450s
run: prometheus: (pid 10622) 309s; run: log: (pid 9928) 419s
run: redis: (pid 9672) 456s; run: log: (pid 9671) 456s
run: redis-exporter: (pid 9957) 411s; run: log: (pid 9956) 411s
run: sidekiq: (pid 11345) 12s; run: log: (pid 9850) 438s
run: unicorn: (pid 9820) 444s; run: log: (pid 9819) 444s
stanhu@geo-secondary-test:/tmp$ ps -ef | grep geo
root 10020 9639 0 14:05 ? 00:00:00 runsv geo-postgresql
root 10021 10020 0 14:05 ? 00:00:00 svlogd -tt /var/log/gitlab/geo-postgresql
gitlab-+ 10044 10020 0 14:05 ? 00:00:00 /opt/gitlab/embedded/bin/postgres -D /var/opt/gitlab/geo-postgresql/data
stanhu 11493 1904 0 14:12 pts/0 00:00:00 grep --color=auto geo