Problem with sysctl settings when migrating from 7.x to 8.x
I've discovered a problem with the migration process from 7.x to 8.x with the sysctl settings used for at least PostgreSQL (and unicorn uses net.core.somaxconn
?).
In 7.x the file /etc/sysctl.d/90-postgresql.conf
contained the following by default:
kernel.shmmax = 17179869184 kernel.shmall = 4194304
In 8.x the desired state is that the file /etc/sysctl.d/90-omnibus-gitlab.conf
contains those lines instead. However, this doesn't happen.
On a live system those sysctl values are already set. When the installation of 8.x happens gitlab-ctl reconfigure
is presumably run. The changes for 8.x tell it to get rid of the old 90-postgresql.conf
file and create the new 90-omnibus-gitlab.conf
file. But there's a test in there that only adds the sysctl entries if they're not currently set. They are set, for now, but after a reboot they're gone and the sysctl.conf files don't have the right settings in.
For me, this results in default shmmax settings that don't let PostgreSQL start, and I get a 500 error. Frustratingly this doesn't happen during the upgrade, but happens when the machine is rebooted which could be some time later.
A manual run of gitlab-ctl reconfigure
fixes this by noticing the running sysctl configuration is no longer correct. It adds the right entries to 90-omnibus-gitlab.conf
and activates them. But really this should be correctly handled by the migration.
My suggestion is to remove line 42 in sysctl.rb. But my Chef knowledge is limited, so there may be a better way.
Relevant commits are:
7cbabcd2 a1556926 2f7656d4 e413a3c1
I find it a little puzzling that more people haven't hit this. Either their OS defaults are more sane than mine, or they just haven't rebooted yet!