Database load balancing check for EEP

As part of our effort on GitLab HA, there is the ability to load balance across multiple database servers. Since this is part of our HA feature, this is an EEP feature.

One key difference though between this feature and most of the rest of Postgres HA, is that this is actually part of our GitLab app code. (Whereas much of the rest of Postgres HA is part of the Omnibus code) Since this feature is inside of GitLab, it is aware of the license applied to the instance.

We should add an EEP license check to the database load balancing code. If there is no license, we shouldn't enable this feature. Some additional details:

• If there is no license and this has been configured, we should put a banner warning up on the site to warn people the feature is not active. (Unless there is a better idea on how to convey this?)
• Will need to keep in mind that if the database is having issues we may not be able to query the database to check the license state. Will need to ensure we can validate the license state without having the check back with the database. (This is both steady state as well as startup, since a GitLab app node could be starting up after a failure and the primary node is down.)

---

• Docs https://docs.gitlab.com/ee/administration/database_load_balancing.html
• Initial MR https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1283

changed the description

mentioned in issue omnibus-gitlab#2442 (closed)

changed milestone to %10.0

changed milestone to %9.5

@mydigitalself, this is the first step in GitLab.com showing status of HA. Essentially, whether or not you are licensed to use it. From here I think we can look at adding additional monitoring and potentially configuration aspects, but we need to deliver the license check first.

Would this be something your team could help with, perhaps in 9.5? We could then explore larger status options with 9.6 or later.

changed title from Database load balancing code is EEP to Database load balancing check for EEP

added ~481018 ~1918971 license labels

added Deliverable label

• If there is no license and this has been configured, we should put a banner warning up on the site to warn people the feature is not active. (Unless there is a better idea on how to convey this?)

@mydigitalself How do you envision this? We are not (yet) doing this for any other features that may be configured but are actually inactive.

• Will need to keep in mind that if the database is having issues we may not be able to query the database to check the license state. Will need to ensure we can validate the license state without having the check back with the database. (This is both steady state as well as startup, since a GitLab app node could be starting up after a failure and the primary node is down.)

@joshlambert Is this basically the same thing that's discussed in https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2442#note_31601114, https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2442#note_33895922 etc?

mentioned in issue #2849 (closed)

@DouweM yes, the goal is the same. The challenge with doing a check in Omnibus is that it is not aware of the license, and trying to enforce one to be present at install time is onerous and results in a poor UX.

Given that the DB load balancing code is in the platform, we have an opportunity to perform the check here. I think long term we should have a "Geo style" status page where this could be less intrusive, but I was thinking the banner would be a good minimal first step. Certainly open to other ideas / UX input here as well!

@joshlambert Is ensuring we can validate the license state without having the check back with the database a requirement? Wouldn't GitLab be completely broken in that case anyway, so that whether or not the license check result is correct doesn't really matter?

@DouweM More of a technical concern than product requirement. The product requirement would be that the GitLab service remain operational during various outage scenarios, when an EEP license is present.

My concern was something like the following flow:

1. License is EEP, load balancing is active.
2. GitLab currently hitting node X.
3. Node X goes down, and becomes unresponsive.
4. GitLab now needs to fail over and select a new DB node.
5. GitLab attempts to perform license check, can't reach node X and fails.

Or alternatively, something like:

1. Outage goes down bringing down part of the fleet. Some DB, some app nodes.
2. GitLab brings up a few new app nodes to recover, attempts to connect to first DB in list (node x)
3. Node x is unresponsive.
4. GitLab performs license check for database load balancing, fails to find a license. Fails.

It perhaps could be simplified to "fail open" in the event of database errors, but would certainly look to all your technical expertise on better approaches.

If there is no license and this has been configured, we should put a banner warning up on the site to warn people the feature is not active. (Unless there is a better idea on how to convey this?)

@DouweM @joshlambert if we're going to be promoting features to users, then I think having some UI/banner is a good thing, although I'm not sure how often it would be seen.

If multiple databases are configured, but there is no EEP license, then we could:

• Put a permanent banner in Admin Area (Overview?) saying as such
• Put a notification banner similar to how we notify for expiring trial licenses (see below) in the app itself, this would only be displayed to admins and could be dismissable.

@mydigitalself agree, I think how visible it should be depends on if we can gate the feature and still support the failure modes. If we can gate it, then we can just display to admins.

If we instead opt to not gate the feature because of technical challenges (the license is in the database which you are trying to reach), then we could put up a banner to all users saying there is a license violation.

@joshlambert Ah, I see the situation you are describing now, thanks.

assigned to @rdavila

@DouweM @joshlambert do you think having a copy of the license in the file system will help us in this situation?

@rdavila I would think that would help existing nodes, although we'd still need to think through how to bring on new nodes in the various outage scenarios that can occur. (It may not have the license on disk, yet.)

An MVP here could be to simply provide a banner warning if GitLab provided PG HA is on, and license type is not EEP.

created branch 2850-database-load-balancing-check-for-eep

mentioned in merge request !2448 (merged)

Will need to keep in mind that if the database is having issues we may not be able to query the database to check the license state. Will need to ensure we can validate the license state without having the check back with the database. (This is both steady state as well as startup, since a GitLab app node could be starting up after a failure and the primary node is down.)

@joshlambert given that DB load balancing is enabled at startup time I'm wondering if the above comment still makes sense. I've added the license check there so it will not be enabled without a Premium License.

A last question, if a DB server is having connectivity issues I think the app server will not be able to start isn't it?

If there is no license and this has been configured, we should put a banner warning up on the site to warn people the feature is not active. (Unless there is a better idea on how to convey this?)

@mydigitalself before sending the MR for review, can you please confirm if I should also implement the above feature? Right now I've added a simple check to enable/disable this feature based on the license key.

@rdavila if we can prevent the feature from being enabled without an EEP license then we don't need the banner.

On the topic of whether or not we can start, I will leave that up to you folks who understand the technical components and order of operations better. But my concern was just to ensure that GitLab can start in all various conditions.

For example if the first database node in the list is down, GitLab should still be able to start up and connect to the next one in the list.

@rdavila if we can prevent the feature from being enabled without an EEP license then we don't need the banner.

@joshlambert the current MR enforce that :)

On the topic of whether or not we can start, I will leave that up to you folks who understand the technical components and order of operations better. But my concern was just to ensure that GitLab can start in all various conditions.

For example if the first database node in the list is down, GitLab should still be able to start up and connect to the next one in the list.

hmmm that's probably a question for @yorickpeterse

closed via merge request !2448 (merged)

mentioned in commit d5c9be42

For example if the first database node in the list is down, GitLab should still be able to start up and connect to the next one in the list.

This is implemented, but only for secondaries. This means that if your primary is down there's nothing GitLab can do.

@yorickpeterse @ibaum on the primary side, the work we did in 9.5 with Consul should address that right?