Licensing for HA omnibus-gitlab package

Part of gitlab-ee#77

Description

High availability is typically an Enterprise feature. However, those packages are included in EE (with Omnibus installs) for everyone, without license check of any kind. That makes it impossible to sell it as Premium, for instance, which I assume we want to do.

Initial discussion

I will have to look into deeper but the current idea is to 
just reuse the license gem we already have in our EE Gemfile. The customer would need
to supply a license file and place it into a certain location in /etc/gitlab
We would then have scripts that would verify the license and allow cookbook to 
execute on reconfigure

Proposal

Two possibilities:

1. Use the EE license that is already in the database and user rails runner to check for the contents before running the cookbook
   • The first reconfigure (ie, during installation) should not be blocked if no license found. Only the subsequent ones.
2. Customer supplies the license file on a location on the FS. We read the license and decide whether to run the cookbooks

There are 2 options as I currently see:

1. Use the EE license that is already in the database and user rails runner to check for the contents before running the cookbook
2. Customer supplies the license file on a location on the FS. We read the license and decide whether to run the cookbooks

Downside of option one is that we need to have a database. This means that user needs to setup GitLab, upload the license and only then would they be able to setup HA. This seems like double work and I don't think anyone would appreciate it.

Downside of option two is that the user needs to place a license file across as many servers as they are using for the HA setup. This is cumbersome however, I do expect everyone to use some sort of configuration management system so this becomes less of a problem.

For our use case on GitLab.com, we already create gitlab.rb on all nodes using a chef cookbook which would make placing an extra file trivial.

From the development side, option 1 is much easier. Option 2 will require more coding.

Since customer comfort > development work option 2 is a clear winner in my mind.

So we should (re)use the gitlab-license gem. Natural thing to do this in a recipe/ separate cookbook however, chef is notorious for having issues with installing of the gem and loading it at runtime.

I think a better way is to create a separate "software" part in the package which will install the gem. Apart from that we will need some code that will read the public key and check the license so all that can be "baked" into the package.

This is in https://gitlab.com/gitlab-org/omnibus-gitlab/tree/create_license_check branch. It is trivial now to use the license check to decide if a cookbook should run.

One downside of this is as previously mentioned, there is a need to upload license 2 times.

I discussed this in a call with DouweM and we concluded that we shouldn't be blocked by this.

Few things:

1. We cannot be sure that the admin managing GitLab is the one that is going to setup GitLab
2. Renewals are also going to be an issue, which license takes precedence

One thing we might want to consider is creating a service that will do the syncing. To do it in GitLab rails we would then have to store the license file in a location where git user can read from. We could then create a job that would read the license from the filesystem and add it to the database on the first install. And if the license gets updated via admin panel, write it to the filesystem.

However, problem is that we need this license across the cluster and some nodes won't have knowledge of the database so it won't be possible to do it this way.

One other possibility is to disable a license view in the admin interface if we detect a license on the filesystem. Then you would have to upload it to one location only, either filesystem or in the interface.

I think this might be the best solution. Something to think about before finalising the feature.

Milestone changed to gitlab-ee%13

Added ~360295 label

Added ~360296 label

Removed ~360295 label

Milestone changed to gitlab-ee%14

Milestone changed to gitlab-ee%15

Added gitlab-ee~481018 label

Milestone changed to gitlab-ee%17

Milestone removed

I've updated the issue slightly to indicate our options.

---

@marin We should aim for the best solution for the customer. In this case, asking him to put the key on the FS somewhere is not what we should aim for:

• As mentioned earlier in the thread, that'd be an headache to know the SSOT between what's added through FS and through the UI
• It's one more thing to do in the CLI.
• You mentioned letting the customer add this key in Chef to propagate the key on all the servers needed for the HA setup, but we can't ask that to the customer. What if he forgets, what if the key changes over time, etc... The least amount of stuff he has to do, the better.

The best would be to read the license key that is uploaded through the GUI and use it.

That means

• we should let the first reconfigure run, but on a second reconfigure, if we notice that an HA product is activated in etc/gitlab.rb, we raise a flag during configuration saying that a license key is needed.
• we don't need to check what happens on slaves. Just do this check on the master if redis_master_role is enabled?

The best would be to read the license key that is uploaded through the GUI and use it.

I agree, admin interface should be the single source of truth. There is something to be said about finishing all work at the place you started, eg. I started setting up all instances and using CLI so I shouldn't go to the web interface. But we'll never win that fight anyway.

if we notice that an HA product is activated in etc/gitlab.rb, we raise a flag during configuration saying that a license key is needed.

This is where things get technically more challenging. From the reconfigure side, it has no idea about what is in GitLab. It has no idea if there is a license and what is its validity. We can fire up a DB query or connect using rails runner to fetch this data but doing this on every reconfigure is going to cost us a lot. So it is hardly an option. I wonder whether we can reuse the admin/health_check for this somehow. Or at least the idea of it where we generate a token and query an endpoint to get a true/false whether the license has HA enabled.

we don't need to check what happens on slaves.

I think that is a good idea, ignore secondaries and just handle all primaries that can get access.

added gitlab-ee333913 gitlab-ee139512 labels

We can fire up a DB query or connect using rails runner to fetch this data but doing this on every reconfigure is going to cost us a lot.

How come? Don't we have to reconfigure only when a change is made to GitLab manually, or during an upgrade?

When a configuration is changed or during an upgrade, correct. But if you are managing your cluster with a CMS, reconfigure can be triggered on every run. We could think of something like:

• Fire a query and get the license expiration date and whether license has an HA purchased
• Save that in our gitlab-secrets.json which is persisted
• If date is less than the current date, do another query and update the json file

This is, however, not covering the case when someone purchases a license after so its not full-proof.

FYI, in 9.2, we won't have the notion of Addon in the license anymore. We will only know if a license is premium or starter, which is enough for us here, and would serve the same purpose.

@marin if someone purchases a license after, and if we choose the solution you've highlighted in your last comment, considering that customers need to upload the license through the UI, can we imagine the application writing to gitlab-secrets.json to update the file with the most relevant info?

@ChadMalchow @amara currently we offer Redis HA to all EE (Starter and Premium), and in 9.1, we will also offer PG HA to all EE.

We think HA = premium.

This issue is used to determine which technical solution we can find to allow us to market, perhaps in 9.2, those HA features, and put them behind a license check like we do with Geo, File locking etc...

@regisF agree HA = Premium

@DouweM @smcgivern what do you think about this issue and more precisely, this comment (https://gitlab.com/gitlab-org/gitlab-ee/issues/654#note_26525750)? The question is how can read the license during reconfigure. Then, can we update an external file on license change through the UI, etc...

The suggestion from @marin is to avoid having to hit the DB every time, right? I think the idea makes sense in general, but the problem is that we need to 'push' updates from the Rails app to the Omnibus configuration, which is never going to be particularly elegant.

@marin this is very probably a dumb question, but earlier there was some talk of (I think) the Rails app writing the license to the filesystem for Omnibus to read. Is that possible?

The suggestion from @marin is to avoid having to hit the DB every time, right? I think the idea makes sense in general, but the problem is that we need to 'push' updates from the Rails app to the Omnibus configuration, which is never going to be particularly elegant.

Correct. At the moment I am just looking for a solution that works, elegance is last thing on my mind ;)

@marin this is very probably a dumb question, but earlier there was some talk of (I think) the Rails app writing the license to the filesystem for Omnibus to read. Is that possible?

The issue with that was that we would be doing 2 operations, namely saving to the DB and also writing to the file. If we are ok with that and all sync issues that can come, I am ok with investigating that road :)

Right, writing a file is decidedly non-elegant

My personal preference would be to write a file on license update, as well as updating the DB. It's not ideal if they get out of sync, but it's the solution that seems to involve the least complexity in Omnibus. We would still have to ignore a missing license on install, though.

@DouweM wdyt?

@marin @smcgivern I'm OK with writing to FS and DB every time a new license is uploaded.

When the instance already had a license in the DB before upgrading to the GitLab version we introduce this in, we can write it to disk at boot time in an initializer.

OK, awesome. @regisF does that help?

It does. So

• @DouweM or @smcgivern (but I have the feeling it's more platform): could you indicate the steps needed to make this work in the app?
• @marin can you write down the tasks needed to make the check work in Omnibus?

Then we can plan this and make it happen.

I'm not 100% sure I'll let @DouweM take that

gitlab-ee~481018 :

• Write license blob to disk at predictable location when new license is uploaded
• Write license blob to disk at predictable location in initializer when no blob exists yet at this location

@marin Do you have a preference for this predictable location? Could it just be in the gitlab-rails root?

gitlab-ee~155979 :

• Create a software definition that would build a script with gitlab/license gem
• Read the license encryption_key and import the license file
• Check the validity of the license
• Take actions based on the validity of the license

@DouweM Rails root is ok as long as we can flush the license to the disk at boot time. I would prefer if we had a better way to check this though, is API not an option?

I would prefer if we had a better way to check this though, is API not an option?

@marin A license API already exists, but that requires the Rails app to be running in order for Omnibus to read the license. I thought we didn't want that.

@DouweM then we have it all. If you need to add a new node, we can simply query the API without needing to sync the license. We can also use this as a form of health check.

/cc @joshlambert

@marin Note that the API currently requires authentication as an admin, but it would be easy enough to create a similar internal API that just requires the shell secret.

To try to recap and get a SSOT:

1. We would implement a license check on all HA nodes
2. On the master node, when loading the EEP license it will output a file on disk
3. On a slave node, Chef will make a call to the master node API to check license status
4. Chef can then read this file and enable HA if license >= EEP

If so, couple questions I have:

1. Do we need to protect the license file at all?
2. Is checking Redis easier, as you won't need to spin up rails? If so, could we save it there for both master and slave nodes? Just thinking of ways to prevent two different check methods.

• We would implement a license check on all HA nodes

Eventually yes. I would focus right now on getting the GitLab Rails nodes figured out.

On a slave node, Chef will make a call to the master node API to check license status

Second iteration. First one should be only master node.

Chef can then read this file and enable HA if license.

Sure. It will be a bit more work and some duplication with the Rails code, but it won't be hard to do.

Do we need to protect the license file at all?

Protect how? If it is just about visibility of the file, we will have to be owned by the correct user and tight permissions but we do that with other secret files too.

Is checking Redis easier, as you won't need to spin up rails?

I don't think it is easier. You still need to spin up Redis AND have GitLab Rails running. But maybe I am not following your suggestion.

mentioned in merge request gitlab-com/www-gitlab-com!6195 (merged)

added gitlab-ee~149397 label

@marin can we move this one to the Omnibus project?

added EE label

added HA label

added feature label

changed milestone to %9.4

@marin wrote down my thoughts on different solutions here: https://docs.google.com/document/d/1n-tBU1uY0y9ShQxp-vREsRzvcfCqaS7-L0bBHsWl8dQ/edit?usp=sharing

At the moment I still think having omnibus look at the filesystem for the file, and having rails write the license to the fileystem if it isn't there, is going to be one of the less complicated, but more flexible approaches.

@twk3 Awesome write up, thanks for that. I will discuss this with @joshlambert and we will try to get a decision out as soon as possible.

@joshlambert the label on this says EEP but on the direction page it is not showing up as a premium label.

@ChadMalchow this will be fixed on the next push to www-gitlab-com

There were couple of discussions on how to proceed further with this issue.

Due to technical limitations noted throughout this issue and summarised in https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2442#note_33723956, the decision was made to not technically gate any of the HA features built in the package. To expand on this:

• Using omnibus-gitlab HA features will still require EE Premium license
• Package won't prevent users from running built in HA features, but without EE Premium license you would be in breach of the license
• With HA solution available for use, we will start charting out what we can do from the UI side of GitLab to provide a "dashboard" that would be visible only to EEP licensed instances
• We will make sure that the documentation states EEP features clearly

/cc @joshlambert @markpundsack @stanhu

closed

@marin Makes sense to me, and thanks for summarizing. Let's just make sure that all stakeholders are aware, and don't have any surprise gotchas we're not aware of. /cc @sytses, @ChadMalchow, and @joe.s.

when we say

" Package won't prevent users from running built in HA features, but without EE Premium license you would be in breach of the license>

What communication/alert can/or is visible to the customer making them aware of what they are about to do or just did is not part of their license and they need to upgrade to EEP? How can we be alerted of who is in breach of the license agreement so we can engage with them too?

From a customer perspective, not knowing that I am in breach is not good. What is worse is not knowing then being told I need to upgrade or remove the features. Thinking out loud what sounds ideal is having an in product alert/message, informing them they need to upgrade to premium if they want to continue with the HA features.

@ChadMalchow I think we are on the same page. Initially we were looking at ways to prevent a user provisioning HA, but this happens very early in the process and before a license is loaded. So we'd have to ask them to take extra steps, complicating the setup process.

Instead what we are proposing is to introduce a place in the UI, similar to Gitlab Geo, where they can look and see the status of HA. For example this could provide statistics on the health of the nodes, the number, etc. (Maybe they self-report this information to Prometheus or the DB, and we can read it from there.)

If we detect an EES license, instead of showing that we would show them they need to upgrade to EEP. Our documentation would clearly state EEP, and then also direct them to this UI as one of the steps. This way there should not be surprises.

@ChadMalchow @joshlambert Apart from making it clear in the documentation, we can also note it down during the runtime if you think this will help. My expectations are that users will still miss the notice since there are a lot of notes during setup, but maybe it would be helpful?

New license check issue: https://gitlab.com/gitlab-org/gitlab-ee/issues/2850

Will not be for the database configuration, but instead at the GitLab app side for the load balancing code.