Port usage ping from EE to CE

Description

We want to add the EE usage ping feature to CE (https://gitlab.com/gitlab-org/gitlab-ce/issues/23361).

We need to fine tune the content of what is sent. For instance, the license_md5 in CE makes no sense.

Data that we will collect will be anonymous. We don't want to record who the owner of the instance is, nor which email address is used for the administrators of the instance.

Proposal

• Currently, usage ping is defined by a license id. We need to add a way to identify a unique instance in order to track the usage over time, but without knowing who the instance actually is: UUID.
• Add a new field in the usage ping to indicate that the ping is CE edition: 'CE'
• Get rid of the following fields from the JSON payload (send NULL for those fields?):
  • license_md5
  • licensee
  • license_user_count
  • license_starts_at
  • license_expires_at
  • license_add_ons
  • In the statistics themselves:
    • geo_nodes
    • protected_branches
• Once set, usage ping preferences should remain unmodified by GitLab when updating to another version.

Reference

• Usage ping is sent from there: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/usage_data.rb

@DouweM what can we use to uniquely identify a CE instance in the version.gitlab.com database (while renaming anonymous)?

@regisF We may want to consider generating a random UUID for each GitLab installation and use that. That would help us identify different instances of EE as well.

mentioned in issue #23361 (closed)

assigned to @rymai

changed milestone to %9.0

changed title from Add usage ping to CE to Port usage ping from EE to CE

removed milestone

removed assignee

@regisF We could use a hash of the hostname.

@DouweM But the UUID shouldn't change if the hostname change?

@rymai indeed, if it'd change, that would defeat the purpose.

@regisF @rymai Of course, but that happens very rarely, and really, what is the definition of "a unique instance" if it isn't "an instance with a specific hostname"?

@DouweM Your solution would work but I don't see the advantage over simply using SecureRandom.uuid?

@rymai We could do that, but the instance would need to store that UUID somewhere, which I'd rather not.

We could do that, but the instance would need to store that UUID somewhere, which I'd rather not.

@DouweM May I ask why you'd rather not do that? It probably depends on what we want to do with the data:

• If we want to track the "lifecycle" of an instance, it's better to have something that will never change, otherwise you'd loose all the "history" of this instance => UUID
• If we just want to track usage in general, e.g. "instances active in the last 30 days", a hash of the hostname would suffice since hashes that don't reference a live instance would become stale and not be included in these usage reports after some time...

By looking at the current usage, https://version.gitlab.com/licenses falls in the first category (usage per instance) while https://dev.gitlab.org/gitlab/version-gitlab-com/issues/47 falls in the second one (global statistics), so .

@rymai Fair enough, I'd be fine with a UUID or hash of the hostname, the latter is just slightly easier to implement :)

Added in the body the edition field to indicate that the usage ping comes from CE.

We'll store a UUID in the application settings model with a migration, and use that as the unique host identifier: https://gitlab.com/gitlab-org/gitlab-ee/issues/1894

cc @dbalexandre

changed milestone to %9.1

@smcgivern who should be the assignee for this one?

@regisF I should be!

assigned to @smcgivern

mentioned in merge request !10481 (merged)

marked the task Currently, usage ping is defined by a license id. We need to add a way to identify a unique instance in order to track the usage over time, but without knowing who the instance actually is: UUID. as completed

marked the task Get rid of the following fields from the JSON payload (send NULL for those fields?): as completed

@smcgivern at the moment the JSON representing the usage ping in the interface is both in the Settings page and the new Cohorts page. We should only have it on the Cohorts page.

The decision to remove the JSON from the Settings page comes from the fact that on this screen, it's one line which makes it hard to read and hard to scroll, therefore making it almost useless to have it here.

Fixed in 8d10add4, which is part of https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10481

closed via merge request !10481 (merged)

mentioned in commit 40a97205

mentioned in commit b9fff815