GitLab Geo: SSH keys Sync

Rationale

Current Geo (#76) implementation allows both ssh and https clone on a secondary node, but only https is being actively synced (it's synced by the database replication).

SSH keys are also store on the database, but requires an extra step to be useful (it must be added to .ssh/authorized_keys by gitlab-shell).

SSH key changes doesn't happen so often, but as changes to the .ssh/authorized_keys can only happen by one process at time (because we have to acquire file locking to prevent data-loss), it's a good candidate to be enqueued/buffered and updated in batches.

gitlab-shell only provides batch support to add keys, not to remove old ones. The primary node also doesn't do any batched operation, so we can safely postpone any decision on that matter and change our minds in the future if that proves to be a bottleneck.

SSH Keys Sync

As changes to .ssh/authorized_keys involve file locking, buffering changes is not useful unless we can also make changes in a batched way on each secondary node.

I'm proposing we send notification changes immediately in an async way to each secondary node. So after any addition, update o removal of keys we should create an async job to notify nodes.

Job retries should be customized to happen more often. It is expected to have at least the first execution failing because of database replication latency. We can either start by scheduling job execution on the secondary node to be performed after X amount of seconds or let a custom retry logic deal with that (documentation here).

I'm more inclined in using a custom retry logic of about 5 to 10 seconds (randomly).

We should always query database for the key when updating in a job or we could replace an older key when multiple updates happen at the same time for the same key.

Checklist

• When a key is created or removed we notify secondary nodes of that change.
• A secondary node must receive a notification to change .ssh/authorized_keys

When a notification to modify authorized_keys is received:

• It must generated async job to add or remove keys
• Generated job should retry a reasonable amount of times for a short period of time (30 times, waiting from 5 to 10 seconds)
• Generated jobs should also have an exponential backoff logic after the short period of time retries.

Title changed from GitLab Geo: SSH Keys Sync to GitLab Geo: SSH keys Sync

Added ~164929 label

mentioned in merge request !236 (merged)

https://gitlab.com/gitlab-com/operations/issues/99 is sort of related

mentioned in issue #76

I have to add an additional data when updating keys: updated_at timestamp.

This is important because database replication is a different communication pipe than our Geo notifications.

It's very unlikely to happen, but theoretically possible, that we have 2 or more different updates very quickly to the same key, that will generate one update notifications for each event, that the last one can happen before database have replicated enough data to get the most recent data change and therefore, not replicate all changes to the disk.

If we send the timestamp and execute the job only if model.updated_at >= notification.updated_at we are good.

action	pre-condition to execute
create	model.id must exist
delete	model.id must exist
update	model.id must exist and model.updated_at >= notification.updated_at

I saw we don't ever update a key, so, we are good :).

mentioned in merge request !282 (merged)

Marked the task Generated jobs should also have an exponential backoff logic after the short period of time retries. as completed

Marked the task Generated job should retry a reasonable amount of times for a short period of time (30 times, waiting from 5 to 10 seconds) as completed

Marked the task A secondary node must receive a notification to change .ssh/authorized_keys as completed

Marked the task It must generated async job to add or remove keys as completed

Marked the task A secondary node must receive a notification to change .ssh/authorized_keys as incomplete

Marked the task When a key is created or removed we notify secondary nodes of that change. as completed

Marked the task A secondary node must receive a notification to change .ssh/authorized_keys as completed

Status changed to closed by commit 891b27c073b359a5928f0f0e89ceff235d6a14b9

mentioned in commit 891b27c0

mentioned in commit 4e883231

Mentioned in commit pfjason/gitlab-ce@891b27c0

Mentioned in issue #76