authorized_keys file not rebuild on Geo secondary nodes

Summary

The authorized_keys file is not rebuild on secondary nodes after the database replication. Meaning only new SSH keys are added via the system web hook.

Steps to reproduce

• Set up primary node.
• Add users, including them adding their SSH key to their profile.
• Set up secondary node (syncing, etc.) according to the documentation.
• Try to clone from the secondary node with the keypair configured in the primary before setting up the secondary.

What is the current bug behavior?

Unable to clone from the secondary node using the key added to the primary node.

What is the expected correct behavior?

Able to clone from both nodes.

Relevant logs and/or screenshots

Primary clone

$ git clone git@first.mrchris.me:root/easy-config.git
Cloning into 'easy-config'...
remote: Counting objects: 1057, done.
remote: Compressing objects: 100% (531/531), done.
remote: Total 1057 (delta 473), reused 1057 (delta 473)
Receiving objects: 100% (1057/1057), 773.95 KiB | 118.00 KiB/s, done.
Resolving deltas: 100% (473/473), done.
Checking connectivity... done.

Secondary clone

$ git clone git@162.243.88.8:root/easy-config.git
Cloning into 'easy-config'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

authorized_keys

The authorized_keys file is not propagated on the secondary node.

Primary

root@first-mrchris-me:~# wc -l /var/opt/gitlab/.ssh/authorized_keys
6 /var/opt/gitlab/.ssh/authorized_keys

Secondary

root@162.243.88.8:~# wc -l /var/opt/gitlab/.ssh/authorized_keys
0 /var/opt/gitlab/.ssh/authorized_keys

Workaround

Execute the following on the secondary node

sudo gitlab-rake gitlab:shell:setup

Links

• https://gitlab.zendesk.com/agent/tickets/76969
• https://gitlab.zendesk.com/agent/tickets/76039

@stanhu @dbalexandre @brodock

Tested this on 9.2.0-rc6. - This issue occurs because we never rebuild the authorized_keys file after replicating the database.

We just need to run something like this on the secondary node - https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/tasks/gitlab/shell.rake#L112-119

    Gitlab::Shell.new.batch_add_keys do |adder|
      Key.find_each(batch_size: 1000) do |key|
        adder.add_key(key.shell_id, key.key)
      end
    end

changed title from SSH keys not initially replicated to secondary Geo nodes to authorized_keys file not rebuild on Geo secondary nodes

changed the description

Ok, so this is actually a step documented in the Geo setup guide https://docs.gitlab.com/ee/gitlab-geo/configuration.html#step-6-regenerating-the-authorized-keys-in-the-secondary-node

As two customers have ran into this issue (plus me). We have a problem with the documentation. Step 6 should be completed before adding the node in admin settings (Step 4). People are assuming once you add a node in the Admin=>Geo Nodes section and the status is OK that the process is complete and this is where customers stop reading.

added ~1052112 feature proposal and removed ~111707 ~91797 labels

We can solve this by updating the docs (moving steps around) or making this process automatic, which should be low weight.

added SP2 and removed SP1 labels

Ultimately, I think we are going to want to push Geo users to set up authorized_keys lookups via the DB: https://gitlab.com/gitlab-org/gitlab-ee/issues/2082

This doesn't work for CentOS, but in checking with @sytses that is okay for now.

@stanhu I'm the guy behind Zendesk ticket 76969.

We're running CentOS 6, so a solution that works for CentOS would be highly appreciated! :-)

@ulja2852 Thanks for the information. As described in #2082 (closed), CentOS currently supports an old version of OpenSSH, which prevents us from using a feature that allows us to have a single source of truth for the SSH keys list.

Would you be open to running GitLab in a Docker container to get around that limitation?

@stanhu I understand why we want to move away from authorized_keys to lookups via DB, but I believe we still need to support both until we deprecate support for all "older" OS versions.

Can we consider the authorized_keys as a fallback but use the DB when system supports it?

I think I agree with @MrChrisW that we can improve the documentation as a quick fix.

I agree with the documentation as the frist step.

Supporting the authorized_keys in a sequential playback log adds quite a bit of complexity to our architecture, as we have seen in the discussion about idempotency and locking.

@mydigitalself This is a support scheduling request. We'd like to get this issue prioritized. I'm assuming Geo falls into Platform, please let me know if doesn't.

@cpallares Would updating the documentation suffice?

@stanhu I guess it would. I volunteer to update the docs

We are going to recommend that people set up Geo SSH authorizations via database lookups. CentOS users can install a custom RPM package for this. For more details, see: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2191/diffs.

mentioned in merge request !2218 (merged)

a note from a support call:

When there are many Geo secondary nodes (it happens with 7). The progress indicators on the Geo nodes page gets collapsed and there is no way to see them in the UI. Whoever is working on the progress indicator, please test with 7 or more secondary nodes.

@mydigitalself Geo is blowing up Support queues all around. I'd love to see this get fixed in a release soon. This is a support prioritization request.

changed milestone to %9.5

@stanhu just to confirm, this is for you to confirm that it can be scheduled as part of ~geo.

cc @brodock

I think the action items were to:

• Update the documentation: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2218
• Support SSH key lookups via the database: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2191/diffs

The other remaining issue we should consider is to make it easier for CentOS users to install a newer version of OpenSSH (e.g. by providing our own RPMs).

Should we create a new project in https://gitlab.com/gitlab-org and code an automated build with the CI based on the documentation?

Also should we provide the packages in the same package server namespace we use for GitLab or a custom one? (I'm assuming a custom one is better as it would not force other customers using gitlab to install the new openssh if they don't want to), or can we decide it's mandatory for for %10.0 ?

cc @marin @stanhu

@brodock We should create an issue on how we might to make make a custom package of OpenSSH work. For example:

• Do we want to replace the existing OpenSSH package and run on port 22 and somehow preserve the existing pam.d rules (or somehow use the ones that RedHat provides by default)?
• Do we run as a separate process (e.g. on port 2222) and co-exist with the existing OpenSSH package?

Related discussion here: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/1471

changed milestone to %10.0

mentioned in issue omnibus-gitlab#1471

removed Platform label

Since we've removed system hooks for %10.0, the only working mechanism is now the database lookup - so we can consider this issue done, I think.

Making life easier for CentOS 6+7 users is being tracked in #1471 so I'll close this one.

closed