Geo: automate SSH authentication for replication
One of the most error prone step in setting up Geo (#76) is the known_hosts step. We should try to make it easier, while keeping it safe.
Geo nodes get two new attributes: ssh_private_key and ssh_known_hosts.
The first is used by secondaries to authenticate against a primary. When authenticating, they verify the primary against its ssh_known_hosts data.
The primary is responsible for filling these fields - in effect, it's handing credentials to the secondaries for them to use, through an authenticated channel.
The existing model (~git/.ssh/id_rsa files automatically generated by Omnibus, manually copying the public key into the Geo nodes screen, etc) will be completely removed. We can manage these details completely automatically.
Based on the public backup-tool from GitHub, looks like they internally use a lot of StrictHostKeyChecking=no. I don't think this is a good default, as it opens to MITM attacks. This is not something we want to happen with a Geographically Distributed install, so I think it's a no go for us.
The other alternatives I can think of is:
- Create a rake task that should be executed from the primary, that will get current machine fingerprint, connect to all the secondary nodes, and make sure
known_hostsare updated if the fingerprint matches. - Require the fingerprint to be defined as part of adding a node, and use that information to automatically add the host to
known_hostswhenever necessary (each node independently after first failure).
First alternative requires someone connecting with either root or git credentials on all machines, to have the rake task executing properly
Second alternative requires another manual step to be made before adding a node (we already require the user to log into each node to generate and retrieve the ssh-key, so it will be an extra similar step).
We can also improve the rake gitlab:check to check that for us and help with the effort of the Monitoring feature (gitlab-org/gitlab-ee#727).
Based on http://stackoverflow.com/questions/12136998/how-to-check-if-a-host-is-in-your-known-host-ssh :
root@gabriel-sentinel-rails:~# ssh-keygen -F gitlab.com
do_known_hosts: hostkeys_foreach failed: No such file or directory
root@gabriel-sentinel-rails:~# ssh gitlab.com
The authenticity of host 'gitlab.com (104.210.2.228)' can't be established.
ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'gitlab.com,104.210.2.228' (ECDSA) to the list of known hosts.
Permission denied (publickey).
root@gabriel-sentinel-rails:~# ssh-keygen -F gitlab.com
# Host gitlab.com found: line 1
|1|gSjzNkA2KzoP/uve6uktbTFieVo=|zutbQd6eOpq3q43fn/fNHRhODaw= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=