Customer would like support for configuring repository mirroring with Git over SSH, and thus, support for SSH keys. This is in addition to the current support for mirroring via http(s).
We need to update the Mirror repository settings view
We need new types of user/server credentials. Hence, we'll add a dropdown in this screen to choose between:
Password authentication (as it is now)
New protocol: SSH public-key authentication. We generate a private key and display the public part to the user so they can copy and paste it onto their server, or into GitLab as a deploy key for the source project.
known_hosts management:
Should be invisible for non-SSH URLs
When we setup SSH (which can be password or pubkey-auth-based), we need to detect the host keys and present their fingerprints to the user to confirm that they're valid.
When we edit SSH setup, we also need to re-trigger the verification
Advanced users may want to enter known_hosts data directly. They should be able to click a button to get a <textarea> they can input directly into, rather than using automatic detection
Who verified the host keys (i.e., submitted the form), and when, should be displayed for audit purposes.
We generate per-project SSH keys for mirroring: we expose the public key in mirroring settings,
We allow to paste private SSH key when setting up the mirror and it is stored internally,
1 requires less work on the user's side, so I prefer that.
By using global GitLab ssh private key we allow anyone to mirror the repository as long as he knows the URL.
Right, so that's not an option.
I'm not sure how to handle the known_hosts issue, but this can lead to silent error when the host keys changes for remote server (quite unlikely to happen).
The error doesn't need to be silent, we already show mirroring error messages:
@vojtasvoboda.cz The SSH key of the GitLab server is located at /var/opt/gitlab/.ssh/id_rsa, with its public key at /var/opt/gitlab/.ssh/id_rsa.pub. If this file doesn't exist yet, run sudo -u git -H ssh-keygen, and accept the defaults when prompted. This SSH key will be used to authenticate with the remote SSH server, which you would need to upload the public key to.
Hello, we would like to mirror remote repository to gitlab.com using ssh key.
However we could not find what public key should we use to set up as deploy key on the remote repository.
Is there a way to solve this now? Is there any way we could help?
Thanks, JohnnyB
If I try to ssh with keys, I get permission denied error on frontend list. When I do the same as git user on gitlab server, it works.
Does the mirror script run under git user?
Before my question: I’m user of gitlab.com service. There is a git with automatic deployment on my hosting. I normally push commits from my local repo with my private key (public is set on hosting – I can set more than one) and it works like a charm. Now I’m trying to achieve a situation, where I push commit on my hosting and gitlab.com mirror the repo or vice versa.
But it doesn’t work. In the setting it says:
If your SSH repository is not publicly accessible, add the public SSH key of the GitLab server to the remote repository.
I suppose the gitlab.com website should tell me the public key, but it’s nowhere. So I put a key which I found on the internet, which is suppose to be from gitlab.com, but I’m getting this error
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
So either I have a bad key or there is some bug in connection between gitlab.com and my hosting.
oops
The page you were looking for doesn't exist
You may have mistyped the address or the page may have moved
It's not there. I don't really get it. This feature has been there for a long time. I found another people asking for the public ssh key of gitlab.com but nobody ever answers. Is it just me or it hasn’t ever worked?
Same here this is completely broken, also if I enter the default github ssh address that use from command line it ends up on a 500 because it gets confused with a semicolon. Also where are the ssh keys.
I'm also getting a lot of 500's now. I figured out, it's based on the url i put there. Some causes error, some doesn't.
It woul'd be just nice to here from the devs 'it's broken guys, just wait for the fix' or at least something.
I've just tried that, used that key on my hosting, right next to my personal public key, which works. Still getting this error
500576Host key verification failed.fatal: Could not read from remote repository.Please make sure you have the correct access rightsand the repository exists.
I think to support SSH properly, we should make encrypted_credentials be structured data. It can then carry username, password, client_pem, and server_pem, supporting the following authentication possibilities:
HTTP(S) username & password
SSH username & password
SSH username & public-key authentication
client_pem and server_pem could also permit client certificate authentication for HTTP(S). I don't know if that's ever been requested, though!
We'll need a bit of a UX overhaul - maybe away from entering userinfo in URLs and towards a tab per protocol. It's too easy to enter invalid URLs at the moment, anyway.
In the "SSH" tab, entering or changing the hostname & port would do a callout to a (Rack::Attack-protected!) endpoint which looks up the 'server_pem' (which is to say, the SSH host key). The fingerprint can be presented to the user immediately for approval or rejection. So by the time the form is submitted in its entirety, the value is already known.
The worker writes out a script that sets up ssh according to the given values, then does GIT_SSH=thatscript git ... as usual.
A naive implementation of that would write the SSH private key to the filesystem, or at least to tmpfs, for the duration of the mirroring operation. That might not be reasonable - thoughts? The expert-level option is to speak the ssh-agent protocol somewhere to provide access to these credentials.
The known_hosts is easy enough, you just write a conforming file and ensure the generated GIT_SSH script has -o UserKnownHostsFile=thatfile
Existing setups where users have tweaked the git user's SSH client configuration on their own server can continue to be supported with a "Let me manage the SSH client configuration manually" checkbox.
When checked, this would hide all the new fields and not set GIT_SSH - so falling back to the default ssh client config for the git user, as happens now.
@regisF@JobV This seems like a highly requested feature (at least from the support perspective) along with support for importing over SSH (need another feature proposal). It could also be seen as a differentiator from other services that don't provide this kind of functionality. Any possibility it can be assigned/milestoned for a future release?
As a workaround, I have managed to mirror things to/from Github by using HTTP authentication and entering the URL in Gitlab like http://user:pass@github.com/skorokithakis/myrepo etc.
a public key, that we generate on a per-project basis, that the user has to paste to the remote server
There is a fourth option: when users want to use their own key instead of the one we provide. I don't know if this use-case is heavily used though. I need your thoughts on that, and data too.
And that's it. There should not be other changes on the user-part.
Option-3 is more likely in a corporate environment (There is a fourth
option: when users want to use their own key instead of the one we provide.)
A team is assigned a 'bot-ci' user and a set of keys that are placed on
Github or external git system. That team is not able to modify or add a
public key to the remote system.
Another thing to consider is known_hosts handling. I wrote something on it a while ago but can't find it now. We have to verify the SSH host key of the remote end. There's no DNS or CA infrastructure to speak of (unlike HTTPS), so the only option we really have is to present the fingerprint to the user and ask them to verify it.
In the "new" action, whenever the hostname or port changes for options 2 & 3, we can use JS to ask a heavily cached, rate-limited and ability-secured endpoint to retrieve the SSH host key fingerprint for the hostname+port. This is presented to the user for them to verify (or blindly click past) and submitted with the rest of the form when creating.
In the "edit" action, changing the hostname or port will again need us to verify the host key. In cases where the hostname and port haven't changed, but the remote has changed the host key and mirroring has broken as a result, we'd need a "rescan" button to retrieve the new one.
Once we have it, getting git clone git+ssh://... to respect it is fairly easy - I suggested a method using GIT_SSH above
The known host issue is easily verifiable with a static remote, however
Github are known to use AWS for their offering and as a result if load
balancers change IP's, then you are likely to get an error on the new
endpoint. That has to be addressed without a sync failure.
the ability to clone via ssh key is quite important, because often username and password give also access to the whole webaccount of the repo, which you might not want to add in the mirror url. ssh keys allow the access to the repo only.
edit:
additional you can't use complex passwords any more, you are restricted to passwords which can't contain specific chars or else you will get:
edit2:
i personally would prefer to fully (manually) control the ssh keys, known_host and maybe also the ssh_config files for the mirror and remote push process.
@regisF I think we support it now; removing support for it would be peculiar.
Accepting the host key without some form of verification would be a security risk - think of it as being on the same scale as configuring gitlab globally to ignore SSL certificate verification.
@nick.thomas for existing setups (as you mentioned here), can't we just let "as is" in terms of UI? That way, we don't say in the UI that we support custom setups, but we still support existing ones (under the hood)?
@regisF we could, but I'm a little worried about the transitionary period where someone has an existing, working setup (albeit undocumented), upgrade their GitLab instance and discover that they can't use it for any new projects.
I don't see how to preserve the current behaviour for new projects without some UI - or how to signal that an existing project is using it, for that matter.
@nick.thomas I see. But why couldn't they still setup custom solution for new projects? I don't see which changes introduced with this feature would prevent them to continue to do so.
@regisF so they create a new project. It doesn't have any mirroring enabled. They go to the "set up project mirroring page", and are presented with a list of options. They need to do something, or mirroring will never be attempted. What do they do?
Say they pick "SSH with public-key authentication" (option 3, above). This will necessarily disable the implicit SSH client configuration to make it pick up the autogenerated key, so their new project won't use it.
@nick.thomas but how are they doing today? They only have one choice (HTTP, HTTPS or git) (which is option 1 above for us). So wouldn't they do the same?
They enter a URL like ssh://username@example.com/project.git and there is no option to enter an SSH key. That, the SSH client configuration that causes it to be used, and the known_hosts entries, are all picked up from the default SSH client configuration on the gitlab server.
My UI proposal above was to split up the view so each existing URL type is handled with separate inputs; if the only options for SSH require password to be entered, or generate and force the use of a specific private key, then there is literally no way to request a new project to use the existing scheme, or to signal that an existing project uses it.
@regisF Do you think we need any guidance next to the SSH key so people know what to do with it? Or maybe a link to some documentation? This might not be needed and I don't want to go overkill -- just want to make sure things are clear to people, though. Thanks!
@hazelyang This would be after hitting "Verify SSH connection" and after saving a new SSH URL. On the command line, it looks like this:
% ssh git@gitlab.comThe authenticity of host 'gitlab.com (52.167.219.168)' can't be established.ECDSA key fingerprint is SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw.Are you sure you want to continue connecting (yes/no)?
Then when you type yes, you get:
Warning: Permanently added 'gitlab.com' (ECDSA) to the list of known hosts.
We could use the same message, but using a button instead of typing yes/no, of course.
@nick.thomas Do you have any extra thoughts? I'm thinking of asking you to work on this for 9.5 :)
@DouweM awesome, this would be right up my street :)
I think I'd have a text input inline (i.e., no modal dialogue box) to display the fingerprint, and a checkbox for the user to confirm that they have verified the fingerprint. When the JS in the browser fetches a new fingerprint, it can uncheck the verified_host_key checkbox.
If you try to create or update SSH mirror settings and the verified_host_key checkbox isn't set, the create / update can be rejected. The form submission can be disabled clientside too. In this way it's roughly analogous to
We do need the button to manually refresh the fingerprint - "Verify SSH connection" - but if the existing fingerprint is blank and unverified, we can probably fetch the new fingerprint automatically when the URL input loses focus and the hostname has changed.
We can probably achieve the 90% case by including a hardcoded hostname -> fingerprint map, and automatically checking verified_host_key if it matches.
If you'll excuse the terrible GIMPing:
The wording is a bit shaky and we might want red / green background hinting and some visual grouping of the text input, checkbox and verify button to make it clear what's happening, but that can be worked out through experimentation.
I think I'd have a text input inline (i.e., no modal dialogue box) to display the fingerprint, and a checkbox for the user to confirm that they have verified the fingerprint. When the JS in the browser fetches a new fingerprint, it can uncheck the verified_host_key checkbox.
@nick.thomas What do you think if we have a dialogue box(01) to let the user confirm the fingerprint after clicking 'Verify SSH connection' button? I think it's more usual to use checkbox to activate a feature than to confirm something.
We could also include some help text on either the input or the checkbox
If we use dialogue box, the help text will be next to the button.
I am wondering if we need to display who verified the fingerprint manually. We perhaps can remove it.
@hazelyang I shyed away from the dialogue box in favour of having the inputs inline because it makes sense for the user to be able to manually input or remove a host key to use.
Checkboxes are used for confirmation in various scenarios - accepting EULAs comes to mind for instance.
What we're asking the user to do is to warrant that the information in front of them is accurate. I guess I don't mind too much whether it's a button or a checkbox, as long the user can't click through without taking some sort of additional action. We could have a pair of buttons - good / bad - instead of a checkbox.
@filipa I may need some frontend help later on in the month - I'm putting together simple, stupid, ugly bootstrap in https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2423 to help me build the backend, but I doubt it'll be production-quality. No action needed yet, just a heads-up :)
Presumably we want to keep the generated SSH private key gitlab-server-side only - so we can't show the public part (ssh-rsa ...) until the mirror settings have been saved at least once.
We'll also need to allow users to regenerate the key somehow.
I propose we generate 4096-bit RSA keys by default. DSS keys are unusual, and the elliptic options are less well-supported. Generating such a key takes perhaps a second (although running out of random bytes will slow it down).
It seems peculiar to implement this for pull mirrors without also implementing it for push mirrors, but I see now there's a lot of wide open space between the two features in the backend. @DouweM should we open a separate issue for push mirroring and perhaps try to get it in in %9.5 as well?
I'm leaning to the tabs beingPassword authentication | SSH public key authentication rather than HTTP | SSH. This is closer to what we're actually doing, and affords very good backward compatibility with these existing use cases:
Global SSH key in ~git/.ssh/id_rsa
SSH password authentication
They're both purposefully undocumented at the moment, but we have existing EE customers using them regardless.
It's also good if we need to add "HTTP client certificate auth" in the future.
We'd move the "Git Repository URL" out of the tabs, as it's common to all of them.
@DouweM should we open a separate issue for push mirroring and perhaps try to get it in in %9.5 as well?
@nick.thomas Please create a new issue, but only try to do it for 9.5 if it doesn't get in the way of any of your other assignments. We are only targeting pull mirrors right now as a way of reducing scope and increasing the likelihood of us shipping something at all :)
I'm leaning to the tabs beingPassword authentication | SSH public key authentication rather than HTTP | SSH. This is closer to what we're actually doing, and affords very good backward compatibility with these existing use cases:
@nick.thomas But isn't there also a difference in what URL format we'll allow? It does feel more like HTTP URL vs SSH URL to me.
@DouweM different URL schemes are valid depending on the authentication system. We can work that into both validations and frontend.
Unless you mean we should support username@foo:bar/baz.git in the SSH authentication case? That would make sense, given it's the form gitlab exposes the SSH repositories in, but we'll still need to support the explicit ssh:// scheme as well to handle custom ports...
Reading the host key fingerprint needs to happen in the background; the frontend will need to poll until it's done, which means we'll need somewhere to store the fingerprints. This is probably best put into redis.
A thought. Currently, user-password authentication over SSH "almost works" for pull mirrors. All that's missing is SSH host key management - right now, the code path uses the system git user's configuration, as it does for public key authentication.
I think we should promote the SSH host key management UI so it applies to both user/pass and public-key authentication, only showing it when the URL is ssh://. For backward compatibility, we can fall back to the system configuration for host keys if this configuration is present.
Two benefits:
Use user+password SSH URLs on GitLab.com, and elsewhere
Pubkey auth gets the global known_hosts config fallback, which may be useful to some on-premises installations.
The UI will also make more sense. Now I've noticed it, the placement of SSH host keys with just public-key authentication is really bugging me!
@nick.thomas Do we still show the user the fingerprint rather than the known_hosts format when they're asked to verify it? Sysadmins of both private and public Git instances often tell their users "The fingerprint for X will change to Y on date Z" over twitter for example, which is much easier to manually verify than the entire key.
I also worry that presenting it as a textarea they can edit may be confusing to users looking for a single fingerprint.
@DouweM the ssh_host_keys detection endpoint presents both the detected known_hosts entries and the fingerprints for each entry (indexed by line number). I'd expect the complete frontend to show the fingerprints to the user, but put the known_hosts entries into the form.
@nick.thomas@hazelyang I've rearranged button position and updated label for SSH host keys input instead of showing checkbox without further complicating UI, it looks like this.
@hazelyang As for fingerprints data, here's the example data that we need to show somewhere on UI when user has provided SSH URL and clicked Detect host keys button;
I think it would be reasonable to disable the detect host keys button once a success result has come back... in case of an error, the user might want to try again. It's all pretty cheap though, so I wouldn't be concerned.
@hazelyang We missed out on adding Copy to clipboard button for copying SSH public key, I've added that button for now, but it doesn't feel right, can you suggest better way to do it?
Awesome that this was recently merged! Any chance to add the same feature for Push mirroring? Use case is we re migrating from Redmine with "gitolite" to Gitlab EE. We'd like to push changes back to Redmine to reflect merged branches but unfortunately gitolite only supports SSH protocol.
It references a couple of technical debt issues thrown up in the implementation of pull mirroring support. Once those are done, it should be quick and easy to add push mirroring support.
Running GitLab EE 9.5.2 on git 2.7.4 in production, the push over SSH is working out for me, target is a simple gitolite 3.6.6 on git 2.7.4, I authorized the key from /var/opt/gitlab/.ssh/id_rsa.pub.
Allows us to have a very easy to deploy/setup/admin remote async backup of your git repos thanks.