Support for SSH with LFS

If you have remote set to SSH at the moment LFS client will automatically fall back to HTTPS.

Would be nice that GitLab sends response to the LFS client with download/upload hypermedia.

Will require changes in gitlab-shell and GitLab. Possibly in GitLab-workhorse.

cc @JobV @DouweM

Added ~127618 label

@marin it seems like it already works now. Is the LFS client guessing the HTTPS URL based on the SSH URL?

@jacobvosmaer Yes, LFS will try to append the correct path and see if it gets a response via HTTPS.

Added lfs label

Had a short brainstorming session with @jacobvosmaer about this:

• If we switch to using sessions to authenticate users, in theory we can get rid of password prompt for cloning via HTTP and reuse that for SSH clones
• We would need to add support for sessions
• Add API endpoint for authenticating user and returning the session
• In gitlab-shell properly handle git-lfs-authenticate, return the hypermedia with session to the client and instruct the client how to reach workhorse
• In workhorse switch from Basic Auth when it comes to lfs request, to first check for the session

This will require some work and I think that showing lfs objects should be higher priority.

@JobV In light of this I would propose that this issue gets moved to a later milestone.

@marin I agree. Is there anything that we can / should split off here to do in this milestone?

Milestone changed to 8.5

For security, some users like to place the gitlab HTTPS port behind a CDN such as cloudflare or akamai. CDNs sometimes have file size limits on uploads or downloads (CF is 100MB on their Pro plan). Uploading a large lfs object exceeds this pretty easily, so if you use lfs you're pretty much forced to not use a CDN on HTTPS (unless there's a workaround I haven't thought of).

If ssh was possible with lfs, that would allow a separate secure channel for git / git-lfs to upload through (by specifying gitlab_ssh_host in the config) while still allowing for HTTPS to be behind a CDN.

I just wanted to throw this out there as a use case for git-lfs ssh support in case it hadn't been considered.

thanks @ajohnsn, I didn't consider that. We fully intend to support it (personally I'd have it rather yesterday than today), but we want to make sure it's stable and works well before adding further complexity (see @marin's comment above).

@ajohnsn Thanks for your input!

SSH support won't change anything in the way Git LFS works, object will still have to go through HTTP(S). Basically the only change here will be in how internally we handle the request but the upload or download will always have to go through HTTP(S).

If you do require large objects to actually be pushed through SSH then git-annex is the only option (for now).

@marin Thanks for the clarification, I misunderstood and thought that the object would be pushed through ssh.

It appears that a pure SSH path would actually require deeper changes to lfs, and much work has been done on this front. Please see: https://github.com/github/git-lfs/pull/378 and https://github.com/github/git-lfs/issues/779

@ajohnsn

@marin I don't think we need to do something special with authentication in gitlab-workhorse for seamless SSH support; it is all just HTTP headers.

I think, provide git-lfs-authenticate command by GitLab LFS would be good enought: #3913 (closed)

mentioned in issue #3913 (closed)

Just visiting to say: It'd be really great to have this feature - authentication via HTTPS isn't ideal for me and my team.

So, thanks for any work being done on this! :thumbs_up_reverse:

I have LFS-enabled repos for which non-LFS on ssh and LFS over https used to work fine without any effort. Now it doesn't on the very same repos. I think the most likely explanation is that the client upgrade from 1.1.0 to 1.1.1 made something bork. True on a 8.2.3 (I know - haven't had time to upgrade) and 8.4.4 testing instance.

Switching remote to https URIs solves everything. Could the fix below, combined with the lack of a git-lfs-authenticate command, have caused this problem?

https://github.com/github/git-lfs/pull/909

It is clear in the logs that the git-lfs-authenticate command is being run:

I, [2016-02-12T16:23:32.110604 #60488] INFO -- : gitlab-shell: executing git command <git-lfs-authenticate /var/opt/gitlab/git-data/repositories/thomas-downes/lfs-test.git> for user with key key-173.

Same issue as @tpdownes:

history:
  509  git clone git@gitlab.xxxxxx.com:Testing/git-lfs.git
  510  cd git-lfs
  511  git lfs init
  512  git lfs track '*.md'
  513  git lfs track '*.txt'
  514  touch this-file-should-not-be-in-lfs.py
  515  echo "This file should be in LFS" >> README.md
  516  echo "This file should be in LFS" >> in-lfs-ideally.txt
  517  git add in-lfs-ideally.txt
  518  git add README.md
  520  git add this-file-should-not-be-in-lfs.py
  526  git add .gitattributes
  527  git commit -m NA

$ git push
Enter passphrase for key '/c/Users/jsmith/.ssh/id_rsa':
Enter passphrase for key '/c/Users/jsmith/.ssh/id_rsa':
Git LFS: (0 of 1 files) 0 B / 27 B
exit status 1
error: failed to push some refs to 'git@gitlab.xxxxxx.com:Testing/git-lfs.git'

Changing the URL to HTTPS works though:

$ grep url .git/config
#       url = git@gitlab.xxxxxx.com:Testing/git-lfs.git
        url = https://gitlab.xxxxxx.com/Testing/git-lfs.git
$ git push
Username for 'https://gitlab.xxxxxx.com': jsmith
Password for 'https://jsmith@gitlab.xxxxxx.com':
Username for 'https://gitlab.xxxxxx.com': jsmith
Password for 'https://jsmith@gitlab.xxxxxx.com':
Git LFS: (1 of 1 files) 27 B / 27 B
Counting objects: 6, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 508 bytes | 0 bytes/s, done.
Total 4 (delta 0), reused 0 (delta 0)
To https://gitlab.xxxxxx.com/Testing/git-lfs.git
   d76572d..fd7bf54  master -> master

@DouweM 8.5 is already expired. Can you reschedule and make sure this gets implemented for the scheduled milestone? A lot of customers are asking about SSH support for LFS.

To clarify, this would be to allow for authentication via SSH and not the actual pull/push operation: https://github.com/github/git-lfs/tree/master/docs/api#authentication

Thanks @marin for pointing that out.

Milestone changed to 8.6

git-lfs 1.1.1 isn't compatible with GitLabs LFS server anymore. This is probably due to this bug because the upload fails right after the attempted execution of git-lfs-authenticate. It is tracked on GitHub's side here.

@patricio Is this still planned for 8.6?

@lorenz the milestone is set to 8.6, but @DouweM could tell you with more certainty whether this will get done by 8.6 or postponed due to lack of development resources.

is there any workaround to use lfs and ssh, without entering password?

You can use some kind of credential cacher. That's about it. We would like to use LFS in our build-bots but we have/choose to make them public-over-https to do so.

@patricio This won't make 8.6 because we don't have the developer capacity on Omnibus right now. Sorry about that, I'm moving it to 8.8 for now.

Milestone changed to 8.8

@DouweM Can you maybe outline the steps necessary to implement that so that I could do it? Also what's the deadline for inclusion to 8.6?

@marin Can you help out @lorenz?

@lorenz The deadline for inclusion is technically the 21st, but chances of inclusion get smaller as that date nears.

I have workaround: you can install Git as Subversion. It's contains embedded Git LFS server with git-lfs-authenticate script.

So, you can setup in for use GitLab authentication and permissions with same LFS file storage (https://bozaro.github.io/git-as-svn/html/en_US/ch04.html).

As result, you will use GitLab LFS for HTTP users and Git as Subversion LFS for SSH users.

This is really overkill, but can be used right now.

The idea is to update gitlab-shell to return response to the LFS client which will contain the necessary data that LFS client requires to know where it can send/receive files from.

Client needs to get the correct hypermedia links, this can be seen in response.rb and router.rb.

One of the necessary headers is HTTP_X_GITLAB_LFS_TMP which gitlab-workhorse will use to know where to store temporary files on upload.
GitLab-workhorse will also need to authorise with gitlab-rails so some thoughts will need to go into how to supply workhorse with the correct auth.

@marin Thanks, I'll see what I can do.

@marin Is there already a way to generate the needed auth token for the HTTPS access?

Added ~113220 label

@DouweM another customer requested this here: https://gitlab.zendesk.com/agent/tickets/19164

Can we make sure we get this on 8.8?

/cc @JobV @marin

@patricio With @twk3 now also on the Omnibus team, I think/hope that @marin will be able to focus on this for 8.8.

I finally had a look at the code last week and while I figured out how the general implementation would look like, the authentication will need some special treatment as LFS is not working with username/password but with tokens. I don't know enough about the structure of GitLab and Ruby/Rails in general to implement that. I'd be grateful if you could implement it in 8.8.

Milestone changed to Backlog

As someone who trial-ed LFS in game development and wasn't able to use due to this bug, I saddened that this has been pushed back. I hope it can be scheduled soon.

@patricio @JobV : Any chance this could get scheduled for one of the next few milestones (8.9/8.10)? Would be extremely useful to have this.

Unfortunately, we're a little under-capacity to work on this for the upcoming release. We'll consider it for 8.10.

:(

Just wanted to add another justification/use case for this future. We use one-time-passwords, therefore it is not possible to use LFS. This feature would allow us to use LFS.

Additional customer request: https://gitlab.zendesk.com/agent/tickets/25419

Any chance this can get some love in 8.10?

This would also be helpful because I believe this would allow for newer versions of git-lfs, correct? Hard to find the older versions with brew.

@JobV I have another customer that can't use LFS because of this, too. Since this has been scheduled for 8.6 and then 8.8, can we please schedule it for 8.10 or 8.11? cc/ @DouweM

@dblessing we're at a capacity problem for this, it's not that we don't want to do this ASAP. I'll look into it.

Thanks for the ping @dblessing

mentioned in issue #19091 (closed)

So by way of #19091 (closed) it seems that it's only being looked at for 8.11 and is out for 8.10 :(

You mention a capacity issue -- is this a thing where the only person/people experienced with LFS are constantly tied up fixing show-stopping bugs 100% of their time? If so, totally understood on the capacity issue. If that's not the case, with all due respect that seems like more of a prioritization issue than a capacity one. Speaking for myself, I would have preferred this LFS issue be addressed above many of the features that have gone into 8.5, 8.6, 8.7, 8.8, and 8.9, (and probably) 8.10. When all these features are chosen before LFS, it makes LFS support feel a bit like a second-class citizen in gitlab. If there's some important API change to git-lfs in the future, I'm now left wondering how long it will take gitlab to catch up to it. It's quite possible that it's just me that feels this way, and people are way more psyched all the stuff that has gone into 8.5 - 8.10, but there may be others like me!

Even as I say all of this, I'm a huge fan of what you've done with gitlab!

There is some confusion for users at the moment, if a repo has LFS assets and they clone over SSH, then the git LFS commands all fail. Is there a way to add a notice to the Gitlab interface if a repo has LFS enabled and let users know they should use the HTTPS url instead?

Good point @bertjwregeer , I'm curious about the answer as well.

Clone via SSH should work fine, although retrieving actual LFS objects will happen over HTTPS. If you're getting an error, such as with lfs-authenticate then you may need to downgrade your client to 1.0.2. We don't yet support lfs-authenticate.

It’s possible to use git-lfs > 1.0.2 with SSH by adding a .lfsconfig file to the repository. In the file, just put:

[lfs]
    url = "https://gitlab.example.com/teamname/project.git/info/lfs"

The LFS objects will still be downloaded over HTTPS, but it’ll work even if you’re using an SSH remote. And you’re not stuck using an old version of the LFS client - this works in 1.1.x and 1.2.x.

The downside is this might not work so well for forks, since it references the URL of the main repo.

+1

This feature would be awesome.

mentioned in issue #20043 (closed)

@JobV @marin As we start planning for 8.11, please consider this. We're getting more and more requests about this.

@dblessing the only thing that stops us is the capacity of @marin's team.

(note that we started planning 8.12 already, 8.11 is already planned: https://gitlab.com/gitlab-org/gitlab-ce/issues/20043)

Added ~480950 label

Milestone changed to %8.12

@DouweM I got a nice suggestion from @marin to put someone on this else than himself or Jacob V, so that we have more people with knowledge about this. Considering this is heavily requested (and a really nice feature), do you think we could do this for 8.12 or 8.13? Any ideas about who?

This is currently blocked by https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5369

As of now, Git-LFS is the main driver for our projects to adopt GitLab and lack of SSH authentication with Git-LFS is the main impediment to using LFS with GitLab in our organization. HTTPS authentication is okay for regular developers but it does not work well for automated processes run with Jenkins or cron. On our systems we are not allowed to store username/passwords in plain text files like with:

$ touch ~/.my-git-credentials
$ chmod go-rwx ~/.my-git-credentials
$ git config --global credential.helper 'store --file ~/.my-git-credentials' 

so we can't set up any automated testing for our projects with git repos that use Git-LFS pulling from GitHub repos. That makes Git-LFS with GitLab a non-starter.

As soon as this is done and released, it will be a great day!

@JobV We've been putting this off for months, so I agree we should just put someone on it and see how far we get. I propose @patricio, and I think starting with 8.12 is a good idea. If we make it within a month, great. If it turns out to be harder than expected, we'll get it with 8.13, and we'll have another LFS expert in @patricio :)

@DouweM @patricio YES!

Added ~122770 label

Reassigned to @patricio

mentioned in merge request !6043 (merged)

mentioned in merge request gitlab-shell!86 (merged)

I am unable to use Unity Cloud Build (a CI solution) as it goes through SSH for git lfs, so the build always fails. It would be nice to be able to use this.

@kristian.d.skistad this feature is almost done (currently under review & testing). But I do urge you to keep in mind that LFS objects will not be sent via SSH. This feature has not been implemented in Git LFS. All LFS objects have to be transported via HTTP.

The merge requests I'm currently working on add support for authentication via SSH, so you don't have to enter credentials every time you need to clone or push LFS objects.

If you'd like to see support added to Git LFS to allow the transport of objects via SSH, please ask the project maintainers for it here: https://github.com/github/git-lfs/issues

If you absolutely need to push binary objects via SSH and don't want to use plain Git, please have a look at git-annex.

@patricio actually there's been a recent improvement to LFS called "transfer adapters", which can be used to provide alternative transports. However, nobody's written an SSH one as far as I can tell. See here for the details.

Still, I'll be pretty happy with SSH authentication already. My users keep complaining about having to re-type their password too often :) Even with a credential cache...

Cheers,

Clément

@patricio

If you'd like to see support added to Git LFS to allow the transport of objects via SSH, please ask the project maintainers for it here

Well, there are some news. But not the good ones. https://github.com/github/git-lfs/pull/378#issuecomment-243146714

@patricio I spoke with someone on the Unity Cloud Build team, and it looks like they just need it to authenticate through SSH to function properly, so this should work fine.

Very happy to hear this is being added! Yay!

This is done now!

Status changed to closed

Mentioned in merge request gitlab-shell!86 (merged)

Mentioned in commit gitlab-shell@6c7f7b4d

Mentioned in issue gitlab-com/support#119

Mentioned in issue #24462 (closed)

mentioned in merge request !14785