Git Annex `ssh://` Git URL doesn't work

• Zendesk ticket: https://gitlab.zendesk.com/agent/tickets/13919
• Dev link: None
• GitLab version: Unknown

Expected behavior

If a GitLab install uses a non-standard SSH port users will need to use a clone URL of format ssh://git@gitlab.example.com:2222/group/project.git. In this case git-annex does not seem to work.

Observed behavior

Normal git operations work fine with the above URL format, but when attempting to interact with annex users receive: Remote origin does not have git-annex installed; setting annex-ignore.

Next steps

I can reproduce this in my testing. @marin should this URL format work? Is there anything blocking support for this or is it just a bug? Thanks for taking a look.

git-annex and GitLab should work with with ssh:// in the url.

git-annex does read git.url.port so it should just be able to use that, this line looks like the one. It uses rsync to sync the data but that also uses ssh options received in previously mentioned piece of code.

It is possible that the things go wrong in gitlab-shell, biggest suspect is this section.

Cannot offer more than guesses. As always when it comes to git-annex and GitLab, it is time consuming to investigate deeper.

@marin Thanks for your input. Let's treat it as a bug, then. I can reproduce it on GitLab.com.

If you have time to investigate this it would be appreciated. Right now this customer is resorting to 'direct remote to the bare repositories'. I'm not sure how they've set that up.

@marin @JobV Can we schedule this, please?

@dblessing bugs go to the next milestone.

Milestone changed to 8.5

@JobV thanks

@marin Have you had an opportunity to look at this for 8.5?

I haven't worked on this issue. The issue is in the milestone and it is up for grabs for all devs.

Milestone changed to 8.6

Reassigned to @jameslopez

I manage to get this fixed locally, I'll submit something soon as I want to make sure I haven't broken anything :)

mentioned in merge request gitlab-shell!45 (merged)

Status changed to closed by commit gitlab-shell@52f73187

mentioned in commit gitlab-shell@52f73187

@jameslopez Customer in https://gitlab.zendesk.com/agent/tickets/13919 is still having the same issue with this after updating to 8.6.1 (docker). Can you take a look at the ticket please and see if you have any ideas?

thanks @dblessing - I will have a look

@dblessing Ok I installed and was testing all of this via docker, etc only to realise somehow we haven't merged the fix in. See https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/45 there was a MR for updating GITLAB_SHELL_VERSION but looking at the history it's not on any 8.6.X tag. It has been merged on master though... /cc @rymai

@dblessing @jameslopez Too bad! :(

@jameslopez Thanks for looking. @rymai Can we ship in a patch? This has been a problem for this customer for a long time and I'd hate to have to tell them to wait another month. Are there other changes in that shell version?

@dblessing Yes, this will go into 8.6.3 (unfortunately I tagged 8.6.2 just before @jameslopez pinged me here).

@rymai Thank you. I reported back to the customer.

Still does not work for me (I'm the customer) in gitlab official docker 8.6.4-ee.0. Steps to reproduce:

• use a custom ssh port (2222)
• create a project
• clone it
• git annex init
• git annex add (any file)
• git annex sync

> git annex sync 
/opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_shell.rb:141:in `exec': No such file or directory - git-annex-shell (Errno::ENOENT)
	from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_shell.rb:141:in `exec_cmd'
	from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_shell.rb:113:in `process_cmd'
	from /opt/gitlab/embedded/service/gitlab-shell/lib/gitlab_shell.rb:34:in `exec'
	from /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell:24:in `<main>'

  Remote origin does not have git-annex installed; setting annex-ignore

  This could be a problem with the git-annex installation on the remote. Please make sure that git-annex-shell is available in PATH when you ssh into the remote. Once you have fixed the git-annex installation, run: git config remote.origin.annex-ignore false
commit  ok
pull origin 
ok
push origin 
Everything up-to-date
ok

Status changed to reopened

@kforner thanks for reporting this. I'll have a look soon.

@kforner I could not replicate the problem using the Docker image for 8.6.4-ee. I've done exactly what you described above and git annex sync succeeded for me.

If it's not too much of a hassle may I suggest you send us the following info so I can compare my setup and yours? You can send us this through the support system if you prefer. It would be extremely helpful!

On the client side, in the folder where you've done the git clone:

cat .git/config |grep ssh

On the running docker container (1 command):

which git-annex && echo GITLAB_SHELL Version: && cat /opt/gitlab/embedded/service/gitlab-rails/GITLAB_SHELL_VERSION /opt/gitlab/embedded/service/gitlab-shell/VERSION && echo GitLab conf: && cat /etc/gitlab/gitlab.rb |grep 'ssh_port\|annex_enabled' &&  echo SSH PORT; cat /etc/ssh/sshd_config |grep Port

Also it would be awesome if you can replicate the issue and get this output straight away (also in the container):

tail -n 20 /var/log/gitlab/gitlab-shell/gitlab-shell.log

Ok. From what I understand git annex is not installed at all in the docker.

cat .git/config |grep ssh

url = ssh://git@gitlab.quartzbio.com:2222/karl/annex_test.git

which git-annex

not found

find /usr/ -name git-annex

not found

ll /opt/gitlab/embedded/bin/git*

-rwxrwxr-x 1 root root 2029623 Apr  4 10:40 /opt/gitlab/embedded/bin/git*
-rwxrwxr-x 1 root root     107 Apr  4 10:40 /opt/gitlab/embedded/bin/git-cvsserver*
lrwxrwxrwx 1 root root       3 Apr  4 10:40 /opt/gitlab/embedded/bin/git-receive-pack -> git*
-rwxrwxr-x 1 root root  990631 Apr  4 10:40 /opt/gitlab/embedded/bin/git-shell*
lrwxrwxrwx 1 root root       3 Apr  4 10:40 /opt/gitlab/embedded/bin/git-upload-archive -> git*
-rwxrwxr-x 1 root root 1077105 Apr  4 10:40 /opt/gitlab/embedded/bin/git-upload-pack*
-rwxrwxr-x 1 root root     494 Apr  4 10:40 /opt/gitlab/embedded/bin/gitlab-logrotate-wrapper*
-rwxrwxr-x 1 root root 8073800 Apr  4 10:40 /opt/gitlab/embedded/bin/gitlab-pages*
-rwxrwxr-x 1 root root    3234 Apr  4 10:40 /opt/gitlab/embedded/bin/gitlab-unicorn-wrapper*
-rwxr-xr-x 1 root root 9503880 Apr  4 10:40 /opt/gitlab/embedded/bin/gitlab-workhorse*
-rwxr-xr-x 1 root root 3311408 Apr  4 10:40 /opt/gitlab/embedded/bin/gitlab-zip-cat*
-rwxr-xr-x 1 root root 3396016 Apr  4 10:40 /opt/gitlab/embedded/bin/gitlab-zip-metadata*

cat /opt/gitlab/embedded/service/gitlab-rails/GITLAB_SHELL_VERSION

2.6.12

cat /opt/gitlab/embedded/service/gitlab-shell/VERSION

2.6.12

grep 'ssh_port|annex_enabled' /etc/gitlab/gitlab.rb

gitlab_shell['git_annex_enabled'] = true gitlab_rails['gitlab_shell_ssh_port'] = 2222

grep Port /etc/ssh/sshd_config Port 22

tail -n 20 /var/log/gitlab/gitlab-shell/gitlab-shell.log

I, [2016-04-05T15:47:14.309036 #12790]  INFO -- : Enable git-annex for repository: /karl/annex_test.git.
I, [2016-04-05T15:47:14.495724 #12790]  INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.18404
I, [2016-04-05T15:47:14.495996 #12790]  INFO -- : gitlab-shell: executing git-annex command <git-annex-shell configlist /var/opt/gitlab/git-data/repositories/karl/annex_test.git> for user with key key-1.
I, [2016-04-05T15:47:16.469364 #12810]  INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 1.70017
I, [2016-04-05T15:47:16.469749 #12810]  INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/karl/annex_test.git> for user with key key-1.
I, [2016-04-05T15:47:16.702169 #12816]  INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.04142
I, [2016-04-05T15:47:16.702366 #12816]  INFO -- : gitlab-shell: executing git command <git-upload-pack /var/opt/gitlab/git-data/repositories/karl/annex_test.git> for user with key key-1.
I, [2016-04-05T15:47:18.700668 #12822]  INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 1.81854
I, [2016-04-05T15:47:18.700959 #12822]  INFO -- : gitlab-shell: executing git command <git-receive-pack /var/opt/gitlab/git-data/repositories/karl/annex_test.git> for user with key key-1.
I, [2016-04-05T15:47:18.907650 #12827]  INFO -- : POST http://127.0.0.1:8080/api/v3/internal/allowed 0.04075
I, [2016-04-05T15:47:18.907941 #12827]  INFO -- : gitlab-shell: executing git command <git-receive-pack /var/opt/gitlab/git-data/repositories/karl/annex_test.git> for user with key key-1.

@kforner yeah, that's what it looks like.

I just simply did apt-get install git-annex and that installed it. Default docker images do not come with git-annex installed so you have to do this manually... (and every time you download a new docker image)

Also, you may have to add another Port 2222 to /etc/ssh/sshd_config You can have both 22 and 2222:

Port 22
Port 2222

Let me know if that fixes it!

@jameslopez it works now, thanks !

Now, should I create a new issue to have git-annex pre-installed in the docker images ? Not only the updates would be more straightforward, but you could control the version of git-annex.

@kforner Great to hear it works :) yes, please open a new one. Closing this now. /cc @dblessing

Status changed to closed

@kforner We considered adding git-annex to the package previously, but it was a huge rabbit hole. See https://gitlab.com/gitlab-org/omnibus-gitlab/issues/431 for comments.

@dblessing Could you please create an issue in omnibus-gitlab so that EE docker image installs git-annex automatically (I am mobile atm).

mentioned in issue omnibus-gitlab#1215 (closed)