Cannot fetch submodules

I have tried to implement the workaround but it's really a lot of work to set up, and of course it would have to be done separately for each project. We have dozens of projects that would have to be set up and maintained this way.

Currently we are using Jenkins instead of GitLab CI, and I like the streamlined nature of GitLab CI. But Jenkins supports submodules natively with no extra work other than checking a box, so requiring installation of SSH, Git, installing private SSH key, etc., in the CI script -- this really destroys the benefit of GitLab CI being streamlined and easy.

mentioned in issue #101 (closed)

@cdb It's not exactly a "simple change", otherwise I'm sure that @ayufan would have implemented it long ago. AFAIK the canonical issue for this shortcoming in GitLab CI is https://gitlab.com/gitlab-org/gitlab-ce/issues/18107. I think this issue is really a duplicate of that. You can see there is an open MR being discussed, https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5735.

As a workaround, it's not exactly pleasant, but we are injecting SSH keys, as well as the GitLab host key via CI "private variables", which are used to enable SSH access during the git submodule update --init --recursive in the before_script. I can provide the exact script if needed.

I'm curious to know how Jenkins handles this so smoothly, considering you still have all of the issues above to deal with.

@jonathon-reinhart Perhaps not, but it seemed like it would be after looking at the CI Runner's Go code where submodule initialization is explicitly disabled.

In Jenkins, it is pretty simple. The job configuration specifies the Git credentials, and SSH private key which authenticates the "Jenkins" user with GitLab. When Jenkins clones the top-level project from GitLab, it uses the SSH transport with the credentials assigned to the Jenkins job. The submodules are initialized and updated in the normal Git way, with the URLs from .gitmodules, and if these require authentication with GitLab, the same credentials are used and this works perfectly. No problem at all.

Now with GitLab, the CI "token" is interesting but it seems like it does add an extra layer of complexity to managing submodule access. Perhaps there would have to be some manual configuration of permissions for private projects, but I can think of some shortcuts that would easily solve some of the common cases -- for instance, allow all "internal" projects to be read as submodules with any valid CI token. A more complete access control scheme would require more thought.

I understand the issues with private submodules and security, but at least for public or internal submodules, the system should "just work".

I have been trying to implement the manual "git submodule" and SSH key setup in my GitLab CI build configuration but haven't quite succeeded yet, and am not happy about the code duplication of copy+pasting and spreading this kludge across every project...

Now with GitLab, the CI "token" is interesting but it seems like it does add an extra layer of complexity to managing submodule access. Perhaps there would have to be some manual configuration of permissions for private projects, but I can think of some shortcuts that would easily solve some of the common cases -- for instance, allow all "internal" projects to be read as submodules with any valid CI token. A more complete access control scheme would require more thought.

It seems to be simple, but there's one edge case: we have external users which doesn't have access to internal projects. This makes a hole in security if enable that.

External users cannot see internal or private projects unless access is explicitly granted. Also, external users cannot create projects or groups.

If such external user creates a CI, or do have access to a project he could exploit this knowledge and steal code from internal project.

@ayufan I hadn't thought of external users, that is an exception to the internal projects rule for sure.

We could add a restriction in GitLab where only users who have clone permissions for all submodule projects can push to the toplevel project. This way, it is safe to allow the toplevel project's CI token to be used for cloning the submodules during a CI build. It seems simple, but I suspect there are shortcomings.

For all those who're looking for an immediate solution, here's a snippet I use in every project with submodules:

.ssh: &ssh
  before_script:
  - mkdir -p ~/.ssh
  - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
  - echo -e "$SSH_DEPLOY_KEY" > ~/.ssh/id_rsa
  - chmod 600 ~/.ssh/id_rsa

I did finally get the workaround going as described here (import the ssh key in the .gitlab-ci.yml from a project variable and manually call git to init the submodule). It's a bit of clutter and an extra bit of stuff to set up for each project, but not a showstopper perhaps.

But the workaround has the same security issue that a naive implementation in GitLab itself would have, namely that someone could push to the parent project adding a new submodule they don't personally have access to and get the CI node to pull it using the SSH key assigned to the project. Even worse, there is nothing keeping them from putting a cat ~/.ssh/id_rsa command in the .gitlab-ci.yml file or somewhere else in the build scripts and obtaining the secret key we have set up in the workaround.

A proper solution would checkout the submodule repository automatically and wouldn't expose any SSH keys to the build system.

Now if that SSH key was created just for this one purpose, and given access to only the exact set of submodule projects that is needed, and read access only, then we have a secure system, assuming that all users who have access to the parent project also have access to all submodules. Perhaps that could be some kind of hard constraint in GitLab. As long as the submodules are hosted on the same server it would be reasonable -- if the submodules were hosted on another server, it would be more difficult to meet and maintain this constraint.

Excuse my ignorance, but is this solved by https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6409?

Yes, you can find draft docs for using submodules here: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6451

With gitlab 8.12, I've added two submodules, but I'm unable to get the contents of those submodules. The gitlab .gitmodules file doesn't seem to be used.

I tried using :

sudo git submodule update --init --recursive

So can please share more documentation regarding submodules in 8.12. Is there anything else I've to do to fetch those submodules ?

I am using docker executor and tried to update my submodules via ssh (https://docs.gitlab.com/ee/ci/ssh_keys/README.html).
Why is gitlab using http to pull submodules? What am I doing wrong?

build output

.gitmodules

relevant .gitlab-ci.yml snippet: (the newlines are just for readability)

@jakobwoegerbauer

Did you follow this guide for getting submodules to work: https://docs.gitlab.com/ce/user/project/new_ci_build_permissions_model.html#git-submodules?

Yes. It worked until I made changes to the submodule. After referencing to the newer version of the submodule I got this error: Then I tried the approach described in this doc: https://docs.gitlab.com/ee/ci/ssh_keys/README.html

@jakobwoegerbauer

Are you member of this project?

@jakobwoegerbauer

Was the token visible the build log that you had to mask it?

I am member (Master) of both projects and yes, the token was visible. The token stayed the same every build. It also isn't the runners token of one of these projects.
I think I'll retry the instructions of the guide you mentioned. Maybe I missed something or made a mistake.

Having the same issue as @jakobwoegerbauer. I'm using relative URLs in my .gitmodules as per the documentation, something like ../otherproject.git (the projects belong to the same group). Token was visible for me and I got "Authentication failed" as well. I also have the Master role on both projects.

I now disabled my own docker runner for this project and ran the pipeline on gitlab shared runners, which worked perfectly fine.
(with this approach: https://docs.gitlab.com/ce/user/project/new_ci_build_permissions_model.html#git-submodules)
Next i will try to re-register my runner. If this doesn't work i will just stick to the gitlab shared runners for now.

Experiencing the exact same issue as @jakobwoegerbauer, though with the added complication of occasional test passes. We're looking at sub-trees as a temporary work around

Are you aware of this: https://gitlab.com/gitlab-org/gitlab-ce/issues/22567?

Thanks @ayufan! I've added that sync to the submodules, will report back with how it works out.

Sub-trees kind of work, however conflicts massively with any cached repo data.

@zoetropelabs I believe that that this case will be solved by: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/merge_requests/341

Took me a couple of days to find this, so I'll push it everyware :)

Seems there is finally a reasonable solution:

http://docs.gitlab.com/ce/user/project/new_ci_build_permissions_model.html#sts=Git submodules %C2%B6

in short from gitlab 8.12 all you need to do is use relative paths in the .submodules, and the git submodule update --init will simply work

Thanks @erubin :) For the time being we have new docs for submodules https://docs.gitlab.com/ce/ci/git_submodules.html

That's great, I'll try it out, because the submodule clone still fails some times, and i need to go to the runner and delete the submodule folder for it to work again. It would be very handy if this was done automatically, as discussed in

https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/101#note_19143888

So, still no easy way to handle SSH submodules in same (GitLab server) and external services with same deploy key?

@shacal Do you have to use SSH submodules? If you follow the docs and use relative paths, then the submodules will be cloned (via HTTP(S)) using the CI build token, just like the super-project. No deploy keys needed.

Still not working correctly for me, every time the submodule is updated the whole directory of the parent project on the build server needs to be deleted (or at least some key part, which is not the submodule itself) for the builder to corretly fetch the updated submodule. other wise the submodule update command fails

If we have to use relative paths for ci, how does it work out our local machine?

@ayufan Should this be closed now that !443 (merged) is merged?

@tom.davidson Are you talking about running the gitlab-runner locally? Or are you asking how to handle using relative paths for normal development? Note that the relative paths tell git where to find the submodules when cloning, not where to store them. I suggest reading https://git-scm.com/docs/git-submodule.

Btw. it is possible to use insteadOf to support all submodules by default, and easily, whether they are from ssh or https as long as they live on the same GitLab instance.

@JonathonReinhart I was talking about git. If I have relative paths for a git submodule will it mess with my local git? What about git submodules that are not on the same gitlab server? relative paths does not seem like a viable solution.

My submodule is on the same gitlab server and relative paths or not and with GIT_SUBMODULE_STRATEGY set yet jobs still failed. It is just a POC so I removed the need for the submodule but will need to put it back in sometime.

@ayufan What is insteadOf? I'm not sure what you mean.

@tom.davidson Relative paths in .gitmodules only applies to server paths. You are thus correct, trying to use relative paths when the submodules are not on the same server doesn't make any sense. You don't have to use relative paths, but if you do, then the authentication will work seamlessly for pulling private submodules (from the same GitLab instance). If your submodules are public, then you don't need to worry about authentication, and don't have to use relative paths. GIT_SUBMODULE_STRATEGY should just work if your submodules are configured correctly.

@JonathonReinhart

We can automatically replace paths like https://gitlab.com or ssh://git@gitlab.com to use ones with bundled credentials and https and making submodules to work automatically whatever protocol you use as long as everything is stored on the same server.

The official documentation states

If you are using GitLab 8.12+ and your submodule is on the same GitLab server, you must update your .gitmodules file to use relative URLs.

so I have been using relative paths like ../../group/project.git in my .gitmodules since Version 8.12 but I'd prefer to use absolute paths if that is also possible.

We can automatically replace paths...

Does it mean "We already do" or "We could/should"

@ayufan do i understand you correctly that those relative paths are no longer necessary? (If so the documentation could be updated accordingly.) Does the automatic replacement also apply to GitLab CE instances (where the domain is not gitlab.com) using the settings external_url (and gitlab_rails['gitlab_ssh_host'] if set) in /etc/gitlab/gitlab.rb for search/replace?

They are necessary, but there's a possibility for as to avoid you to have to change yours .gitmodules with simple git function.

My project has the following structure:

project/
  apps/
    submodule_1/
      folder/
        submodule_1_1/
          ..
      ..
    ...
    submodule_n/
      ..

project is located at: https://gitlab.example.com/group/subgroup/project.git

submodule_1 is located at: https://git.example.com/group/subgroup/apps/submodule_1.git

submodule_1_1 is a submodule of submodule_1 and located at https://git.example.com/othergroup/submodule_1_1.git

project/.gitmodules contains the following entry for submodule_1:

[submodule "apps/submodule_1"]
  path = apps/submodule_1
  url = ../../../group/subgroup/apps/submodule_1.git
  branch = master

This works as expected. Now I have to configure project/apps/submodule_1/folder/submodule_1_1. I tried setting:

[submodule "apps/submodule_1"]
  path = apps/submodule_1
  url = ../../../othergroup/submodule_1_1.git
  branch = master

But this doesn't work. What does work is setting:

[submodule "apps/submodule_1"]
  path = apps/submodule_1
  url = ../../../../../othergroup/submodule_1_1.git
  branch = master

but I don't understand why this works. If I checkout submodule_1 stand-alone and try to clone the submodules, this fails. It only works from within gitlab-ci.

For this reason, gitlab-ci is currently unusable at this point, because I can't break the gitmodules config for stand-alone builds.

I wonder if I'm doing something wrong or if this is a bug in gitlab-ci.

Cannot fetch submodules

Why is recursive clone disabled?

Workaround?

Possible fix

New variable GIT_SUBMODULES

Access protection

Designs

Child items 0

Activity

Admin message

Admin message

Cannot fetch submodules

Why is recursive clone disabled?

Workaround?

Possible fix

New variable GIT_SUBMODULES

Access protection

Activity