Inject self signed certificates into docker executor

Cloning into '/builds/***/***'...
fatal: unable to access 'https://gitlab-ci-token:xxxxxx@***/***/***/': server certificate verification failed.
CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

Build failed with exit code 128

We have GitLab and GitLab CI running on the same server with self signed certificates and the GitLab CI multi-runner on another server that talks to both without any issues.

What does not work however is cloning the repositories inside of a Docker executor since the Docker image does not contain our self-signed certificates. (see output above)

Is there a way to inject our self-signed certificates automatically into the container that is created by the runner? I have found the tls_cert_path option, but that seems to be unrelated to our current issue.

Possible solution:

1. new injected_certificates option
2. create temporary, shared volume on container creation
3. move configured certificates into /usr/share/ca-certificates/ (container) via that volume
4. add certificates to /etc/ca-certificates.conf (i.e. echo "gitlab-ci.crt" >> /etc/ca-certificates.conf)
5. run update-ca-certificates

This solution might only work for Debian/Ubuntu-based images though. It might make sense to use a more generic after_creation option instead, that can be used to run those commands mentioned above.

while it does not work for using curl, git clone seems to actually work by using the following line in config.toml:

volumes = ["/etc/ssl/certs:/etc/ssl/certs:ro"]

It should do the trick: inject ssl certificates into container, but I still don't have idea how to resolve it for all :(

I am having a slightly different issue where my tests clone a bunch of repos for fixture data but the git repos are private so it requires a deploy key in the container. While I could create the docker image with my deploy key I would prefer to run a script before job execution that would inject whatever I need. I could keep this in the build script but I don't want to expose the keys in every git repo and want it specified in the runner config.toml.

It should do the trick: inject ssl certificates into container, but I still don't have idea how to resolve it for all :(

@cosman2001 for the deploy key, the simplest is to store private key in secure variables and save the private key to id_rsa:

before_script:
- echo $SSH_PRIVATE_KEY > ~/.ssh/id_rsa
- git submodule update --init

As for ssl certificates, this is hard part. Maybe adding the:

before_clone:
- do-something-before-git-clone

Would do the work?

@sytses What do you think about adding before_clone: step to yaml?

@ayufan I'm OK with that.

I'm having a similar issue and haven't found a solution. We have an internal root certificate authority installed on our servers, but Docker does not inherit the system certificates which causes it throw a fatal error:

fatal: unable to access 'https://gitlab-ci-token:...@gitlab.../...git/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

Using shell executor works, but it makes it difficult to manage installing dependencies via apt-get and npm install -g that are easier to deal with within Docker.

@sytses has there been any progress on this problem?

Milestone changed to v0.7.0

@tobias.bieniek Sorry, not yet.

The course of action will be to create separate container which will do git clone and you will have options to configure that.

@ayufan thanks for the quick reply. do you know any kind of workaround that I could use for now?

Add host based bind volume pointing to /etc/ssl/ and putting the cert there.

unfortunately now java is giving me problems: InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

I'm using java:openjdk-8u66 and overwriting /etc/ssl apparently killed the java certificates :-/

You need to fill the /etc/ssl with data that was previously in container.

mentioned in issue #191 (closed)

@tobias.bieniek

You can also use:

1. GIT_SSL_NO_VERIFY
2. GIT_SSL_CAINFO
3. GIT_SSL_CAPATH

This variables can be specified in config.toml and as well in variables: section of .gitlab-ci.yml.

The most secure would be to use host-based volume and specify path to GIT_SSL_CAINFO or GIT_SSL_CAPATH.

adding environment = ["GIT_SSL_NO_VERIFY=1"] to the runner instead of the host volume works well too. since we are running the servers in the same controlled network this should not be a big security issue for us.

Hi,

I followed https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/install/docker.md#installing-trusted-ssl-server-certificates but this does not seem to be working to trust SSL server certificate (using Gitlab installed with Omnibus). Is it expected behavior or a bug? I understood that this should solve this issue.

The GIT_SSL_NO_VERIFY solution is working, but is not very clean and secure =(

And at the moment, embedding a custom certificate in the docker image is prevented by https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/225.

Hi all,

I believe that I hit a road block on this issue as well. Has there been a fix other than the various workarounds proposed?

My opinion on this issue:

Ideally, the solution would modify config.toml, but not a modify .gitlab-ci.yml. I believe that cloning the repository, including passing the certificate information to docker, should be the responsibility of the runner and should be detached from the .gitlab-ci.yml file. I always assumed that the .gitlab-ci.yml file should not be dependent upon the configuration of the runner.

Potential issues with using before_clone to pass the certificate:

1. Anyone that clones a repository from a private gitlab server with a private gitlab ci runner would need to modify the .gitlab-ci.yml file to work with their runner. Since this is committed to the repository, I think requiring modification is a bad idea.
2. If there are multiple runners that each require their own unique configuration, a single before_clone section would not be able to elegantly handle this. What happens if one runner uses docker but another does not?

For this reason, I don't like the solution of using a before_clone step or the variables: section in .gitlab-ci.yml.

Instead, shouldn't the configuration of the docker container be delegated to the runner?

@ayufan Would it be possible to configure this completely within config.toml? Specifically the runners.docker section seems appropriate. Some type of before_clone section there would make more sense to me.

This issue seems related to #129 (closed). I think the idea 2 proposed here may be better if it means no modification necessary to .gitlab-ci.yml.

@voidstarstar

My idea is to not require any config.toml configuration at all. Since we connect to server already we do have all TLS certificates. We can read these certificates and inject them into build environment. Then use env variables to configure git clone and artifacts uploader.

We can do it this way, because we trust that runner can do secure connection and runner did verify the certificate already.

@ayufan Oh, I see. So that would make this completely automated without any extra configuration in either config.toml or .gitlab-ci.yml.

I had completely forgotten that the git repo could just use the same TLS certificate that the runner uses to connect, such as the one provided by tls-ca-file.

If this is the case, please disregard my entire previous post. What you described sounds perfect. I look forward to this change! Thanks for clearing that up for me and for all your hard work!

The Bleeding Edge with TLS certificate inject is compiling. You may want to try: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/builds/395827

@ayufan I just tried it out and I can confirm that it's working! Thank you so much! You're awesome!!

@voidstarstar I need to think when to merge it now :)

mentioned in issue #310 (closed)

The Bleeding Edge with TLS certificate inject is compiling. You may want to try: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/builds/395827

@ayufan I'm struggling to find the code or branch that has certificate injection. Is it in master? Do you know, in which version of the runner this will be released?

The patch was 1ef82ea7 which is in master. You can grab the latest package from the builds section.

e.g. #526093 look for Download URLs beginning with https://s3.amazonaws.com/gitlab-ci-multi-runner-downloads/master/binaries/

This is fixed in https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/merge_requests/68

Status changed to closed

I also have problems with this. Is this in all runners for all executors? I am basically using shell executor and the clone fails also with

Cloning into '/builds/***/***'...
fatal: unable to access 'https://gitlab-ci-token:xxxxxx@***/***/***/': server certificate verification failed.
CAfile: /etc/ssl/certs/ca-certificates.crt

When I add my intermediate(?) Comodo Cert to /etc/ssl/certs/ca-certificates.crt the clone works, but how can I make this more generic?

I am not using a self-signed cert, so I believe there has to be a a way to import these certs somehow.

@t-munk Are your intermediates served with your nginx? You can check that with something like ssllabs.com.

I think I am struck with a similar bug, like https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/1261855

This is really weird! My only solution was to add this cert from Comodo: https://github.com/schmunk42/docker-gitlab-runner/blob/ac15f79cf7e573733f2af881c66b945b26f1b624/Dockerfile#L13

I tried and checked like everything, only git was complaining.

Then I tried ubuntu:trusty, ubuntu:wily and debian:8 - just plain git and access to our domain.

Only trusty fails, maybe you could update the base image :)

It looks like your gitlab nginx is not configured to send the intermediate ca cert to the connecting client. see https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md#change-the-default-port-and-the-ssl-certificate-locations (and http://nginx.org/en/docs/http/configuring_https_servers.html#chains for general info about ssl on nginx) ... assuming you are using the bundled omnibus nginx, but it should be similar for non-nginx/non-omnibus installations

The file you are manually adding is an intermediate CA ("COMODO RSA Domain Validation Secure Server CA"). If the ROOT CA ("COMODO RSA Certification Authority" aka "COMODO SECURE™") which signed the intermediate CA is installed you should not need to manually add the intermediate as trusted but instead tell the https server to send the intermediate cert to connecting clients.

@p-schneider

Here's a shortened output of openssl s_client -connect git.mydomain.com:443 -showcerts -CApath /etc/ssl/certs - https://gist.github.com/schmunk42/2dbb9163321fba586ef0

Looks OK to me, or not?

@t-munk that looks good. However my computer has "COMODO RSA Certification Authority" installed as self signed root, but your chain depends on another ROOT CA ("AddTrust External CA Root"). Also your chain has an interesting order, maybe changing the order fixes issues but I'm not sure. (see http://stackoverflow.com/questions/20409534/how-does-an-ssl-certificate-chain-bundle-work )

@p-schneider Thank you for the links. I'll try this again if I find the time. I also found issues about certificate chain ordering but had no luck in solving it this way.

What made me wonder is the fact that only git on trusty complained about the cert. I also tried to tell git the cert path and so on.

Mentioned in issue #1894 (closed)