SSL certificate problem: unable to get issuer certificate

@ayufan What are your thoughts on this?

is it because in the code we have the build container being created by the line

// const PreBuildImage = "gitlab/gitlab-runner:build"
preContainer, err := s.createContainer("pre", PreBuildImage, nil, *options)

with options coming from the function:

func (s *DockerExecutor) prepareBuildContainer() (options *docker.CreateContainerOptions, err error) {
	options = &docker.CreateContainerOptions{
		Config: &docker.Config{
			Tty:          false,
			AttachStdin:  true,
			AttachStdout: true,
			AttachStderr: true,
			OpenStdin:    true,
			StdinOnce:    true,
			Env:          s.BuildScript.Environment,
		},
		HostConfig: &docker.HostConfig{
			Privileged:    s.Config.Docker.Privileged,
			RestartPolicy: docker.NeverRestart(),
			ExtraHosts:    s.Config.Docker.ExtraHosts,
			Links:         s.Config.Docker.Links,
		},
	}

on which we can see there's no mounting of the certificates files, so all the custom certificates are not being transfered to this docker container running the git clone ?

@allan-simon So it sounds like this will need to be updated? I did try potential mounting the certificates through a volume, but I don't think that was working for me either.

@TRPmwiesen I'm not familiar at all with this code, it just happens I've already played a bit with docker and go (but not both together though) so it would make sense to me if the problem was coming from there, but I would prefer to wait for the point of view of a more seasoned developer.

what i'm going to try is to modify the code to have one more option, the one to mount the exact same /etc/ssl/certs/ca-certificates.crt as at this step it already contains what you want.

I think i know why the tls chain contains only one entry

the git clone command is written by this line https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/shells/bash.go#L92

	b.executeTlsCommand(w, "GIT_SSL_CAINFO", "git", "clone", build.RepoURL, projectDir)

and executeTlsCommand actually replace the content of GIT_SSL_CAINFO with the value of CI_SERVER_CA_CHAIN

https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/shells/bash.go#L44

list = append(list, caBundle+"=<(echo \"$CI_SERVER_CA_CHAIN\")")

so Git only see this ca chain :(

adding these lines at executors/docker/executor_docker.go line 500 seems to not work (certainly for the reason explained above)

+            Mounts: []docker.Mount{
+                docker.Mount {
+                   Source: "/etc/ssl/certs/ca-certificates.crt",
+                   Destination: "/etc/ssl/certs/ca-certificates.crt",
+                   Mode: "ro,Z",
+                   RW: false,
+                },
+            },

right now my hack is to add this line in shells/bash.go line 149

     b.executeCommand(w, "export", "GIT_SSL_NO_VERIFY=true")

This hack you can add to .gitlab-ci.yml:

variables:
  GIT_SSL_NO_VERIFY: "true"

It will be solved in 1.0.

You can test builds from this MR: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/merge_requests/68 once they get build.

mentioned in merge request !68 (merged)

indeed simply adding

variables:
  GIT_SSL_NO_VERIFY: "true"

seems to work, thanks

@allan-simon @ayufan I'm successfully able to bypass the error by using the work around for setting the GIT_SSL_NO_VERIFY: true.

I'll look for the updated release/build for 1.0 that will solve the issue. Let me know if I can assist with any testing.

Thanks

also I would like to add that for me the problem was solved in a clean way not on gitlab-ci-runner but by adding the whole certificate chains in the .crt of the nginx of the gitlab server

(i.e as some of the intermediate certificate were missing, it was causing the ssl connection to fail)

Yes, adding intermediate is required. It's good to check your ssl server for missing chain on: ssllabs.com

I'm unable to test with ssllabs.com since our GitLab instance is hosted internally behind our corporate firewall.

I did however add the full cert chain to nginx certificate /etc/gitlab/ssl/<server>.crt, which does appear to "solve" the git clone issue when I remove the GIT_SSL_NO_VERIFY: "true" and push up change to have CI rebuild, which completes successfully.

I think what your saying is that this is another way to work around the problem and at least re-enable the Git SSL Verification, which is great.

Is a fix still planned to address the underlying issue which seems to be that if I'm specifying the CA that gitlab-runner will use via tls-ca-file from /etc/gitlab-runner/config.toml, that it should import all certificates as I have already listed both Root and Intermediate as trusted, so then I won't be required to add the intermediate and/or root certificates to the actual webserver, since to some degree a CA file with a certificate or certificate chain must be imported to gitlab-runner from somewhere to do the validation as it can't rely on the server to provide this.

@TRPmwiesen If you run: gitlab-runner verify is everything OK?

Generally I try to get the certificate chain from runner connection and inject that later when executing git clone. So as long as gitlab-runner verify passes everything should be working OK.

Here are the results:

(server)# gitlab-runner --debug verify
INFO[0000] Running in system-mode.
DEBU[0000] Trying to load /etc/ssl/certs/ca-bundle.crt ...
INFO[0000] Veryfing runner... is alive                   runner=6822dcd4

@TRPmwiesen Could you try latest master and see if it works?

I've upgraded to GitLab EE 8.4 and upgraded GitLab Multi runner to latest RPM install.

I restored the nginx cert to just list the server (no root/intermediate certs), but am still getting the git clone error.

gitlab-ci-multi-runner 1.0.0~beta.151.g056a7a1 (056a7a1)
Using Docker executor with image node:latest ...
Pulling docker image node:latest ...

Running on runner-6822dcd4-project-11-concurrent-1 via <server>...
Cloning repository...
Cloning into '/builds/<username>/test'...
fatal: unable to access 'https://gitlab-ci-token:xxxxxx@gitlab.domain.com/<userame>/test.git/': SSL certificate problem: unable to get issuer certificate

ERROR: Build failed with: exit code 1

Reverting back to the nginx cert that lists root/intermediate certs, it succeeds:

gitlab-ci-multi-runner 1.0.0~beta.151.g056a7a1 (056a7a1)
Using Docker executor with image node:latest ...
Pulling docker image node:latest ...

Running on runner-6822dcd4-project-11-concurrent-1 via <server>...
Cloning repository...
Cloning into '/builds/<username>/test'...
Checking out a7d92931 as master...
Note: checking out 'a7d92931ffca489538ae97b56da4428baaa56bb6'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at a7d9293... Changing test bash command

$ which node
/usr/local/bin/node
$ node -v
v5.5.0
$ npm -v
3.3.12


Build succeeded.

So your nginx exposes only server certificate? It doesn't have intermediates to build the complete chain?

Yes, I submit my CSR to our Internal CA group and I'm not exactly sure which Internal CA cert will be used to sign our certificates and since the servers and browsers already have the Root CA cert and all Intermediate CA certs included as trusted certificates, I really only need to provide the actual server certificate which is issued.

For additional context:

Internal CA certs (Root and Intermediates) are placed in /etc/pki/ca-trust/source/anchors/, when new default certs are upgraded, the use of update-ca-trust can extract/rebuild the default certificate chain to include our own Internal CA as a trusted default on the server, which is then already linked to by the /etc/ssl/certs/ca-bundle.crt symlink:

(<server>)# ls -la /etc/ssl/certs/ca-bundle.crt
lrwxrwxrwx 1 root root 49 Jan  8 10:57 /etc/ssl/certs/ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

This extracted PEM certificate list is what is built by update-ca-trust which now includes our internal CA certificates.

Since these are already included here, when you test curl to any internal domain that has a SSL certificate signed by our Internal CA the verification is automatic since the server has all Internal CA certificates.

Additional info here:

We have the same setup as @TRPmwiesen and would really appreciate a fixed for this.

Can you check if the golang reads this certificate store?

@ayufan I don't understand your question? Right now the runner only passes a single certificate via "tlscachain" from the bundle provided with the "tls-ca-file" option. Our certificate setup needs both required certs to passed for host verification to succeed.

@iiSeymour It passes the certificates that are required to reconstruct the chain to make the verification pass. The rest of certificiates is sent by your server (nginx)?

This is not what we are seeing in practice, I have seen the same behaviour as @TRPmwiesen originally reported, switching the runner to debug mode shows only a single certificate gets passed and not both. Our certificate authority chain contains 2 certificates, both of these are required to do host verification on our gitlab instance certificate.

@iiSeymour I'll look at it before the 1.1. Thanks for reports.

Milestone changed to v1.1

Added ~58730 label

I do see that an official 1.0.4 is released. I'll update off the beta and see if I can test again, but it sounds like @iiSeymour is seeing the same issue.

I think once that all certificates from the tls-ca-chain are passed into the runner this issue will be resolved.

@dblessing sorted me out on this one. Adding the intermediate after the server certificate allows the chain to be verified by git inside the build container.

Status changed to closed

To follow up again, the issue still exists.

While there is a work around to also include the intermediate CA certificate into the webserver certificate.

I don't really understand why this is necessary. If don't include the intermediate CA certificate with my webserver certificate, but it is provided via the tls-ca-chain where the root certificate and intermediate CA certificate already exist. It should still be possible to validate the certificate.

@TRPmwiesen Yes, it should work as long as you can reconstruct the chain. Maybe for some reason we are able to read only one certificate from tls-ca-chain?

Here is another capture of the debug output under gitlab-runner 1.0.4.

As shown below it appears that only one certificate is being passed into the runner, although both root and intermediate are located in /etc/ssl/certs/ca-bundle.crt

(<server>)# gitlab-runner --version
gitlab-runner version 1.0.4 (014aa8c)
(<server>)# gitlab-runner --debug run
Starting multi-runner from /etc/gitlab-runner/config.toml ...  builds=0
Running in system-mode.

Feeding runners to channel                          builds=0
Starting worker 0                                   builds=0
Checking runner centos-6-docker url=https://gitlab.domain.com/ci token=<token> executor=docker on 0  builds=0
Trying to load /etc/ssl/certs/ca-bundle.crt ...
Starting worker 1                                   builds=0
Checking for builds... nothing                      runner=6822dcd4
Feeding runners to channel                          builds=0
Checking runner centos-6-docker url=https://gitlab.domain.com/ci token=<token> executor=docker on 1  builds=0
Checking for builds... nothing                      runner=6822dcd4
Feeding runners to channel                          builds=0
Checking runner centos-6-docker url=https://gitlab.domain.com/ci token=<token> executor=docker on 0  builds=0
Checking for builds... nothing                      runner=6822dcd4
Feeding runners to channel                          builds=0
Checking runner centos-6-docker url=https://gitlab.domain.com/ci token=<token> executor=docker on 1  builds=0
Checking for builds... nothing                      runner=6822dcd4
Feeding runners to channel                          builds=0
Checking runner centos-6-docker url=https://gitlab.domain.com/ci token=<token> executor=docker on 0  builds=0
Checking for builds... nothing                      runner=6822dcd4
Feeding runners to channel                          builds=0
Checking runner centos-6-docker url=https://gitlab.domain.com/ci token=<token> executor=docker on 1  builds=0
Checking for builds... nothing                      runner=6822dcd4
Feeding runners to channel                          builds=0
Checking runner centos-6-docker url=https://gitlab.domain.com/ci token=<token> executor=docker on 0  builds=0
Checking for builds... received                     runner=6822dcd4
Received new build for 6822dcd4 build 81            builds=0
Added a new build id: 81
projectid: 13
commands: |2-

  npm -v
  node -v
  npm install
repourl: https://gitlab-ci-token:<token>@gitlab.domain.com/<user>/test.git
sha: 4913c101c608fa274caeed5dacca497f8787dd74
refname: master
beforesha: "0000000000000000000000000000000000000000"
allowgitfetch: true
timeout: 3600
variables:
- key: CI_BUILD_NAME
  value: test:node
  public: true
  internal: false
  file: false
- key: CI_BUILD_STAGE
  value: test
  public: true
  internal: false
  file: false
options:
  image: node:latest
token: <token>
name: test:node
stage: test
tag: false
tlscachain: |
  -----BEGIN CERTIFICATE-----
  MIIJwDCCB6igAwIBAgITXwAAABnOZvBC8H6Q0wAAAAAAGTANBgkqhkiG9w0BAQUF
  ADB0MRMwEQYKCZImiZPyLGQBGRYDbmV0MRowGAYKCZImiZPyLGQBGRYKdHJvd2Vw
  cmljZTEUMBIGCgmSJomT8ixkARkWBGNvcnAxKzApBgNVBAMTIlQuIFJvd2UgUHJp
  ################################################################
  #                          Cipher Text                         #
  ################################################################
  B3l85KuDaq2G+jkR8T8jt3fVkAdv6dKGnrz8eOgeElgZsAvQQEElwd40BvIO9yZz
  gM7Ell+Y65ez+HvV9PEu7JMh2/8ZQd8mwOmH/oB1bKXcwUnAnO3a4j5FD6BiXX3h
  VDYMwg==
  -----END CERTIFICATE-----
network: {}
buildstate: ""
buildstarted: {}
buildfinished: {}
buildduration: "0"
runner:
  name: centos-6-docker
  limit: 0
  disableverbose: null
  outputlimit: null
  runnercredentials:
    url: https://gitlab.domain.com/ci
    token: <token>
    tlscafile: /etc/ssl/certs/ca-bundle.crt
  runnersettings:
    executor: docker
    buildsdir: null
    cachedir: null
    environment:
    - MYSQL_ALLOW_EMPTY_PASSWORD=1
    shell: bash
    ssh: null
    docker:
      dockercredentials:
        host: null
        certpath: null
        tlsverify: null
      hostname: null
      image: centos6
      privileged: false
      disablecache: null
      volumes:
      - /cache
      cachedir: null
      extrahosts: []
      links: []
      services: []
      waitforservicestimeout: null
      allowedimages: []
      allowedservices: []
      imagettl: null
    parallels: null
    virtualbox: null
globalid: 0
runnerid: 0
projectrunnerid: 0
  builds=1
gitlab-ci-multi-runner 1.0.4 (014aa8c)              build=81 runner=6822dcd4
Shell script: environment: []
prescript: |
  #!/usr/bin/env bash

  set -eo pipefail
  set +o noclobber
  : | eval $'echo "Running on $(hostname) via <server>..."\nexport MYSQL_ALLOW_EMPTY_PASSWORD=1\nexport CI=$\'true\'\nexport CI_BUILD_REF=$\'4913c101c608fa274caeed5dacca497f8787dd74\'\nexport CI_BUILD_BEFORE_SHA=0000000000000000000000000000000000000000\nexport CI_BUILD_REF_NAME=$\'master\'\nexport CI_BUILD_ID=81\nexport CI_BUILD_REPO=$\'https://gitlab-ci-token:<token>@gitlab.domain.com/<user>/test.git\'\nexport CI_PROJECT_ID=13\nexport CI_PROJECT_DIR=$\'/builds/<user>/test\'\nexport CI_SERVER=$\'yes\'\nexport CI_SERVER_NAME=$\'GitLab CI\'\nexport CI_SERVER_VERSION=\'\'\nexport CI_SERVER_REVISION=\'\'\nexport GITLAB_CI=$\'true\'\nexport CI_BUILD_NAME=$\'test:node\'\nexport CI_BUILD_STAGE=$\'test\'\nmkdir -p "/builds/<user>/test.tmp"\necho -n $\'-----BEGIN CERTIFICATE-----\\nMIIJwDCCB6igAwIBAgITXwAAABnOZvBC8H6Q0wAAAAAAGTANBgkqhkiG9w0BAQUF\\nADB0MRMwEQYKCZImiZPyLGQBGRYDbmV0MRowGAYKCZImiZPyLGQBGRYKdHJvd2Vw\\ncmljZTEUMBIGCgmSJomT8ixkARkWBGNvcnAxKzApBgNVBAMTIlQuIFJvd2UgUHJp\\n################################################################\\n#                          Cipher Text                         #\\n################################################################\\nB3l85KuDaq2G+jkR8T8jt3fVkAdv6dKGnrz8eOgeElgZsAvQQEElwd40BvIO9yZz\\ngM7Ell+Y65ez+HvV9PEu7JMh2/8ZQd8mwOmH/oB1bKXcwUnAnO3a4j5FD6BiXX3h\\nVDYMwg==\\n-----END CERTIFICATE-----\\n\' > "/builds/<user>/test.tmp/GIT_SSL_CAINFO"\nexport GIT_SSL_CAINFO="/builds/<user>/test.tmp/GIT_SSL_CAINFO"\nif [[ -d "/builds/<user>/test/.git" ]]; then\n  echo $\'\\x1b[32;1mFetching changes...\\x1b[0;m\'\n  $\'cd\' "/builds/<user>/test"\n  $\'git\' "clean" "-ffdx"\n  $\'git\' "reset" "--hard"\n  $\'git\' "remote" "set-url" "origin" "https://gitlab-ci-token:<token>@gitlab.domain.com/<user>/test.git"\n  $\'git\' "fetch" "origin"\nelse\n  echo $\'\\x1b[32;1mCloning repository...\\x1b[0;m\'\n  $\'rm\' "-r" "-f" "/builds/<user>/test"\n  $\'git\' "clone" "https://gitlab-ci-token:<token>@gitlab.domain.com/<user>/test.git" "/builds/<user>/test"\n  $\'cd\' "/builds/<user>/test"\nfi\necho $\'\\x1b[32;1mChecking out 4913c101 as master...\\x1b[0;m\'\n$\'rm\' "-f" ".git/index.lock"\n$\'git\' "checkout" "4913c101c608fa274caeed5dacca497f8787dd74"\nif [[ -e "../../../cache/<user>/test/test:node/master/cache.zip" ]]; then\n  echo $\'\\x1b[32;1mRestoring cache...\\x1b[0;m\'\n  $\'/usr/bin/gitlab-runner-helper\' "extract" "--file" "../../../cache/<user>/test/test:node/master/cache.zip"\nelse\n  if [[ -e "../../../cache/<user>/test/test:node/master/cache.zip" ]]; then\n    echo $\'\\x1b[32;1mRestoring cache...\\x1b[0;m\'\n    $\'/usr/bin/gitlab-runner-helper\' "extract" "--file" "../../../cache/<user>/test/test:node/master/cache.zip"\n  fi\nfi\n'
buildscript: |
  #!/usr/bin/env bash

  set -eo pipefail
  set +o noclobber
  : | eval $'export MYSQL_ALLOW_EMPTY_PASSWORD=1\nexport CI=$\'true\'\nexport CI_BUILD_REF=$\'4913c101c608fa274caeed5dacca497f8787dd74\'\nexport CI_BUILD_BEFORE_SHA=0000000000000000000000000000000000000000\nexport CI_BUILD_REF_NAME=$\'master\'\nexport CI_BUILD_ID=81\nexport CI_BUILD_REPO=$\'https://gitlab-ci-token:<token>@gitlab.domain.com/<user>/test.git\'\nexport CI_PROJECT_ID=13\nexport CI_PROJECT_DIR=$\'/builds/<user>/test\'\nexport CI_SERVER=$\'yes\'\nexport CI_SERVER_NAME=$\'GitLab CI\'\nexport CI_SERVER_VERSION=\'\'\nexport CI_SERVER_REVISION=\'\'\nexport GITLAB_CI=$\'true\'\nexport CI_BUILD_NAME=$\'test:node\'\nexport CI_BUILD_STAGE=$\'test\'\n$\'cd\' "/builds/<user>/test"\necho $\'\\x1b[32;1m$ npm -v\\x1b[0;m\'\nnpm -v\necho $\'\\x1b[32;1m$ node -v\\x1b[0;m\'\nnode -v\necho $\'\\x1b[32;1m$ npm install\\x1b[0;m\'\nnpm install\n'
postscript: |
  #!/usr/bin/env bash

  set -eo pipefail
  set +o noclobber
  : | eval $'export MYSQL_ALLOW_EMPTY_PASSWORD=1\nexport CI=$\'true\'\nexport CI_BUILD_REF=$\'4913c101c608fa274caeed5dacca497f8787dd74\'\nexport CI_BUILD_BEFORE_SHA=0000000000000000000000000000000000000000\nexport CI_BUILD_REF_NAME=$\'master\'\nexport CI_BUILD_ID=81\nexport CI_BUILD_REPO=$\'https://gitlab-ci-token:<token>@gitlab.domain.com/<user>/test.git\'\nexport CI_PROJECT_ID=13\nexport CI_PROJECT_DIR=$\'/builds/<user>/test\'\nexport CI_SERVER=$\'yes\'\nexport CI_SERVER_NAME=$\'GitLab CI\'\nexport CI_SERVER_VERSION=\'\'\nexport CI_SERVER_REVISION=\'\'\nexport GITLAB_CI=$\'true\'\nexport CI_BUILD_NAME=$\'test:node\'\nexport CI_BUILD_STAGE=$\'test\'\n$\'cd\' "/builds/<user>/test"\nif [[ -e "artifacts.zip" ]]; then\n  echo $\'\\x1b[32;1mUploading artifacts...\\x1b[0;m\'\n  mkdir -p "/builds/<user>/test.tmp"\n  echo -n $\'-----BEGIN CERTIFICATE-----\\nMIIJwDCCB6igAwIBAgITXwAAABnOZvBC8H6Q0wAAAAAAGTANBgkqhkiG9w0BAQUF\\nADB0MRMwEQYKCZImiZPyLGQBGRYDbmV0MRowGAYKCZImiZPyLGQBGRYKdHJvd2Vw\\ncmljZTEUMBIGCgmSJomT8ixkARkWBGNvcnAxKzApBgNVBAMTIlQuIFJvd2UgUHJp\\n################################################################\\n#                          Cipher Text                         #\\n################################################################\\nB3l85KuDaq2G+jkR8T8jt3fVkAdv6dKGnrz8eOgeElgZsAvQQEElwd40BvIO9yZz\\ngM7Ell+Y65ez+HvV9PEu7JMh2/8ZQd8mwOmH/oB1bKXcwUnAnO3a4j5FD6BiXX3h\\nVDYMwg==\\n-----END CERTIFICATE-----\\n\' > "/builds/<user>/test.tmp/CI_SERVER_TLS_CA_FILE"\n  export CI_SERVER_TLS_CA_FILE="/builds/<user>/test.tmp/CI_SERVER_TLS_CA_FILE"\n  $\'/usr/bin/gitlab-runner-helper\' "artifacts" "--url" "https://gitlab.domain.com/ci" "--token" "<token>" "--id" "81" "--file" "artifacts.zip"\n  $\'rm\' "-f" "aritfacts.zip"\nfi\n'
command: bash
arguments: []
passfile: false
extension: ""
  build=81 runner=6822dcd4
Using Docker executor with image node:latest ...    build=81 runner=6822dcd4
Applying docker.Client transport fix: &{false 0x1eb72c0 <nil> 0xc208186700 unix:///var/run/docker.sock 0xc2082b6230 0xc2082b62a0 [1 18] [] [] <nil>}  host=unix:///var/run/docker.sock
Starting Docker command...                          build=81 runner=6822dcd4
Creating services...                                build=81 runner=6822dcd4
Creating cache directories...                       build=81 runner=6822dcd4
Using container f9920f5306b2c576711037d045b8465221440e092700fb6adaaa2a85d4972b82 as cache /cache ...  build=81 runner=6822dcd4
Using container 1ce6cc761bc69ef838f8eb5d3b45b873c7b0762842d96fbd343704c929bef477 as cache /builds/<user> ...  build=81 runner=6822dcd4
Looking for prebuilt image gitlab-runner-build:014aa8c ...  build=81 runner=6822dcd4
Looking for image 053f7acdcb5307c50fcdc27a50857793329d5a68e6b2c71baad3f8c65e4b9f95 ...  build=81 runner=6822dcd4
Removed container runner-6822dcd4-project-13-concurrent-0-pre with No such container: runner-6822dcd4-project-13-concurrent-0-pre  build=81 runner=6822dcd4
Creating container runner-6822dcd4-project-13-concurrent-0-pre ...  build=81 runner=6822dcd4
Looking for image 053f7acdcb5307c50fcdc27a50857793329d5a68e6b2c71baad3f8c65e4b9f95 ...  build=81 runner=6822dcd4
Removed container runner-6822dcd4-project-13-concurrent-0-post with No such container: runner-6822dcd4-project-13-concurrent-0-post  build=81 runner=6822dcd4
Creating container runner-6822dcd4-project-13-concurrent-0-post ...  build=81 runner=6822dcd4
Looking for image node:latest ...                   build=81 runner=6822dcd4
Pulling docker image node:latest ...                build=81 runner=6822dcd4
Removed container runner-6822dcd4-project-13-concurrent-0-build with No such container: runner-6822dcd4-project-13-concurrent-0-build  build=81 runner=6822dcd4
Creating container runner-6822dcd4-project-13-concurrent-0-build ...  build=81 runner=6822dcd4
6822dcd4 81 Waiting for signals...
Starting container 87e3ab8783ce5ae51f078c2227f498bea6d91f10148d285199957be7a6c09f3d ...  build=81 runner=6822dcd4
Attaching to container...                           build=81 runner=6822dcd4
Feeding runners to channel                          builds=1
Checking runner centos-6-docker url=https://gitlab.domain.com/ci token=<token> executor=docker on 1  builds=1
Checking for builds... nothing                      runner=6822dcd4
81 Submitting build to coordinator... ok            runner=6822dcd4
Waiting for container...                            build=81 runner=6822dcd4
ERROR: Build failed with: exit code 1               build=81 runner=6822dcd4
Build took 5.135048498s                             build=81 runner=6822dcd4
Waiting for build log updater to finish             build=81 runner=6822dcd4
Build log updater finished.                         build=81 runner=6822dcd4
Build log:  gitlab-ci-multi-runner 1.0.4 (014aa8c)
Using Docker executor with image node:latest ...
Pulling docker image node:latest ...

Running on runner-6822dcd4-project-13-concurrent-0 via <server>...
Fetching changes...
HEAD is now at d2e7d42 Changing test bash command
fatal: unable to access 'https://gitlab-ci-token:<token>@gitlab.domain.com/<user>/test.git/': SSL certificate problem: unable to get issuer certificate

ERROR: Build failed with: exit code 1
  build=81 runner=6822dcd4
updateBuildLog Received finish.                     build=81 runner=6822dcd4
PushTrace finished                                  build=81 runner=6822dcd4
81 Submitting build to coordinator... ok            runner=6822dcd4
Build finished                                      build=81 runner=6822dcd4
Feeding runners to channel                          builds=1
Checking runner centos-6-docker url=https://gitlab.domain.com/ci token=<token> executor=docker on 1  builds=1
Checking for builds... nothing                      runner=6822dcd4
Removed container 87e3ab8783ce5ae51f078c2227f498bea6d91f10148d285199957be7a6c09f3d with <nil>  build=81 runner=6822dcd4
Removed container b2a7e91db9c911eecccbad1e9caccf51f7b00eda35cddb74c2e82b1a1f925a0b with <nil>  build=81 runner=6822dcd4
Removed container 9decf9a4df9ca21c62bfa525f35a6ab9cd7b6cf28e50dbd7da81b952d2ad9b0a with <nil>  build=81 runner=6822dcd4
Closed all idle connections for docker.Client: &{false 0xc2081a47b0 <nil> 0x1eb2f80 unix:///var/run/docker.sock 0xc2082b6230 0xc2082b62a0 [1 18] [1 19] [1 18] 0xc2081a53b0}
Cleanup finished                                    build=81 runner=6822dcd4
Build removed id: 81
projectid: 13
commands: |2-

  npm -v
  node -v
  npm install
repourl: https://gitlab-ci-token:<token>@gitlab.domain.com/<user>/test.git
sha: 4913c101c608fa274caeed5dacca497f8787dd74
refname: master
beforesha: "0000000000000000000000000000000000000000"
allowgitfetch: true
timeout: 3600
variables:
- key: CI_BUILD_NAME
  value: test:node
  public: true
  internal: false
  file: false
- key: CI_BUILD_STAGE
  value: test
  public: true
  internal: false
  file: false
options:
  image: node:latest
token: <token>
name: test:node
stage: test
tag: false
tlscachain: |
  -----BEGIN CERTIFICATE-----
  MIIJwDCCB6igAwIBAgITXwAAABnOZvBC8H6Q0wAAAAAAGTANBgkqhkiG9w0BAQUF
  ADB0MRMwEQYKCZImiZPyLGQBGRYDbmV0MRowGAYKCZImiZPyLGQBGRYKdHJvd2Vw
  cmljZTEUMBIGCgmSJomT8ixkARkWBGNvcnAxKzApBgNVBAMTIlQuIFJvd2UgUHJp
  ################################################################
  #                          Cipher Text                         #
  ################################################################
  B3l85KuDaq2G+jkR8T8jt3fVkAdv6dKGnrz8eOgeElgZsAvQQEElwd40BvIO9yZz
  gM7Ell+Y65ez+HvV9PEu7JMh2/8ZQd8mwOmH/oB1bKXcwUnAnO3a4j5FD6BiXX3h
  VDYMwg==
  -----END CERTIFICATE-----
network: {}
buildstate: failed
buildstarted: {}
buildfinished: {}
buildduration: 5.135048498s
runner:
  name: centos-6-docker
  limit: 0
  disableverbose: null
  outputlimit: null
  runnercredentials:
    url: https://gitlab.domain.com/ci
    token: <token>
    tlscafile: /etc/ssl/certs/ca-bundle.crt
  runnersettings:
    executor: docker
    buildsdir: null
    cachedir: null
    environment:
    - MYSQL_ALLOW_EMPTY_PASSWORD=1
    shell: bash
    ssh: null
    docker:
      dockercredentials:
        host: null
        certpath: null
        tlsverify: null
      hostname: null
      image: centos6
      privileged: false
      disablecache: null
      volumes:
      - /cache
      cachedir: null
      extrahosts: []
      links: []
      services: []
      waitforservicestimeout: null
      allowedimages: []
      allowedservices: []
      imagettl: null
    parallels: null
    virtualbox: null
globalid: 0
runnerid: 0
projectrunnerid: 0
  builds=0
ReadTrace finished                                  build=81 runner=6822dcd4
Feeding runners to channel                          builds=0
Checking runner centos-6-docker url=https://gitlab.domain.com/ci token=<token> executor=docker on 1  builds=0
Checking for builds... nothing                      runner=6822dcd4
Feeding runners to channel                          builds=0
Checking runner centos-6-docker url=https://gitlab.domain.com/ci token=<token> executor=docker on 0  builds=0
Checking for builds... nothing                      runner=6822dcd4
Feeding runners to channel                          builds=0
Checking runner centos-6-docker url=https://gitlab.domain.com/ci token=<token> executor=docker on 1  builds=0
Checking for builds... nothing                      runner=6822dcd4
^CWARNING: Requested service stop                     builds=0
All workers stopped. Can exit now                   builds=0
Stopping worker 0                                   builds=0
Stopping worker 1                                   builds=0

CI build log:

gitlab-ci-multi-runner 1.0.4 (014aa8c)
Using Docker executor with image node:latest ...
Pulling docker image node:latest ...

Running on runner-6822dcd4-project-13-concurrent-0 via <server>...
Fetching changes...
HEAD is now at d2e7d42 Changing test bash command
fatal: unable to access 'https://gitlab-ci-token:xxxxxx@gitlab.domain.com/<user>/test.git/': SSL certificate problem: unable to get issuer certificate

ERROR: Build failed with: exit code 1

mentioned in issue gitlab-mattermost#22 (closed)

Any news on this, as this bug was marked for v1.1 milestone? We are having the same CA verification issue behind a corporate firewall.

The only way I was able to get around this issue was to directly include both server cert and intermediate cert into the nginx certificate of GitLab. Then the CI runner will have both of those certs and validate it with the root cert as specified in its config (even though it also should be trusting the intermediate).

I'm going to link to a few other issues that relate to needing to set up GitLab with your own CA:

mentioned in issue omnibus-gitlab#712 (closed)

mentioned in issue #2367 (closed)

There is another way: You can inject a bundled certificate file into each docker container and adjust the container's git configuration to trust your selfsigned certificates by executing git config --global http.sslCAInfo /etc/ssl/certs/your_self_signed_ca_cert.crt before cloning the git repository. Additionally you need to reference a CA file with tls-ca-file.

A config.toml might look like this:

[[runners]] name = "AwesomeRunner" url = "https://git.example.net" token = "123546879" executor = "docker" tls-ca-file = "/etc/gitlab-runner/certs/ca.crt" [runners.docker] volumes = ["/cache", "/etc/gitlab-runner/certs/ca.crt:/etc/ssl/certs/your_self_signed_ca_cert.crt"] pre_clone_script = "git config --global http.sslCAInfo /etc/ssl/certs/your_self_signed_ca_cert.crt"

Doing it like this you do not need to turn off the verification or include both server cert and intermediate cert into the nginx certificate of GitLab. Furthermore it does not add any unnecessary configuration to your .gitlab-ci.yml.

SSL certificate problem: unable to get issuer certificate

Designs

Child items ...

Activity

Admin message

Admin message

SSL certificate problem: unable to get issuer certificate

Activity