Sometimes SSL certificate problem
Approximateley every 20. build interrupts with the following error:
gitlab-ci-multi-runner 1.1.3 (a470667)
Using Shell executor...
Running on GITLAB-SRV-07...
Fetching changes...
HEAD is now at 0d44302 Merge branch 'fix-compression-rate' into 'master'
fatal: unable to access 'https://gitlab-ci-token:xxxxxx@domain.tld/group-namespace/repository.git/': SSL certificate problem: unable to get local issuer certificate
ERROR: Build failed: exit status 1In this case I press "retry" and the build works as expected. But somethimes I get this error two or more times in sequence.
concurrent = 1
[[runners]]
url = "https://domain.tld/ci"
token = "xxxxxx"
name = "GITLAB-SRV-07"
executor = "shell"
shell = "powershell"
[runners.ssh]
[runners.docker]
image = ""
privileged = false
[runners.parallels]
base_name = ""Activity
I got the same problem. Platform: Windows Server 2008 R2 (x64) Sometimes happens, press "retry" maybe works.
gitlab-ci-multi-runner 1.1.4 (9e2fd1a) Using Shell executor... Running on CX-TECH... Fetching changes... HEAD is now at 14f0e2d XXXXXXXX fatal: unable to access 'https://gitlab-ci-token:xxxxxx@xxxxxx.com/xxx/xxxxxx.git/': SSL certificate problem: unable to get issuer certificate ERROR: Build failed: exit status 1Here is my configure file.
concurrent = 1 [[runners]] url = "https://xxxxxx.com/ci" token = "xxxxxxxxxxxxxxxxxxxxxxxxxx" tls-skip-verify = false tls-ca-file = "/usr/ssl/certs/ca-bundle.crt" name = "VS2013 Runner" executor = "shell" shell = "powershell" [runners.ssh] [runners.docker] image = "" privileged = false [runners.parallels] base_name = ""the problem still perstists in v1.2 :(
/cc @ayufan @tmaczukin
@tmaczukin Can you debug this?
@ayufan Yes, I'll look on this. It looks like a same/similar problem that are reported in #1103 (closed) and #1300 (closed).
@tmaczukin do you have a solution approach?
@maxraab Not yet. We have several issues that are pointing us to this problem - different versions, different platforms, different settings. I need to find a way to reproduce it on my local machine (or any other machine). Then I can look for a reasons and for solutions.
@tmaczukin Let me know if I can help you. I can test specific versions in our build system.
Milestone changed to %v1.3
Hmm.. I also got this problem, but with a 100% possibility. My certificate is signed and trusted. I'm running 1.2.0.
Edited by username-removed-108132@tmaczukin, @ayufan: Even when I set
git config --global http.sslVerify false, the problem persists. How can I configure the build-in runner-git?@maxraab Try setting
variables: - GIT_SSL_NO_VERIFY: "1"@ayufan After about 30 minutes of testing, it seems to work. But now I have to update about 60 repositories
😢 Could you reproduce this issue? Can I help with debug information, ...?
@maxraab Be aware that this disables certificate testing. It would be better to discover why SSL checks fails for you in the first place.
Is this everything related to Windows Server? Did you try to specify SSL certificate in runner configuration?
@ayufan I know this isn't a good solution but currently we have about 200 builds a day and approximate 30 build fails with the named error. All runners are installed on normal Win7 machines (no Windows Server). My
config.tomlcontains thetls-ca-fileentry with path to theGitLabCA.ctr.all configurations expect:
concurrent = 1 [[runners]] url = "https://domain.tld/ci" token = "xxxxxx" name = "GITLAB-SRV-07" executor = "shell" shell = "powershell" environment = ["GIT_SSL_NO_VERIFY=1"] [runners.ssh] [runners.docker] image = "" privileged = false [runners.parallels]get the certificate problem.
Edited by username-removed-289790@maxraab Can you check if the same happens when doing
git clonefrom command line?@ayufan I can't clone directly via command line.
I tested the SSL url, the HTTPS url and the runner url (and replaced the xxxxx through the access token). Even when I go to
/builds/*subdirectory, where the runner store it's builds, I can notgit pullon one of these repos.cc @tmaczukin
The certificate we are using is certified by
GlobalSign Domain Validation CA - SHA256 - G2and valid in all browsers and connections.I tried to clone one of our repos on a developers machine via
HTTPSand I get the same error. (SSLworks well).Some GitLab-Settings:
- repo is in an internal group
- repo is internal
- no registration supported
- you have to sign in to use gitlab
- username-pattern: firstname.lastname
- projectname-pattern: firm.technology.project
Even when I try to clone the repo via
https://user:password@subdomain.domain.tld/group/project.gitdirectly via git-bash, I get the same error.@ayufan @tmaczukin Is there a general misconfiguration?
@maxraab Yes, it's possible that you are missing intermediate certificates.
Check your site with: https://www.ssllabs.com/ssltest/.
It should show you reduced grade if you miss intermediates.
@ayufan Unfortunately is the server is only reachable from the intranet/VPN so I could not check on this site. I setup a fresh ubuntu and install the omnibus packages. After that i followed the instructions on: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md
The
domain.crtanddomain.keyfiles are located under/etc/gitlab/ssl/.The browser shows a "green" connection, when I our GitLab instance.
You should put only and in this order:
-----BEGIN CERTIFICATE----- #Your GlobalSign SSL Certificate# -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- #GlobalSign Intermediate Certificate# -----END CERTIFICATE-----Having
rootin same cases can be considered as an error.@ayufan Now I can access my repo via curl, git and browsers without the intermediate cached files. I also can clone the repo via the runner's machines.
The runner has changed his error too:
gitlab-ci-multi-runner 1.3.2 (0323456) Using Shell executor... Running on GITLAB-SRV-07... Fetching changes... HEAD is now at 0099d02 Merge branch 'implements-using-of-reportdesignerworksheetdocument' into 'master' fatal: unable to access 'https://gitlab-ci-token:xxxxxx@domain.tld/group-namespace/repository.git/': SSL certificate problem: unable to get issuer certificate ERROR: Build failed: exit status 1from: "SSL certificate problem: unable to get
localissuer certificate" to: "SSL certificate problem: unable to get issuer certificate"@ayufan The validation with
openssllooks good after your configuration help. But the runners still fail approx. every 3. build.$ openssl s_client -showcerts -connect domain.tld:443 CONNECTED(000001AC) --- Certificate chain 0 s:/OU=Domain Control Validated/CN=*.domain.tld i:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA - SHA256 - G2 -----BEGIN CERTIFICATE----- ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC/BtuQlV9gDDqSmad88F1S MGAxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMTYwNAYD VQQDEy1HbG9iYWxTaWduIERvbWFpbiBWYWxpZGF0aW9uIENBIC0gU0hBMjU2IC0g RzIwHhcNMTUxMTE5MTM1MTIxWhcNMTYxMTE5MTM1MTIxWjA7MSEwHwYDVQQLExhE b3NpdG9yeS8wYgYDVR0RBFswWYINKi5uc2MtZ21iaC5kZYIYYXV0b2Rpc2NvdmVy rTraOHokbyqK/EsqpsRUxO0X0fJDFerfr+1XAkTB+MN8MiQz8Pom/vhwlV2u6Wjn DBnekFahlFr9qPCdteO5yiAzzNrq5RN0bpqKH7pggSoAYG67xREHmrfVSDqdn7d+ MIIFMjCCBBqgAwIBAgISESFcBd12lxEwg9HKgfD2t9FNMA0GCSqGSIb3DQEBCwUA lw8wDQYxxxxxxxxxxQELBQADggEBAJU/6diRPYIf5k0Lo6Mf6KPiKZyfYfHJGie/ 0+W18f3wDTQhmF7JcLabOBPsa/dwEYk1jYWu1zHNcKDTyeL+8/2Pq2uStnPntNF3 b21haW4gQ29udHJvbCBWYWxpZGF0ZWQxFjAUBgNVBAMMDSoubnNjLWdtYmguZGUw HYkHwd1AZQfZnDecPBpSJiE/T7oKDpv7EE/aBXYH+n/+9SHSn07tZsqI8kwvzGHR 6fBRRR4RqCGwVCXaJJMLCpkArZzm8Wj/2DNlKltoY9uhzPrAsIMLqtvFyJC56jVL AgMBAAGjggIJMIICBTAOBgNVHQ8BAf8EBAMCBaAwSQYDVR0gBEIwQDA+BgZngQwB Z3Nkb21haW52YWxzaGEyZzJyMS5jcnQwOQYIKwYBBQUHMAGGLWh0dHA6Ly9vY3Nw AgEwNDAyBggrBgEFBQcCARYmaHR0cHM6Ly93d3cuZ2xvYmFsc2lnbi5jb20vcmVw b3NpdG9yeS8wYgYDVxxxxxxxxxxxxxxxxxxtZ21iaC5kZYIYYXV0b2Rpc2NvdmVy Lm5zYy1nbWJoLmRlghBtYWlsLm5zYy1nbWJoLmRlgg9vd2EubnNjLWdtYmguZGWC C25zYy1nbWJoLmRlMAkGA1UdEwQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsG AQUFBwMCMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5j b20vZ3MvZ3Nkb21haW52YWxzaGEyZzIuY3JsMIGUBggrBgEFBQcBAQSBhzCBhDBH BggrBgEFBQcwAoY7aHR0cDovL3NlY3VyZS5nbG9iYWxzaWduLmNvbS9jYWNlcnQv 6fBRRR4RqCGwVCXaJJMLCpkArZzm8Wj/2DNlKltoY9uhzPrAsIMLqtvFyJC56jVL Mi5nbG9iYWxzaWduLmNvbS9nc2RvbWFpbnZhbHNoYTJnMjAdBgNVHQ4EFgQUQSXP QWBIZYdll2q+bqdDy017RqYwHwYDVR0jBBgwFoAU6k581IAt5RWBhiaMgm3AmKTP b3NpdG9yeS8wYgYDVR0RBFswWYINKi5uc2MtZ21iaC5kZYIYYXV0b2Rpc2NvdmVy HbiCUo+30ZTinGvBjL+SrME3ezgsHdgGiS9QeqCKwQzz3ZAI3kAgbLdwpJCKRZpO 5nAu39eJ8XE6x4oyOf/1lql7vR180GeLLcXnoPFF/8KKkL69Fq5S5uoF9eWrhMZ0 WmL8efQSFHg13fxxxxxxxxxxxxxxxxxxxxxZPpROVhrmmN9RIUIZmmNgz/ZNe2mH Cbaqo1Q2QOHDpz3ueQFg4akVn4iZeLbQhlDPBkxi49u6S/Qx8RE5bGujycKlM7Jq HUavp1aXYU7opLPL0BgIOhNjvmuUzlgdf7TRKmElnWQUDBU881c= -----END CERTIFICATE----- 1 s:/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA - SHA256 - G2 i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA -----BEGIN CERTIFICATE----- MIIEYzCCA0ugAwIBAgILBAAAAAABRE7wPiAwDQYJKoZIhvcNAQELBQAwVzELMAkG tywdO02L3ORkHQRYM68bVeerDL8wBHTk8w4vMDmNSwSMHnVmZkngvkA0x1xaUZK6 b3QgQ0ExGzAZBgNVBAMTEkdsb2JhbFNpZ24gUm9vdCBDQTAeFw0xNDAyMjAxMDAw MDBaFw0yNDAyMjAxMDAwMDBaMGAxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9i YWxTaWduIG52LXNhMTYwNAYDVQQDEy1HbG9iYWxTaWduIERvbWFpbiBWYWxpZGF0 aW9uIENBIC0gU0hBMjU2IC0gRzIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK AoIBAQCp3cwOs+IyOd1JIqgTaZOHiOEM7nF9vZCHll1Z8syz0lhXV/lG72wm2DZC jn4wsy+aPlN7H262xxxxxxxxxxxxxxxxxxxeyr3sBppqKqAZUn9R0XQ5CJ+r69eG 4bjyqdKKEYRxkRWJ3AKdC8tsM4U0KJ4gsrGX3G0LEME8zV/qXdeYMcU0mVwAYVXE Imh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5uZXQvcm9vdC5jcmwwPQYIKwYBBQUHAQEE ExWXrjbDVGYOWvKgc4Ux47JkFGr/paKOJLu9hVIVonnu8LXuPbj0fYC82ZA1ZbgX qa2zmJ+gfn1u+z+tfMIbWTaW2jcyS0tdNQJjjtunz2LuzC7Ujcm9PGqRcqIip3It INH6yjfaGJjmFiRxJUvE5XuJUgkC/VkrBG7KB4HUs9ra2+PMgKhWBwZ8lgg3nds4 tmIXxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxgElMIIBITAOBgNVHQ8BAf8EBAMC AQYwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHQ4EFgQU6k581IAt5RWBhiaMgm3A mKTPlw8wRwYDVR0gBEAwPjA8BgRVHSAAMDQwMgYIKwYBBQUHAgEWJmh0dHBzOi8v d3d3Lmdsb2JhbHNpZ24uY29tL3JlcG9zaXRvcnkvMDMGA1UdHwQsMCowKKAmoCSG b3QgQ0ExGzAZBgNVBAMTEkdsb2JhbFNpZ24xxxxxxxxxxxxxxxxxxxxxMjAxMDAw MTAvMC0GCCsGAQUFBzABhiFodHRwOi8vb2NzcC5nbG9iYWxzaWduLmNvbS9yb290 cjEwHwYDVR0jBBgwFoAUYHtmGkUNl8qJUC99BM00qP/8/UswDQYJKoZIhvcNAQEL BQADggEBANdFnqDc4ONhWgt9d4QXLWVagpqNoycqhffJ7+mG/dRHzQFSlsVDvTex A1UEBhMCQkUxGTAXBgNVBAoTEEdsb2JhbFNpZ24gbnYtc2ExEDAOBgNVBAsTB1Jv GwJbxeOJyLS4bx448lYm6UHvPc2smU9ZSlctS32ux4j71pg79eXw6ImJuYsDy1oj H6T9uOr7Lp2uanMJvPzVoLVEgqtEkS5QLlfBQ9iRBIvpES5ftD953x77PzAAi1Pj tyxxxxxxxxxxxQRYM68bVeerDL8wBHTk8w4vMDmNSwSMHnVmZkngvkA0x1xaUZK6 EjxS1QSCVS1npd+3lXzuP8MIugS+wEY= -----END CERTIFICATE----- --- Server certificate subject=/OU=Domain Control Validated/CN=*.domain.tld issuer=/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Domain Validation CA - SHA256 - G2 --- No client certificate CA names sent Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 3139 bytes and written 434 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 5C5273425D011EBFC52E77CCFBD44D3xxxxxxxxxxxxxxxC7D326DBB385BF7D23 Session-ID-ctx: Master-Key: B6D22D776C806F46808xxxxxxD38F8FB13D5xxxxxxxxxxC96EA6D85A9B482B23EBBxxxxxx82651EA59EB653AB82561 Key-Arg : None PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 300 (seconds) TLS session ticket: 0000 - 06 17 72 78 26 db 25 87-a0 22 98 c8 ce e6 00 9c ..rx&.%.."...... 0010 - xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx ......ke.`.....t 0020 - d1 cc 30 9a 4b b5 ed 12-38 c1 46 87 09 3d a1 c4 ....h..<8.F..=.. 0030 - xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx *V..C.q.n....NE. 0040 - 6d 9a f5 36 9e 23 23 e7-cc 1e 7b ee 76 f7 a0 91 m..6......{.v... 0050 - xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx h%+.]........@.. 0060 - 46 a6 f9 7d dd 69 54 ae-58 f8 fe a1 e1 96 5d 9e F..}.iT.X.....]. 0070 - xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx .c....N....!).C. 0080 - 04 3c cf ca 3d 1f b0 b6-c8 88 14 b2 e0 a8 b1 78 .<.............x 0090 - xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx }..s.....Yu..n.8 00a0 - a5 e6 4a 52 f4 eb 57 5f-b8 f5 ba aa cf cb f2 3a ..WE..W_...<...: Start Time: 1467312129 Timeout : 300 (sec) Verify return code: 0 (ok) --- closed depth=2 C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA verify return:1 depth=1 C = BE, O = GlobalSign nv-sa, CN = GlobalSign Domain Validation CA - SHA256 - G2 verify return:1 depth=0 OU = Domain Control Validated, CN = *.domain.tld verify return:1@maxraab This seems more than strange that it fails every 3rd time :) Do you have multiple runners or only one?
@ayufan I have 2 runners and especially if there are many builds at the same time (i.e. 10 Builds in the queue) the fail-rate is higher than normal. Each build runs approx. 3 min. Also builds which get executed immediate after a merge request is merged (accept mr and direct sync a fork) fails often too.
Today I registered a new runner (1.3.2) on Win10. But still the same error.
Edited by username-removed-289790@maxraab Please let me know when we could do a Hangout session to try to debug your problem.
@ayufan Do you need access to one of our build-machines?
@maxraab Yes, it would be handy.
@ayufan I could provide a TeamViewer session, because our installations are not public. But I have to talk to our administration first.
@maxraab No problem. TeamViewer works for me, anything that will make it easier to debug.
mentioned in merge request !230 (closed)
@ayufan Still the same problem in v1.4.1
@luxferresum Maybe. But we don't get any updates here .. After 4 months the issue gets still ignored :(
In the meantime we're considering switching to another ci service. This bug prevents a smooth workflow.
Edited by username-removed-289790@tmaczukin any thoughts on this?
Milestone changed to %Backlog
I've been having a look at this at @ayufan's request.
As far as I can see, the certificate data flow is:
- gitlab-ci-multi-runner has a cacert.pem file which it uses to verify the GitLab server (network/client.go:38)
- When receiving a build, the TLS certificate chain used by GitLab is read from the TLS connection state (network/client.go:89) and stuffed into the build config (network/gitlab.go:97)
- These certificates are written to a file in the project-specific directory by the prepare script
The code that does the conversion from TLS state to PEM-encoded certificate data is suspect. Is there a reason why we don't just copy the file used by gitlab-ci-multi-runner proper, for the builds to use?
A specific problem I can see in the conversion function here: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/network/client.go#L89
list := make(map[string]*x509.Certificate) # ... for _, certificate := range list { certificates += string(pem.EncodeToMemory(&pem.Block{ Type: "CERTIFICATE", Bytes: certificate.Raw, })) }This will non-deterministically shuffle the order of the certificates - a map is not a list, and order is not guaranteed in a list. Could this account for the symptoms we're seeing? Here's an example of what I mean: https://play.golang.org/p/N-A0dJGXdp
Edited by Nick Thomas@nick.thomas Could you rework this function and we give it a try?
mentioned in merge request !299 (merged)
@maxraab see !299 (merged).
If there is a reason for filtering the certificates a build receives, I'll push a second MR to at least preserve the order.
Reassigned to @nick.thomas
mentioned in issue #1103 (closed)
mentioned in issue #1300 (closed)
mentioned in issue #1297 (closed)
mentioned in issue #1396 (closed)
Milestone changed to %v1.6
Status changed to closed by merge request !299 (merged)
@djthayer Does https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/merge_requests/299 solve that for you? If not, can you create separate issue?
@ayufan that new issue already exists since a month now: #2148 (closed) ... Its simple to say: the ssl-certificate-problem is not solved. Cloning submodules only works with the "old" method (using
before_script) presumably due to the fact that that happens in the actual build-container which is in control by the user and thus usually contains the necessary stuff for ssl to work properly.mentioned in issue #2367 (closed)
