Slack service no longer working afer 7.10.0 upgrade

Error goes as follows:

Started GET "/wrt/client/services/slack/test" for <ip> at 2015-04-22 19:19:32 -0400
Processing by Projects::ServicesController#test as HTML
Parameters: {"namespace_id"=>"wrt", "project_id"=>"client", "id"=>"slack"}
Completed 500 Internal Server Error in 749ms
OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed):

I tried downgrading back to 7.9.4 and the service worked just fine (And it's been working for months).

openssl s_client -showcerts -connect hooks.slack.com:443 -CApath /etc/ssl/certs yields

Verify return code: 0 (ok)

and no errors.

Am I missing something or is it an issue with gitlab?

+1

This just happened to me as well. I tried replacing /opt/gitlab/embedded/ssl/certs/cacert.pem with the one from v7.9.4, but that did not fix it. I wonder if the upgrade to ruby v2.1.6 did something bad.

Oh! I found the problem: the cert file has the wrong permissions. Make sure you restart after setting the permissions properly:

$ ls -al /opt/gitlab/embedded/ssl/certs/cacert.pem
-rw------- 1 root root 258424 Apr 22 15:35 /opt/gitlab/embedded/ssl/certs/cacert.pem
$ sudo chmod a+r /opt/gitlab/embedded/ssl/certs/cacert.pem
$ sudo gitlab-ctl restart

@jacobvosmaer, do you know what happened here with the permissions?

@stanhu Thank you!

/cc @marin

mentioned in issue #1438 (closed)

@stanhu Cacerts are updated to the latest version that got released yesterday. This updates everytime the same way so I have no clue why would it be set with the wrong permissions.

@stanhu This is also quite weird as I see on the build machines that the cacert.pem has the correct permissions:

-rw-r--r-- 1 omnibus-build omnibus-build 258424 Apr 22 15:20 cacert.pem

@martin That is quite bizarre. What are the timestamps of the omnibus packages compared to the modification timestamp for this file? I wonder if the permissions were somehow updated after package creation.

I can confirm that all the posted 7.10 .deb and .rpm packages have this permission issue. Comparing against 7.9.3, I believe that cacert.pem is the only file that received these new restrictive permissions. Do you happen to have the packages for the RCs to check?

@stanhu Hmm, it is quite weird that all the packages have this seeing that on the build machine I have this for the package:

-rw-r--r-- 1 omnibus-build omnibus-build 309466748 Apr 22 15:35 gitlab-ce_7.10.0~omnibus+20150422152017-1_amd64.deb

And the contents show:

$dpkg -c gitlab-ce_7.10.0~omnibus+20150422152017-1_amd64.deb | grep cacert
-rw-r--r-- root/root    258424 2015-04-22 15:30 ./opt/gitlab/embedded/ssl/certs/cacert.pem

And the end package has this:

-rw------- root/root    258424 2015-04-22 15:42 ./opt/gitlab/embedded/ssl/certs/cacert.pem\

I think this discussion belongs in omnibus-gitlab#523 (closed) ; it's strictly about omnibus packages.

Fixed in the latest omnibus packages, also see https://about.gitlab.com/2015/04/23/gitlab-7-dot-10-dot-0-omnibus-patch-release/

Status changed to closed