Install and scrape pgbouncer exporter in production

We finally have a version that we can use with an MIT license: https://github.com/larseen/pgbouncer_exporter/releases

Let's use it.

mentioned in issue #2478 (closed)

"good news everyone"

@_stark would you consider doing this as your first assignment? This could work as a good introduction to the whole infrastructure side of things.

cc/ @yorickpeterse @sitschner

@pcarranza Shouldn't we also ship this with Omnibus? If so I'd rather have us add it to Omnibus first instead of again setting things up manually. Looking at past experiences it usually takes too long to get rid of those manual setups.

@marin how do you feel about shipping an exporter that we don't know how (or if it will) work at all?

Do you think we should try it first in production?

@yorickpeterse you are actually bringing a really good point: we should only use what we ship in Omnibus.

Unless we get someone up in the chain of command vetting this. @sitschner do you think it makes sense to investigate this monitoring solution to prevent outages originating in PGBouncer before pushing it into the package or should we just push it into the package and hope for the best?

Can I haz a signoff?

Obviously we want to ship what we use and use what we ship. However in there I think we should also make sure it works before we ship it.

We have to at least test it before shipping. @yorickpeterse I understand this will cause duplication of work, but that is better than shipping something that doesn't work or worse, causes issues to users.

you has your signoff, test it.

I agree with Yorick that it is a bit tiring to test things manually but it has helped us in the past so +1 @sitschner

@_stark please pick this up and reach out to people for any help you need.

assigned to @_stark

mentioned in issue #2719 (closed)

So far this is what I have.

• Create a recipe to add the exporter on pgbouncer hosts https://gitlab.com/gitlab-cookbooks/gitlab-prometheus/merge_requests/301

• Add the exporter recipe to the db role https://dev.gitlab.org/cookbooks/chef-repo/merge_requests/1137

@bjk-gitlab I'm mostly just pattern matching here. Not sure if I've hit all the right places or I'm on the right track.

mentioned in issue #2762 (closed)

So on further analysis I think this is on hold.

1. I don't think it makes any sense to deploy this now with our current pgbouncer config only to have to change it in a matter of days when https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2571 / https://gitlab.com/gitlab-com/infrastructure/issues/2581 is rolled out in production. When that's rolled out we would have to change the roles attached and possibly other config as well.

2. I will be focusing more on database tasks and think this should be handed off to @jtevnan or @jarv or some other victim more experienced with chef automation.

marked this issue as related to #2581

added prometheus label

mentioned in issue gitlab-org/omnibus-gitlab#2815