502 errors on dev due to Prometheus metrics
dev deployed the latest nightly build with gitlab-org/gitlab-ce!13674, but today I noticed it was unusable due to 502 errors. It looks like all unicorn workers were tied up. A quick backtrace showed:
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/actionpack-4.2.8/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
from /opt/gitlab/embedded/service/gitlab-rails/app/controllers/metrics_controller.rb:9:in `index'
from /opt/gitlab/embedded/service/gitlab-rails/app/services/metrics_service.rb:24:in `metrics_text'
from /opt/gitlab/embedded/service/gitlab-rails/app/services/metrics_service.rb:14:in `prometheus_metrics_text'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta12/lib/prometheus/client/formats/text.rb:40:in `marshal_multiprocess'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta12/lib/prometheus/client/formats/text.rb:96:in `load_metrics'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta12/lib/prometheus/client/formats/text.rb:96:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta12/lib/prometheus/client/formats/text.rb:102:in `block in load_metrics'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta12/lib/prometheus/client/formats/text.rb:102:in `each'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta12/lib/prometheus/client/formats/text.rb:116:in `block (2 levels) in load_metrics'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/prometheus-client-mmap-0.7.0.beta12/lib/prometheus/client/formats/text.rb:116:in `zip'
I've turned off Prometheus metrics until this is sorted out.
/cc: @pchojnacki, @joshlambert, @bjk-gitlab, @marin