Oj::ParseError for Gitlab Metrics after few minutes
Summary
After enabling prometheus metrics and get everything running for some minutes the page responds with a 500.
We are running Gitlab 10.0.1
What is the current bug behavior?
/-/metrics
returns error 500
What is the expected correct behavior?
It should return the metrics
Relevant logs and/or screenshots
Processing by MetricsController#index as HTML
Parameters: {"token"=>"[FILTERED]"}
Completed 500 Internal Server Error in 101ms (ActiveRecord: 0.6ms)
Oj::ParseError (Array not terminated at line 1, column 108 [parse.c:894]):
app/services/metrics_service.rb:14:in `prometheus_metrics_text'
app/services/metrics_service.rb:24:in `metrics_text'
app/controllers/metrics_controller.rb:9:in `index'
lib/gitlab/middleware/multipart.rb:93:in `call'
lib/gitlab/request_profiler/middleware.rb:14:in `call'
lib/gitlab/middleware/go.rb:17:in `call'
lib/gitlab/etag_caching/middleware.rb:11:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:20:in `call'
lib/gitlab/metrics/rack_middleware.rb:29:in `block in call'
lib/gitlab/metrics/transaction.rb:49:in `run'
lib/gitlab/metrics/rack_middleware.rb:29:in `call'
lib/gitlab/request_context.rb:18:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'