Add all InfluxDB metrics to Prometheus

Description

Add Prometheus metrics wherever existing InfluxDB hooks exist.

Proposal

Links / references

https://gitlab.com/gitlab-org/gitlab-ce/issues/29118

Documentation blurb

Here is what I've found so far for influxdb metrics being generated by Rails and Sidekiq:

Metrics from "web" unicorn:

• rails_file_descriptors
• rails_gc_statistics_...
  • rails_gc_statistics_count
  • rails_gc_statistics_heap_allocatable_pages
  • rails_gc_statistics_heap_allocated_pages
  • rails_gc_statistics_heap_available_slots
  • rails_gc_statistics_heap_eden_pages
  • rails_gc_statistics_heap_final_slots
  • rails_gc_statistics_heap_free_slots
  • rails_gc_statistics_heap_live_slots
  • rails_gc_statistics_heap_marked_slots
  • rails_gc_statistics_heap_sorted_length
  • rails_gc_statistics_heap_swept_slots
  • rails_gc_statistics_heap_tomb_pages
  • rails_gc_statistics_major_gc_count
  • rails_gc_statistics_malloc_increase_bytes
  • rails_gc_statistics_malloc_increase_bytes_limit
  • rails_gc_statistics_minor_gc_count
  • rails_gc_statistics_old_objects
  • rails_gc_statistics_old_objects_limit
  • rails_gc_statistics_oldmalloc_increase_bytes
  • rails_gc_statistics_oldmalloc_increase_bytes_limit
  • rails_gc_statistics_remembered_wb_unprotected_objects
  • rails_gc_statistics_remembered_wb_unprotected_objects_limit
  • rails_gc_statistics_total_allocated_objects
  • rails_gc_statistics_total_allocated_pages
  • rails_gc_statistics_total_freed_objects
  • rails_gc_statistics_total_freed_pages
  • rails_gc_statistics_total_time
• rails_memory_usage

Rails metrics by Controller#Action

• rails_method_calls_call_count
• rails_method_calls_cpu_duration
• rails_method_calls_duration
• rails_transactions_allocated_memory
• rails_transactions_banzai_cacheless_render_call_count
• rails_transactions_banzai_cacheless_render_cpu_time
• rails_transactions_banzai_cacheless_render_real_time
• rails_transactions_cache_count
• rails_transactions_cache_duration
• rails_transactions_cache_read_count
• rails_transactions_cache_read_duration
• rails_transactions_cache_read_hit_count
• rails_transactions_cache_read_miss_count
• rails_transactions_cache_write_count
• rails_transactions_cache_write_duration
• rails_transactions_commit_raw_diffs_call_count
• rails_transactions_commit_raw_diffs_cpu_time
• rails_transactions_commit_raw_diffs_real_time
• rails_transactions_duration
• rails_transactions_git_blob_load_all_data_call_count
• rails_transactions_git_blob_load_all_data_cpu_time
• rails_transactions_git_blob_load_all_data_real_time
• rails_transactions_new_redis_connections
• rails_transactions_project_raw_show_call_count
• rails_transactions_project_raw_show_cpu_time
• rails_transactions_project_raw_show_real_time
• rails_transactions_rails_queue_duration
• rails_transactions_sql_count
• rails_transactions_sql_duration
• rails_transactions_view_duration

Rails metrics by Controller#Action and view

• rails_views_duration

Sidekiq

Sidekiq also exposes the same process and gc stats as Rails. InfluxDB prefixes these with sidekiq_.

Sidekiq Object counts by type

• sidekiq_object_counts_count

Not a cumulative counter, a gauge of how many objects by things like API::APIGuard::ClassMethods.

Some information about metric.

• InfluxDB does not have explicit TYPEs the same way Prometheus does.
• _count metrics tend to be cumulative counters
• _duration metrics are sent as event samples, likely candidates for counter or histogram conversion.
• Most other metrics are also sent as event samples and need to be manually converted to gauges or counters.

CC @pchojnacki @joshlambert

Thanks @bjk-gitlab! Looking at the code GC statistics should be straightforward to implement.

However the 'transaction' related metrics seem to be the most complex. I also think that cardinality of some (if not all) of transaction metrics might be quite high especially:

rails_method_calls_call_count
rails_method_calls_cpu_duration
rails_method_calls_duration

Looking at the code we should expect almost the same set of metrics but prefixed with sidekiq_ instead of rails_ for Sidekiq based metrics.

\cc @yorickpeterse @sitschner @pcarranza FYI.

For Rails and Sidekiq we generate the same metrics, and indeed only the prefix is different so we can visualise/browse the data separately.

@yorickpeterse We will be able to drop the prefix for Prometheus. We will have the job label to separate them.

We already have the http histogram from the rack middleware. I would suggest we modify that to include separate controller and action labels instead of the single action label we get out of influxdb. While the cardinality is high for controller and action. There is a large value in having response times separated out.

Based on the survey of access to staging so far, there are about 250 Controller#Action combinations. If we had 10 histgram buckets + Inf + _sum + _count we get about 3200 series. For a typical GitLab install with one unicorn server, this is not too bad. Even for gitlab production where we have 40 unicorns, that's 130k series. This is quite a lot, but the value for monitoring is important.

We already have the http histogram from the rack middleware. I would suggest we modify that to include separate controller and action labels instead of the single action label we get out of influxdb. While the cardinality is high for controller and action. There is a large value in having response times separated out.

This would require changing a ton of things in Grafana, so I'm not a fan of this. Further, this doesn't play nice with Grape which doesn't really have a concept of controllers vs actions. I'd rather just keep action with values such as UsersController#show instead of controller = UsersController, action = show.

We can split the controller and action labels for Prometheus, but leave the internal representation as-is for Grape.

We already have to completely re-do Grafana as the InfluxDB query language (SQL-ish) is completely different from Prometheus PromQL.

There are a number of advantages to splitting up controller and action labels in Prometheus.

• It's easier to do selections like metric{controller="Foo"} than have to resort to regexps like metric{controller=~"Foo#.*"}.
• The label indexing in Prometheus allows for faster selection when using string literals like above than regexps that have to do full index scans.
• We can still use Grafana formatting to produce the same "view" with things like {{controller}}#{{action}}.
• Grafana will template each label individually, so you can select a Controller, and see all actions. Or select a Controller and a specific set of Action(s).
• Grafana's templating is smart enough to give you only the Actions related to a specific Controller.

changed milestone to %10.0 in gitlab-foss

Setting 10.0 for this.

Let's make sure we hit this, not just because it's our OKR's but also this is the 10.0 release.

mentioned in merge request gitlab-foss!13366 (closed)

assigned to @pchojnacki

mentioned in merge request prometheus-client-mmap!6 (merged)

mentioned in issue gitlab-foss#35870 (closed)

mentioned in merge request gitlab-foss!13891

mentioned in issue gitlab-com/infrastructure#2379 (closed)

https://gitlab.com/gitlab-com/infrastructure/issues/2379 to be unblocked by this

mentioned in issue gitlab-com/infrastructure#1955

Current sample output with all the actions split into controller & action labels. I've also split call signatures for autoinstrumented methods. https://gitlab.com/snippets/1674154

Performance of the serialization requests have approached 1s for 95th percentile. I'm worried longer running instance with more workers than 2 will take couple times more to process all metrics.

Requests per second:    1.38 [#/sec] (mean)
Time per request:       726.380 [ms] (mean)
Time per request:       726.380 [ms] (mean, across all concurrent requests)
Transfer rate:          433.66 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       0
Processing:   480  726 669.0    590    4758
Waiting:      479  670 338.5    589    2567
Total:        480  726 669.1    590    4759

Percentage of the requests served within a certain time (ms)
  50%    590
  66%    608
  75%    681
  80%    744
  90%    942
  95%   1140
  98%   4759
  99%   4759
 100%   4759 (longest request)

changed milestone to %10.1

Moving this to 10.1, since we missed 10.0 freeze.

mentioned in issue gitlab-foss#38568

changed milestone to %10.1

moved from gitlab-foss#33643 (moved)