Skip to content

Fix bug in config for prometheus_listen_addr

James Nutt requested to merge jmd-fix-praefect-prometheus-listen-port into master

What does this MR do and why?

https://gitlab.com/gitlab-org/gitlab-qa/-/commit/fc0f8be5fef99a3d0e0e3c1eaade74705e281149#ebab9b4d4194f0ed83c8c7d98e8c51e0e4b8d5bc_175_194 erroneously updated the prometheus['scrape_configs'] which led to the prometheus endpoint not collecting metrics from the praefect node.

This will fix E2E https://gitlab.com/gitlab-org/gitlab/-/issues/378174 which broke as the metrics exposed by prometheus on the praefect node contain data required for this test.

How to set up and validate locally

  1. Start the environment
    bundle exec exe/gitlab-qa Test::Integration::GitalyCluster EE --no-tests

  2. Navigate to the environment, and create a repository, commits etc to generate some traffic to populate metrics.

    • docker ps to identify what port GitLab is available on e.g. 0.0.0.0:56787->80/tcp gitlab-gitaly-cluster
    • In this case navigate to localhost:56787 to access GitLab

  3. identify some test metrics exposed by the praefect node
    docker exec -ti praefect bash -c "curl -s http://localhost:9652/metrics"

    • take note of some of the gitaly_praefect... metrics e.g. gitaly_praefect_read_distribution gitaly_praefect_subtransactions_per_transaction_total_bucket

  4. verify the metrics are available docker exec gitlab-gitaly-cluster bash -c 'curl -s http://localhost:9090/api/v1/query?query=gitaly_praefect_read_distribution'

    • The response should look something approximately like this, (ensure the result array is not empty)
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"gitaly_praefect_read_distribution","instance":"praefect.test:9652","job":"praefect","storage":"gitaly1","virtual_storage":"default"},"value":[1669012745.52,"165"]},{"metric":{"__name__":"gitaly_praefect_read_distribution","instance":"praefect.test:9652","job":"praefect","storage":"gitaly2","virtual_storage":"default"},"value":[1669012745.52,"45"]},{"metric":{"__name__":"gitaly_praefect_read_distribution","instance":"praefect.test:9652","job":"praefect","storage":"gitaly3","virtual_storage":"default"},"value":[1669012745.52,"187"]}]}}%

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by James Nutt

Merge request reports