Skip to content

After upgrade to 9.4.1 path "/-/metrics" returns error 404 and it is written to the log file

Summary

After upgrade to 9.4.1 path "/-/metrics" returns error 404. And I can see it also in the log. Other monitoring paths "/-/liveness" and "/-/readiness" work properly.

Steps to reproduce

  1. Update your omnibus gitlab instance from 9.3.8 to 9.4.1
  2. Call curl https://gitlab.example.com/-/metrics
  3. You get:
<!DOCTYPE html>
<html>
<head>
  <meta content="width=device-width, initial-scale=1, maximum-scale=1" name="viewport">
  <title>The page you're looking for could not be found (404)</title>
  <style>
...

What is the current bug behavior?

You get 404 error. And in log you can see from time to time:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/-/metrics" for 127.0.0.1 at 2017-07-27 14:27:43 +0200
Processing by MetricsController#index as HTML
Filter chain halted as :validate_ip_whitelisted_or_valid_token! rendered or redirected
Completed 404 Not Found in 2ms (Views: 0.8ms | ActiveRecord: 0.0ms)
Scoped order and limit are ignored, it's forced to be batch order and batch size

What is the expected correct behavior?

Some stats of this gitlab instance. And there should be no error message in log.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System: 
Current User:   git
Using RVM:      no
Ruby Version:   2.3.3p222
Gem Version:    2.6.6
Bundler Version:1.13.7
Rake Version:   10.5.0
Redis Version:  3.2.5
Git Version:    2.13.0
Sidekiq Version:5.0.0
Go Version:     unknown

GitLab information
Version:        9.4.1
Revision:       6678b4e
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://gitlab.example.com
HTTP Clone URL: https://gitlab.example.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.example.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: no


GitLab Shell
Version:        5.3.1
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab Shell ...


GitLab Shell version >= 5.3.1 ? ... OK (5.3.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
7/3 ... repository is empty
2/4 ... ok
18/6 ... ok
18/7 ... ok
18/8 ... ok
6/10 ... ok
6/11 ... ok
6/12 ... ok
6/13 ... ok
6/14 ... ok
6/15 ... ok
6/16 ... ok
45/17 ... ok
6/18 ... ok
47/19 ... ok
2/20 ... ok
49/21 ... ok
49/22 ... ok
49/23 ... repository is empty
49/24 ... repository is empty
49/25 ... ok
8/26 ... ok
49/27 ... ok
49/28 ... ok
56/30 ... ok
56/31 ... ok
49/32 ... ok
50/33 ... ok
50/34 ... ok
58/35 ... ok
58/36 ... ok
50/37 ... ok
50/38 ... ok
51/39 ... ok
51/40 ... ok
51/41 ... ok
51/42 ... ok
51/43 ... ok
49/44 ... repository is empty
8/45 ... ok
6/46 ... ok
62/47 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Reply by email ...


Reply by email is disabled in config/gitlab.yml


Checking Reply by email ... Finished


Checking LDAP ...


LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
7/3 ... yes
2/4 ... yes
18/6 ... yes
18/7 ... yes
18/8 ... yes
6/10 ... yes
6/11 ... yes
6/12 ... yes
6/13 ... yes
6/14 ... yes
6/15 ... yes
6/16 ... yes
45/17 ... yes
6/18 ... yes
47/19 ... yes
2/20 ... yes
49/21 ... yes
49/22 ... yes
49/23 ... yes
49/24 ... yes
49/25 ... yes
8/26 ... yes
49/27 ... yes
49/28 ... yes
56/30 ... yes
56/31 ... yes
49/32 ... yes
50/33 ... yes
50/34 ... yes
58/35 ... yes
58/36 ... yes
50/37 ... yes
50/38 ... yes
51/39 ... yes
51/40 ... yes
51/41 ... yes
51/42 ... yes
51/43 ... yes
49/44 ... yes
8/45 ... yes
6/46 ... yes
62/47 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.3 ? ... yes (2.3.3)
Git version >= 2.7.3 ? ... yes (2.13.0)
Active users: ... 49


Checking GitLab ... Finished

Possible fixes

I can see that the help page about health check https://gitlab.com/help/user/admin_area/monitoring/health_check mentions only "/-/readiness" and "/-/liveness". My monitoring page "https://gitlab.example.com/admin/health_check" mentions also "/-/metrics". I monitor my gitlab instance using https://gitlab.example.com/health_check . Maybe it is only some forgotten code that checks the outdated "/-/metrics" path?