CI breaks if Prometheus metrics are not working
It looks like if Prometheus metrics are broken, it also breaks CI pipelines. It took me awhile to figure out what is what was happening, but I was getting a large number of pipelines being created for the same action. (Say a commit) These pipelines were stuck in the created
status, and would never be processed. Over time, the number would continue to grow. If I manually create a pipeline however, it would run fine.
I'm not sure if this is what is truly causing it, but after disabling Prometheus metrics this behavior no longer occurred.
This is on the GDK 9.3 RC3 branch 9-3-stable-rc3
, which has been tagged:
22:43:17 rails-background-jobs.1 | 2017-06-19T02:43:17.497Z 52858 TID-ovbj9d1r0 WARN: {"context":"Job raised exception","job":{"class":"PostReceive","args":["/Users/joshua/projects/gdk-ce/repositories/root/metricstest.git","user-1","ZGMyNThmYzZlNTJkMWUyY2VhZTRkZjAzMzBhODNlOWVjNTQwZTI1NyA0NTRi\nZGFiZDhhNTdkNWRjYmM4ZmVjODE5NjMxNjRlYWQ0ZGI2MmQ4IHJlZnMvaGVh\nZHMvbWFzdGVyCg==\n"],"jid":"1357bc18ca3c7dc65fdc9046","enqueued_at":1497840193.678171,"retry":3,"queue":"post_receive","error_message":"No such file or directory @ rb_sysopen - /var/run/gitlab/unicorn/counter_52858.db","error_class":"Errno::ENOENT","failed_at":1497840111.9471428,"retry_count":2,"created_at":1497840164.1908731,"retried_at":1497840197.4961221},"jobstr":"{\"class\":\"PostReceive\",\"args\":[\"/Users/joshua/projects/gdk-ce/repositories/root/metricstest.git\",\"user-1\",\"ZGMyNThmYzZlNTJkMWUyY2VhZTRkZjAzMzBhODNlOWVjNTQwZTI1NyA0NTRi\\nZGFiZDhhNTdkNWRjYmM4ZmVjODE5NjMxNjRlYWQ0ZGI2MmQ4IHJlZnMvaGVh\\nZHMvbWFzdGVyCg==\\n\"],\"jid\":\"1357bc18ca3c7dc65fdc9046\",\"enqueued_at\":1497840193.678171,\"retry\":3,\"queue\":\"post_receive\",\"error_message\":\"No such file or directory @ rb_sysopen - /var/run/gitlab/unicorn/counter_52858.db\",\"error_class\":\"Errno::ENOENT\",\"failed_at\":1497840111.9471428,\"retry_count\":1,\"created_at\":1497840164.1908731,\"retried_at\":1497840164.520341}"}
22:43:17 rails-background-jobs.1 | 2017-06-19T02:43:17.497Z 52858 TID-ovbj9d1r0 WARN: Errno::ENOENT: No such file or directory @ rb_sysopen - /var/run/gitlab/unicorn/counter_52858.db
22:43:17 rails-background-jobs.1 | 2017-06-19T02:43:17.497Z 52858 TID-ovbj9d1r0 WARN: /Users/joshua/.rvm/gems/ruby-2.3.3/gems/prometheus-client-mmap-0.7.0.beta5/lib/prometheus/client/valuetype.rb:119:in `initialize'
Edited by Joshua Lambert