Broken liveness probe
🐞 Bug report
Since one of the last updates it seems that liveness probe is broken. Kubernetes will kill the worker pod when Can't find the alive key
occurres in the log. The dep-update job is taking a while (due to a core issue with gomod). The web pod is working well.
Is there an existing issue for this?
Please search existing issues to avoid creating duplicates
-
I have searched the existing issues
App version
Using 0.32.0/0.33.0
Broken at least since 0.31.0 (I don't know the last working version anymore)
Execution mode
deployed via helm chart
Package ecosystem
not applicable
Package manager version
not applicable
Language version
not applicable
dependabot.yml
not applicable
Updated dependency
not applicable
Expected outcome
The liveness probe should work as intended even the dep-update job is taking a while.
Native package manager behaviour
not applicable
Log output
[2023-01-16 15:38:47 +0000 tid=4205] INFO -- start
[2023-01-16 15:38:49 +0000 tid=4205] INFO -- [dep-update: xxx=>maven] Fetching info for org.apache.maven.doxia:doxia-module-markdown
[2023-01-16 15:38:49 +0000 tid=4205] INFO -- [dep-update: xxx=>maven] updating org.apache.maven.doxia:doxia-module-markdown: 1.11.1 => 1.12.0
[2023-01-16 15:38:49 +0000 tid=423h] INFO -- [dep-update: xxx=>maven] Fetching info for xxx
[2023-01-16 15:38:50 +0000 tid=423h] INFO -- [dep-update: xxx=>maven] xxx: 4.10.1 is up to date
[2023-01-16 15:38:50 +0000 tid=422d] INFO -- [dep-update: xxx=>gomod] Fetching info for github.com/bwmarrin/go-objectsid
[2023-01-16 15:38:50 +0000 tid=422d] INFO -- [dep-update: xxx=>gomod] github.com/bwmarrin/go-objectsid: 0.0.0-20191126144531-5fee401a2f37 is up to date
[2023-01-16 15:38:50 +0000 tid=423h] INFO -- [dep-update: xxx=>maven] Fetching info for xxx
[2023-01-16 15:38:50 +0000 tid=423h] INFO -- [dep-update: xxx=>maven] xxx: 0.19.4-SNAPSHOT is up to date
[2023-01-16 15:38:50 +0000 tid=422d] INFO -- [dep-update: xxx=>gomod] Fetching info for github.com/go-ldap/ldap/v3
[2023-01-16 15:38:51 +0000 tid=422d] INFO -- [dep-update: xxx=>gomod] updating github.com/go-ldap/ldap/v3: 3.4.1 => 3.4.4
[2023-01-16 15:38:51 +0000 tid=4205] INFO -- [dep-update: xxx=>maven] merge request https://gitlab.xxx/xxx/-/merge_requests/117 doesn't require updating
[2023-01-16 15:40:11 +0000 tid=41yn] ERROR -- Can't find the alive key
[2023-01-16 15:42:11 +0000 tid=422b] ERROR -- Can't find the alive key
[2023-01-16 15:42:11 +0000 tid=19nd] INFO -- Received TSTP, no longer accepting new work
[2023-01-16 15:42:11 +0000 tid=43ld] INFO -- Scheduler exiting...
[2023-01-16 15:42:11 +0000 tid=19nd] INFO -- Terminating quiet threads
[2023-01-16 15:42:11 +0000 tid=19nd] INFO -- Shutting down sidekiq-alive!
[2023-01-16 15:42:11 +0000 tid=19nd] INFO -- [SidekiqAlive] Purging 0 pending for dependabot-dependabot-gitlab-worker-565d577f9d-bkzgw
[2023-01-16 15:42:11 +0000 tid=19nd] INFO -- [SidekiqAlive] Removing queue healthcheck-dependabot-dependabot-gitlab-worker-565d577f9d-bkzgw
[2023-01-16 15:42:11 +0000 tid=43lx] INFO -- Scheduler exiting...
[2023-01-16 15:42:11 +0000 tid=19nd] INFO -- Shutting down
[2023-01-16 15:42:11 +0000 tid=19nd] INFO -- going to shutdown ...
[2023-01-16 15:42:11 +0000 tid=19nd] INFO -- WEBrick::HTTPServer#start done.
[2023-01-16 15:42:11 +0000 tid=19nd] INFO -- Shutting down sidekiq-alive!
[2023-01-16 15:42:11 +0000 tid=19nd] INFO -- [SidekiqAlive] Purging 0 pending for dependabot-dependabot-gitlab-worker-565d577f9d-bkzgw
[2023-01-16 15:42:11 +0000 tid=19nd] INFO -- [SidekiqAlive] Removing queue healthcheck-dependabot-dependabot-gitlab-worker-565d577f9d-bkzgw
[2023-01-16 15:42:12 +0000 tid=19nd] INFO -- Pausing to allow jobs to finish...
[2023-01-16 15:42:36 +0000 tid=19nd] WARN -- Terminating 10 busy threads
[2023-01-16 15:42:36 +0000 tid=19nd] INFO -- Pushed 10 jobs back to Redis
#<Thread:0x00007ffa48075090 /usr/local/lib/ruby/3.1.0/open3.rb:296 run> terminated with exception (report_on_exception is true):
/usr/local/lib/ruby/3.1.0/open3.rb:296:in `read': stream closed in another thread (IOError)
from /usr/local/lib/ruby/3.1.0/open3.rb:296:in `block (2 levels) in capture3'
#<Thread:0x00007ffa48074d20 /usr/local/lib/ruby/3.1.0/open3.rb:297 run> terminated with exception (report_on_exception is true):
/usr/local/lib/ruby/3.1.0/open3.rb:297:in `read': stream closed in another thread (IOError)
from /usr/local/lib/ruby/3.1.0/open3.rb:297:in `block (2 levels) in capture3'
[2023-01-16 15:42:36 +0000 tid=4255] INFO -- fail
[2023-01-16 15:42:36 +0000 tid=426t] INFO -- fail
[2023-01-16 15:42:36 +0000 tid=427d] INFO -- fail
[2023-01-16 15:42:36 +0000 tid=422x] INFO -- fail
[2023-01-16 15:42:36 +0000 tid=421t] INFO -- fail
[2023-01-16 15:42:36 +0000 tid=424l] INFO -- fail
[2023-01-16 15:42:36 +0000 tid=4241] INFO -- fail
[2023-01-16 15:42:36 +0000 tid=4205] INFO -- fail
[2023-01-16 15:42:36 +0000 tid=423h] INFO -- fail
[2023-01-16 15:42:38 +0000 tid=19nd] INFO -- Bye!
Smallest manifest that reproduces the issue
not applicable
Edited by George Koltsov