Retrying a pipeline that has failed more than once duplicates all jobs
Summary
Retrying a pipeline that has already failed more than once through the Pipelines -> Pipelines project section retries all individual failed jobs. This leads to exponential growth of the number of jobs executed as long as the pipeline jobs still fail.
Steps to reproduce
- Have a project with CI/CD where you can induce error on pipeline execution (e.g. by removing a vital tool from a shell-based runner's environment).
- Set the runner's environment to fail and run a pipeline from Pipelines -> Pipelines project section. It should result in a failure.
- Retry the pipeline using the button on the right. It fails again, as expected.
- Retry the pipeline the second time. Again it fails but it has now duplicated all previously failed jobs.
What is the current bug behavior?
Repeatedly retrying a failed pipeline results in exponential growth of queued jobs.
What is the expected correct behavior?
Failed jobs of a same stage should not be repeated.
Relevant logs and/or screenshots
Retrying a twice failed pipeline of two stages:
Results in four jobs being queued (in red, original execution in blue, first retry in green):
Output of checks
Results of GitLab environment info
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.3.3p222
Gem Version: 2.6.6
Bundler Version:1.13.7
Rake Version: 10.5.0
Redis Version: 3.2.5
Git Version: 2.10.2
Sidekiq Version:4.2.7
GitLab information
Version: 9.0.0
Revision: cf020e6
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: http://REMOVED
HTTP Clone URL: http://REMOVED/some-group/some-project.git
SSH Clone URL: git@REMOVED:some-group/some-project.git
Using LDAP: yes
Using Omniauth: no
GitLab Shell
Version: 5.0.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Checking GitLab Shell ...
GitLab Shell version >= 5.0.0 ? ... OK (5.0.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
3/1 ... ok
7/3 ... ok
3/4 ... ok
3/6 ... ok
3/9 ... ok
3/10 ... ok
4/11 ... ok
18/12 ... ok
2/21 ... ok
47/28 ... ok
3/29 ... ok
106/44 ... ok
106/45 ... ok
106/48 ... ok
106/49 ... ok
61/118 ... ok
67/120 ... ok
41/126 ... ok
69/127 ... ok
26/128 ... ok
68/151 ... repository is empty
69/152 ... ok
69/153 ... ok
69/154 ... ok
3/156 ... ok
3/157 ... ok
3/159 ... ok
106/160 ... ok
67/162 ... ok
85/163 ... ok
3/164 ... ok
3/165 ... ok
73/166 ... ok
95/169 ... ok
95/170 ... ok
3/171 ... ok
47/172 ... ok
102/173 ... repository is empty
94/174 ... ok
68/175 ... ok
106/176 ... ok
106/177 ... ok
95/179 ... ok
95/180 ... ok
95/181 ... ok
18/182 ... ok
110/184 ... ok
110/185 ... ok
110/186 ... ok
110/187 ... ok
110/188 ... ok
113/189 ... ok
113/190 ... ok
113/191 ... ok
113/192 ... ok
113/193 ... ok
113/194 ... ok
113/195 ... ok
114/197 ... ok
114/198 ... ok
114/199 ... ok
114/200 ... ok
114/201 ... ok
114/202 ... ok
114/203 ... ok
114/204 ... ok
114/205 ... ok
114/206 ... ok
114/207 ... ok
114/208 ... ok
114/209 ... ok
114/210 ... ok
114/211 ... ok
115/212 ... ok
115/213 ... ok
115/214 ... ok
115/215 ... ok
116/216 ... ok
116/217 ... ok
116/218 ... ok
116/219 ... ok
116/220 ... ok
116/221 ... ok
116/222 ... ok
116/223 ... ok
116/224 ... ok
116/225 ... ok
116/226 ... ok
116/227 ... ok
116/228 ... ok
116/229 ... ok
116/230 ... ok
116/231 ... ok
117/232 ... ok
117/233 ... ok
117/234 ... ok
117/235 ... ok
117/236 ... ok
117/237 ... ok
117/238 ... ok
117/239 ... ok
117/240 ... ok
117/241 ... ok
117/242 ... ok
117/243 ... ok
117/244 ... ok
117/245 ... ok
118/246 ... ok
118/247 ... ok
118/248 ... ok
118/249 ... ok
119/250 ... ok
119/251 ... ok
119/252 ... ok
119/253 ... ok
119/254 ... ok
119/255 ... ok
119/256 ... ok
119/257 ... ok
106/258 ... ok
120/259 ... ok
120/260 ... ok
120/261 ... ok
120/262 ... ok
120/263 ... ok
121/264 ... ok
121/265 ... ok
121/266 ... ok
122/267 ... ok
124/268 ... ok
125/269 ... ok
119/270 ... ok
123/271 ... ok
113/272 ... ok
123/273 ... ok
116/274 ... ok
123/275 ... ok
123/276 ... ok
123/277 ... ok
123/278 ... ok
114/280 ... ok
123/281 ... ok
122/284 ... repository is empty
122/285 ... repository is empty
137/286 ... ok
137/288 ... ok
140/291 ... ok
113/292 ... ok
116/293 ... ok
145/305 ... repository is empty
145/306 ... ok
145/307 ... ok
145/308 ... ok
3/309 ... ok
140/310 ... ok
140/311 ... ok
147/313 ... ok
106/315 ... ok
106/316 ... ok
149/317 ... ok
106/318 ... ok
114/319 ... ok
85/321 ... ok
134/325 ... ok
3/327 ... ok
95/328 ... ok
145/329 ... ok
4/330 ... ok
2/331 ... ok
208/332 ... ok
52/334 ... ok
78/335 ... ok
94/336 ... ok
94/337 ... ok
94/338 ... ok
3/339 ... ok
94/340 ... ok
94/341 ... ok
149/345 ... ok
67/347 ... ok
3/348 ... ok
3/349 ... ok
3/350 ... ok
3/351 ... ok
69/352 ... ok
2/353 ... ok
149/354 ... ok
149/355 ... ok
3/356 ... ok
186/357 ... repository is empty
3/358 ... repository is empty
49/359 ... ok
49/360 ... ok
122/361 ... ok
117/362 ... ok
3/364 ... ok
18/366 ... ok
33/367 ... repository is empty
131/369 ... ok
178/370 ... ok
106/371 ... ok
230/372 ... ok
92/373 ... ok
3/374 ... ok
4/375 ... repository is empty
69/376 ... ok
3/377 ... ok
214/378 ... ok
185/379 ... ok
205/380 ... ok
213/382 ... ok
213/384 ... ok
214/385 ... ok
120/386 ... ok
69/387 ... ok
69/388 ... ok
69/389 ... ok
218/390 ... ok
218/391 ... ok
218/392 ... ok
218/393 ... ok
218/394 ... ok
219/395 ... ok
219/396 ... ok
219/397 ... ok
213/400 ... ok
121/401 ... ok
145/402 ... ok
85/404 ... ok
149/405 ... ok
115/406 ... ok
115/407 ... ok
95/408 ... ok
149/409 ... ok
43/410 ... ok
225/414 ... ok
149/415 ... ok
61/416 ... ok
210/417 ... repository is empty
167/418 ... ok
42/419 ... ok
94/420 ... ok
94/421 ... ok
230/422 ... ok
115/423 ... ok
232/426 ... ok
230/427 ... ok
233/428 ... ok
114/429 ... ok
232/430 ... ok
232/431 ... ok
230/432 ... ok
117/433 ... ok
116/434 ... ok
234/435 ... repository is empty
236/436 ... ok
212/437 ... ok
238/438 ... ok
238/439 ... ok
94/440 ... ok
94/441 ... ok
61/442 ... ok
67/443 ... ok
214/445 ... ok
197/446 ... ok
228/448 ... ok
14/449 ... ok
3/452 ... ok
3/453 ... ok
178/455 ... ok
90/456 ... ok
90/457 ... ok
90/458 ... ok
90/461 ... ok
145/462 ... ok
145/463 ... ok
90/464 ... ok
90/465 ... ok
90/466 ... ok
244/467 ... ok
214/468 ... ok
90/469 ... ok
14/470 ... ok
234/471 ... repository is empty
232/472 ... ok
85/473 ... repository is empty
214/474 ... ok
248/476 ... repository is empty
249/477 ... repository is empty
249/478 ... ok
251/479 ... repository is empty
110/481 ... ok
232/482 ... ok
255/484 ... repository is empty
114/485 ... ok
232/486 ... ok
232/487 ... ok
26/488 ... ok
145/490 ... ok
228/491 ... ok
3/492 ... repository is empty
236/493 ... ok
235/494 ... ok
38/495 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
3/1 ... yes
7/3 ... yes
3/4 ... yes
3/6 ... yes
3/9 ... yes
3/10 ... yes
4/11 ... yes
18/12 ... yes
2/21 ... yes
47/28 ... yes
3/29 ... yes
106/44 ... yes
106/45 ... yes
106/48 ... yes
106/49 ... yes
61/118 ... yes
67/120 ... yes
41/126 ... yes
69/127 ... yes
26/128 ... yes
68/151 ... yes
69/152 ... yes
69/153 ... yes
69/154 ... yes
3/156 ... yes
3/157 ... yes
3/159 ... yes
106/160 ... yes
67/162 ... yes
85/163 ... yes
3/164 ... yes
3/165 ... yes
73/166 ... yes
95/169 ... yes
95/170 ... yes
3/171 ... yes
47/172 ... yes
102/173 ... yes
94/174 ... yes
68/175 ... yes
106/176 ... yes
106/177 ... yes
95/179 ... yes
95/180 ... yes
95/181 ... yes
18/182 ... yes
110/184 ... yes
110/185 ... yes
110/186 ... yes
110/187 ... yes
110/188 ... yes
113/189 ... yes
113/190 ... yes
113/191 ... yes
113/192 ... yes
113/193 ... yes
113/194 ... yes
113/195 ... yes
114/197 ... yes
114/198 ... yes
114/199 ... yes
114/200 ... yes
114/201 ... yes
114/202 ... yes
114/203 ... yes
114/204 ... yes
114/205 ... yes
114/206 ... yes
114/207 ... yes
114/208 ... yes
114/209 ... yes
114/210 ... yes
114/211 ... yes
115/212 ... yes
115/213 ... yes
115/214 ... yes
115/215 ... yes
116/216 ... yes
116/217 ... yes
116/218 ... yes
116/219 ... yes
116/220 ... yes
116/221 ... yes
116/222 ... yes
116/223 ... yes
116/224 ... yes
116/225 ... yes
116/226 ... yes
116/227 ... yes
116/228 ... yes
116/229 ... yes
116/230 ... yes
116/231 ... yes
117/232 ... yes
117/233 ... yes
117/234 ... yes
117/235 ... yes
117/236 ... yes
117/237 ... yes
117/238 ... yes
117/239 ... yes
117/240 ... yes
117/241 ... yes
117/242 ... yes
117/243 ... yes
117/244 ... yes
117/245 ... yes
118/246 ... yes
118/247 ... yes
118/248 ... yes
118/249 ... yes
119/250 ... yes
119/251 ... yes
119/252 ... yes
119/253 ... yes
119/254 ... yes
119/255 ... yes
119/256 ... yes
119/257 ... yes
106/258 ... yes
120/259 ... yes
120/260 ... yes
120/261 ... yes
120/262 ... yes
120/263 ... yes
121/264 ... yes
121/265 ... yes
121/266 ... yes
122/267 ... yes
124/268 ... yes
125/269 ... yes
119/270 ... yes
123/271 ... yes
113/272 ... yes
123/273 ... yes
116/274 ... yes
123/275 ... yes
123/276 ... yes
123/277 ... yes
123/278 ... yes
114/280 ... yes
123/281 ... yes
122/284 ... yes
122/285 ... yes
137/286 ... yes
137/288 ... yes
140/291 ... yes
113/292 ... yes
116/293 ... yes
145/305 ... yes
145/306 ... yes
145/307 ... yes
145/308 ... yes
3/309 ... yes
140/310 ... yes
140/311 ... yes
147/313 ... yes
106/315 ... yes
106/316 ... yes
149/317 ... yes
106/318 ... yes
114/319 ... yes
85/321 ... yes
134/325 ... yes
3/327 ... yes
95/328 ... yes
145/329 ... yes
4/330 ... yes
2/331 ... yes
208/332 ... yes
52/334 ... yes
78/335 ... yes
94/336 ... yes
94/337 ... yes
94/338 ... yes
3/339 ... yes
94/340 ... yes
94/341 ... yes
149/345 ... yes
67/347 ... yes
3/348 ... yes
3/349 ... yes
3/350 ... yes
3/351 ... yes
69/352 ... yes
2/353 ... yes
149/354 ... yes
149/355 ... yes
3/356 ... yes
186/357 ... yes
3/358 ... yes
49/359 ... yes
49/360 ... yes
122/361 ... yes
117/362 ... yes
3/364 ... yes
18/366 ... yes
33/367 ... yes
131/369 ... yes
178/370 ... yes
106/371 ... yes
230/372 ... yes
92/373 ... yes
3/374 ... yes
4/375 ... yes
69/376 ... yes
3/377 ... yes
214/378 ... yes
185/379 ... yes
205/380 ... yes
213/382 ... yes
213/384 ... yes
214/385 ... yes
120/386 ... yes
69/387 ... yes
69/388 ... yes
69/389 ... yes
218/390 ... yes
218/391 ... yes
218/392 ... yes
218/393 ... yes
218/394 ... yes
219/395 ... yes
219/396 ... yes
219/397 ... yes
213/400 ... yes
121/401 ... yes
145/402 ... yes
85/404 ... yes
149/405 ... yes
115/406 ... yes
115/407 ... yes
95/408 ... yes
149/409 ... yes
43/410 ... yes
225/414 ... yes
149/415 ... yes
61/416 ... yes
210/417 ... yes
167/418 ... yes
42/419 ... yes
94/420 ... yes
94/421 ... yes
230/422 ... yes
115/423 ... yes
232/426 ... yes
230/427 ... yes
233/428 ... yes
114/429 ... yes
232/430 ... yes
232/431 ... yes
230/432 ... yes
117/433 ... yes
116/434 ... yes
234/435 ... yes
236/436 ... yes
212/437 ... yes
238/438 ... yes
238/439 ... yes
94/440 ... yes
94/441 ... yes
61/442 ... yes
67/443 ... yes
214/445 ... yes
197/446 ... yes
228/448 ... yes
14/449 ... yes
3/452 ... yes
3/453 ... yes
178/455 ... yes
90/456 ... yes
90/457 ... yes
90/458 ... yes
90/461 ... yes
145/462 ... yes
145/463 ... yes
90/464 ... yes
90/465 ... yes
90/466 ... yes
244/467 ... yes
214/468 ... yes
90/469 ... yes
14/470 ... yes
234/471 ... yes
232/472 ... yes
85/473 ... yes
214/474 ... yes
248/476 ... yes
249/477 ... yes
249/478 ... yes
251/479 ... yes
110/481 ... yes
232/482 ... yes
255/484 ... yes
114/485 ... yes
232/486 ... yes
232/487 ... yes
26/488 ... yes
145/490 ... yes
228/491 ... yes
3/492 ... yes
236/493 ... yes
235/494 ... yes
38/495 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.10.2)
Active users: 202
Checking GitLab ... Finished
Possible fixes
N/A