Disable concurrency for shell builds by default?

It seems that gitlab-ci-multi-runner tries to run two shell builds in parallel by default. For some builds this leads to very strange behavior. Maybe it would be better to disable concurrency by default?

cc @ayufan @marin @jnijhof

Yes. It depends on number of CPU cores. I would prefer to have it working the way it's now. It may not work in cases when all builds use the same database and other shared resources.

The old Ruby runner had no concurrency by default, we are in the process of diagnosing broken builds due to concurrency. I think that designing CI jobs for concurrency is hard. Why force people to fight with this?

I think it should be disabled by default.

There is no good number (2,3,4,etc.) if you want concurrency. Many people will not want concurrency. Less advanced users are more likely to not want concurrency.

Added improvement label

I am speaking from the perspective of having used the 'old' gitlab-ci-runner in the past, which did not do concurrent builds on the same host by default. The default 'one build per core' concurrency in gitlab-ci-multi-runner proved to be a source of trouble for the CI infrastructure used on dev.gitlab.org and I can imagine it can also hurt others who are migrating from the old Ruby runner to gitlab-ci-multi-runner.

I agree with Jacob. User should be able to determine when their infrastructure is safe enough to switch to concurrency.

@jacobvosmaer @marin Your explanation is good enough for me. Let's do it. The shell and ssh executors will have default limit set to 1.

We can think about preparing good tutorials for use with Docker to have full concurrency.

Thanks for feedback.

@ayufan Thanks! (BTW I think docker should also default to 1, otherwise it is not consistent, but good idea to make a good tutorial)

@sytses I think that it is, because Docker by default supports well concurrency. This is quite different from the shell. We should reconsider something different: Some time ago I changed default value of concurrency to number of CPUs, maybe this should be set back to 1?

@ayufan I propose to set it back to 1 for all cases, including Docker.

@sytses @marin @jacobvosmaer default concurrent limit for new configs is set to 1, default limit for all runners during registration is set to 1 with easy switch to enlarge it: gitlab-ci-multi-runner register --limit 2

It doesn't change current installations, only new ones.

Commits: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/commit/6c17d0af9bd7397d058808eaf9a674e47240f3b1 https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/commit/89afdea7454a84685a0a8449abb9c1ff67183988

@jacobvosmaer Please verify and close it.

Milestone changed to v0.4.0

@ayufan Thanks!

Reassigned to @ayufan

Status changed to closed