Add a way to change some default settings.
CheckInterval, UpdateInterval and MaxTraceOutputSize can now be configured in the config.toml (multi runner) or on the command line (single runner)
Merge request reports
Closed by
(Mar 14, 2025 1:40pm UTC)
Activity
mentioned in issue #23 (closed)
@PuckCh what is the satisfying MaxTraceOutputSize? Maybe we can make all these parameters auto-tune?
It's a huge C++ project with many dependencies. Actually, these are the sizes of the logs of the overnight builds, where we build everything from scratch. For the continuous build we build the dependencies only after version change, otherwise we pick them up from a location where they had been built previously. So the log is usually much smaller.
I agree that most people do not need such a high limit.
Yes. I'm a little against allowing to change build log size limit. The main problem is with the performance. Such build logs makes big load on database, takes long time to transform with ruby and render with the browser - in the end it makes the UX bad. What is more important that such long logs are hard to read and doesn't present any practical value IMHO.
I think that we should rise the limit from 1MB to something more reasonable: maybe 4MB? but I'm little against adding additional configuration option.
The problem is not only the log size. The actual code (without this patch) cause the runner to hang if we reach the log limit, because of the break of the for loop reading the runner output and writing the log output. I'm not against keeping the 1M log size, but you should at least correct the bug with the runner hanging forever if the log limit is reached. (which is in the 2nd commit of this merge request).
@PuckCh I did check. On what system did you see hanging? In all my tests (with limit set to 1024 bytes) it behaved not consistently, but didn't hang.
I think that problem here is that once you have pretty large build log it takes a long way to send it to coordinator, because
GitLab CIcurrently doesn't support incremental uploads and most likely it ate all your upload.Hi,
I'm testing on Mac OS X (10.10.3) and Debian GNU/Linux 8.0. Both have the same problem. The problem is with the pipe buffer size. Using the script at http://unix.stackexchange.com/questions/11946/how-big-is-the-pipe-buffer I have a buffer size of 65536 on my two test systems (Mac OS X and Linux).
Here is a script sample you can put in your CI job script:
i=1 while [ 1 ] do echo $i echo $i > tmp_output.tmp ((i++)) doneThis fake test script just output a counter to the log, and save it in a disk file. If we script continue to run, we can assume the output log will end with:
Build log exceed limit of 1048576 bytes.and the tmp_output.tmp file will continue to be incremented. But because of the maximum size of the pipe buffer, the tmp_output.tmp file will stop to be incremented. Here is my log on the Mac OS X (log on the Linux is the same kind):
... 165620 165621 165622 165623 165624 165625 165626 165627 165628 165629 165630 16 Build log exceed limit of 1048576 bytes.and tmp_output.tmp is:
174994even after 30 minutes of waiting. A small estimation:
In [2]: (174994-165630) * 7 Out[2]: 65548confirm to me that the stop occur when the pipe output buffer is full.
So, the patch don't break the for loop. You must continue to read from the pipe, even if you don't write out the read values.
Can you reproduce the errors with these informations? I suspect you don't see this error with your previous tests because the size of your output job is lower than the 1M + the size of your pipe buffer.
Thanks for your attention.
Yes, the main problem is the hanging. At the moment I cannot use multi-runner, simply because it does not complete its job. Increasing the output limit to 4MB would not make any difference for me, as the output is longer on each platform.
I would still prefer to have a configuration option for this. I agree that the large output has performance issues, it can make the build page very slow. But we cannot reduce the build output from over 5MB to less than 1MB. I think, we can assume that people who change a configuration option, are doing this for good reason. There could be a note in the configuration file about consequences on the performance.
So, if the bug about the hanging is resolved, what would happen with the output over the 1MB limit? Would it simply be swallowed? It would make it hard to discover why a build fail. Would it be possible to keep the last 1MB of the output? Would it require API change in the coordinator? Does this make sense?
Will you accept the merge request which makes the output size configurable? (Putting it on the users responsibility, of course.)
Do you think it would be a good compromise on usability and performance to show the last 1MB of the output? Usually that part contains the error. Would you accept a merge request for that?
@PuckCh fix for pipe was merged (with small changes): https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/commit/683c14ca63077f945b68ecfd182105fbf40e90ec
It's available on bleeding edge release
