Logging improvements
What does this MR do and why?
Describe in detail what your merge request does and why.
Aim of this change is to have a readable, not too verbose log in the job outputs with debug log still present in jobs artifacts when required for actual debugging.
- Add broadcast logger and stream both to stdout and file
- Always use debug level for file logger
- Stream docker command outputs through logger
- Add option to explicitly set log level instead of plain toggle between debug and info (with backwards compatible QA_DEBUG option usage for now)
This allows to reduce extreme logging verbosity to stdout which always exceeds our runner limits and forces to look at logs directly in GCP and at the same time keeps the verbose debug log as artifact in cases when it is actually needed.
Part of: https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/1201
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.