karma failures on master #18524895 #18524184

Cannot start ChromeHeadless: https://gitlab.com/gitlab-org/gitlab-ce/builds/18524895

Disconnected, because no message in 10000 ms: https://gitlab.com/gitlab-org/gitlab-ce/builds/18524184

I'm 99% sure the "disconnected" error existed before the use of chrome, so is probably not related to the binary used.

/cc @mikegreiling

@mikegreiling @lbennett https://gitlab.com/gitlab-org/gitlab-ce/builds/18547666 not sure if this is helpful, but as requested in the team call here's a link that shows some problems

I'm seeing this quite a bit in master

Going to dig into the karma-chrome-launcher source and see if I can find some clues

Judging by a quick sample of recent commits to master, it looks like this "Cannot start ChromeHeadless" issue is occurring on about 1/3rd of the pipelines. I think we're going to need to revert !12036 (merged) and do some further research.

mentioned in merge request !12137 (merged)

assigned to @mikegreiling

mentioned in merge request !12144 (merged)

here's a snippet of the build log, now with debug info

13 06 2017 18:00:09.149:INFO [karma]: Karma v1.4.1 server started at http://0.0.0.0:9876/
13 06 2017 18:00:09.150:INFO [launcher]: Launching browser ChromeHeadless with unlimited concurrency
13 06 2017 18:00:09.160:INFO [launcher]: Starting browser ChromeHeadless
13 06 2017 18:00:09.161:DEBUG [temp-dir]: Creating temp dir at /tmp/karma-35951605
13 06 2017 18:00:09.162:DEBUG [launcher]: google-chrome --user-data-dir=/tmp/karma-35951605 --no-default-browser-check --no-first-run --disable-default-apps --disable-popup-blocking --disable-translate --disable-background-timer-throttling --disable-renderer-backgrounding --disable-device-discovery-notifications http://localhost:9876/?id=35951605 --headless --disable-gpu --remote-debugging-port=9222
13 06 2017 18:00:09.389:DEBUG [launcher]: Process ChromeHeadless exited with code 0
13 06 2017 18:00:09.390:ERROR [launcher]: Cannot start ChromeHeadless
	
13 06 2017 18:00:09.390:DEBUG [temp-dir]: Cleaning temp dir /tmp/karma-35951605

so, Chrome is exiting with status 0 prematurely?

going to see if I can get chrome itself to produce some verbose log output

mentioned in issue #30876 (moved)

ok... after dozens of failed pipelines where I tried to enable debug output in Chrome I finally came across the following bugfixes relating to Karma's handling of stderr:

https://github.com/karma-runner/karma/pull/2639
https://github.com/karma-runner/karma/pull/2664

These two fixes landed in recent versions of karma, so upgrading to v1.7.0 solved the issue.

We can now see the real error message that was previously being silently discarded (build log):

Failed to move to new namespace: PID namespaces supported, Network namespace supported, but failed: errno = Operation not permitted

It seems this error is due to Chrome's sandbox feature. Some missing kernel privileges when running in a docker container appear to be at fault, and recommendations online suggested using docker run --cap-add=CAP_SYS_ADMIN. Since that isn't an option, I'm going to use --no-sandbox.

I still do not know why this failure was transient...

closed via merge request !12144 (merged)

mentioned in commit jt-gitlab/gitlab-ce@446f3787