Replace PhantomJS with headless Chrome

Since Chrome canary builds aren't available on linux platforms, we'll either have to use a nightly Chromium PPA (which is basically the same, sans PDF/Flash/FFMpeg support) or wait for Chrome 59 to release (Chrome 57 is the latest stable version as of Apr 12, 2017)

https://launchpad.net/~chromium-daily/+archive/ppa

@mikegreiling per this page 59 will become stable on June 6th, 2017.

mentioned in issue gitlab-foss#30906 (closed)

mentioned in issue gitlab-foss#30912 (closed)

Looks like PhantomJS will no longer be maintained: https://groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuNE

Unfortunately it's not as simple as just telling Capybara "use Chrome now". It needs a driver, which AFAIK doesn't exist yet, but I'm sure is being worked on somewhere.

There is this though: http://blog.faraday.io/headless-chromium-with-capybara-and-selenium-webdriver/

Yeah, I suggested possibly looking into the selenium driver in the issue description... Ideally though, somebody will develop something more tailored to Chrome.

There's this bug currently being worked on in the chromium issue tracker: https://bugs.chromium.org/p/chromium/issues/detail?id=604324, which covers the use of drivers for selenium, so probably this work might lead to the much needed drivers for capybara?

It looks like selenium webdriver's chrome driver works with it I suppose it's in their interest to keep it compatible with chromedriver, but not sure if it actually uses chromedriver or what.

We need to transition from PhantomJS to headless Chrome with frontend performance monitoring too. I went with PhantomJS initially because headless Chrome was proving difficult to get working with Selenium. Maybe I need to look at that again. @mikegreiling Let me know if there's any way I can be helpful on this.

mentioned in issue gitlab-foss#30996 (closed)

mentioned in issue gitlab-foss#31627 (closed)

if anyone needs docker image with Ruby 2.4.1 with NodeJS, PhantomJS and Selenium Chrome driver with chrome then use the image below.

image: saimonsl/ubuntu-16.04:nodejs-ruby-4.1-phantomjs

Excellent for testing.

@slovell Thanks for pointing us to it, however, we don't use images hosted outside of build images project to make our deploy easier.

Also, using non default images might have security implications.

mentioned in issue gitlab-foss#31198 (moved)

@victorwu @timzallmann can we schedule this?

@timzallmann I'd like to take this one when we schedule it (if you don't mind )

mentioned in issue gitlab-foss#32519 (closed)

mentioned in merge request gitlab-foss!11597 (closed)

mentioned in issue gitlab-qa#48 (closed)

@mikegreiling Implementation of headless chrome for RSpec and Capybara is here -> https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11680. This is a solution for GitLab QA, but this should look very similar in CE / EE.

One thing to note is that this solution is indeed using Selenium. This means that it needs a chrome webdriver. Current version of chrome webdriver 2.29 does not support Chrome 59 yet (however it works quite well in GitLab QA case).

I suppose that it might be a good idea to wait for next version of chrome driver that will support more headless-things. Currently it still needs xvfb so it probably does not fully support headless browser and could be faster.

Notes from official release post:

Can I use this with Selenium / WebDriver / ChromeDriver?

Right now, Selenium opens a full instance of Chrome. In other words, it's an automated solution but not completely headless. However, Selenium could use --headless in the future.

If you want to bleed on the edge, I recommend Running Selenium with Headless Chrome to set things up yourself.

Note: you may encounter bugs using ChromeDriver. At the time of writing, the latest release (2.29) only supports Chrome 58. Headless Chrome requires Chrome 59 or later.

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11680 is using bleeding edge approach.

@adamniedzielski I just took a shot to get If-None-Match(For Etag tests in middleware) with https://gitlab.com/gitlab-org/gitlab-ce/issues/30876#note_30581674 approach, which is using headless chrome. And I confirmed it returns env['HTTP_IF_NONE_MATCH'] successfully. (Here is the minimal test project) Probably we should wait a bit like @grzesiek suggested, but once we moved to the headless chrome, we can write tests for Etag Caching.

@grzesiek awesome!

Once Chrome 59 gets out of beta on June 6th we can add the latest stable version to our CI docker image and go from there. Hopefully the drivers will update quickly.

As I understand it, headless Chrome already works with Karma, so we should at least be able to start running our frontend tests with it out of the gate.

Hi gang: Long time listener, first time poster.

I found this thread because we're working on similar issues over here: https://www.drupal.org/node/2775653 .

Also, to share the excitement that Chrome 59 is out so Headless Chrome is in stable.

Currently, we're looking closely at https://packagist.org/packages/dmore/chrome-mink-driver and at the present it appears that it is fundamentally different than the php-webdriver we were using. And that the current conventional wisdom is to try to figure out how to use selenium + Headless Chrome.

mentioned in issue gitlab-foss#33392 (moved)

changed milestone to %9.4 in gitlab-foss

FWIW, I could successfully use Chrome Canary locally (on macOS) instead of PhantomJS (using Selenium Driver and the work from @grzesiek in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11680/diffs#58c93d24222c4a1bd2252a974003ffc3d32f3033):

› chromedriver --version
ChromeDriver 2.30.477690 (c53f4ad87510ee97b5c3425a14c0e79780cdf262)

› chrome-canary --version
Google Chrome 61.0.3124.0 canary

changed the description

added Deliverable in GitLab FOSS label

mentioned in issue gitlab-build-images#10 (closed)

assigned to @mikegreiling

Looks like the karma-chrome-launcher plugin has added support for headless Chrome in v2.1.0. Going to test this out now.

mentioned in merge request gitlab-foss!12036 (merged)

mentioned in issue gitlab-foss#33538 (closed)

changed the description

marked the checklist item Update docker images to include headless Chrome pre-installed (gitlab-build-images!41 (merged)) as completed

marked the checklist item Remove PhantomJS from these images as completed

changed the description

marked the checklist item Remove PhantomJS-specific hacks from our frontend tests as completed

mentioned in merge request gitlab-foss!12144 (merged)

Had to revert gitlab-foss!12036 (merged) due to intermittent CI failures. Discussion ongoing in gitlab-foss#33633 (closed) and gitlab-foss!12144 (merged)

changed the description

marked the checklist item Update Karma config to utilize Chrome instead of PhantomJS (gitlab-ce!12036) as completed

mentioned in commit gitlab-foss@ddbb8801

mentioned in merge request gitlab-foss!12244

Thanks to @grzesiek's work in gitlab-foss!11680 (merged) and https://robots.thoughtbot.com/headless-feature-specs-with-chrome, I did a quick test to see how well headless Chrome would work with our current rspec and Spinach tests. It looks like trigger('click') isn't supported. Anyone is welcome to contribute to https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12244.

/cc: @rymai

Most of the trigger('click') statements were workarounds for poor PhantomJS page rendering anyway, so reverting them to a simple .click will probably work in most cases.

It seems that the main problem right now is that most of our feature specs are failing (e.g. https://gitlab.com/gitlab-org/gitlab-ce/builds/18895708) because of the error unknown error: an X display is required for keycode conversions, consider using Xvfb.

For GitLab QA, I see that @grzesiek runs xvfb. Is this what we want to do?

There is discussion about this problem here: https://bugs.chromium.org/p/chromedriver/issues/detail?id=1772. Some users mention compiling with use_ozone on Chrome and ChromeDriver solves this problem.

that's a shame because avoiding a virtual frame buffer like xvfb is pretty much the whole point of the --headless feature. should we try compiling chromium in the dockerfile ourselves with that use_ozone flag? seems like a lot of hassle...

For GitLab QA, I see that @grzesiek runs xvfb. Is this what we want to do?

@stanhu In GitLab QA we use xvfb only because back when I was implementing this solution, there was no chrome driver with support for v59 headless chrome yet. It appears that it is now available, since a few days, so I would vote for not using xvfb.

that's a shame because avoiding a virtual frame buffer like xvfb is pretty much the whole point of the --headless feature

so, it appears we're currently blocked by https://bugs.chromium.org/p/chromedriver/issues/detail?id=1772 unless we want to try a workaround:

we can compile chromium ourselves with the use_ozone flag

Cons: our Dockerfile becomes way more complex and compile time will increase

we can hack a sendKeys implementation using javascript

Cons: this might be really messy, it would require refactoring a lot of tests, and might not completely replicate the behavior we need.

we can install xvfb

Cons: this might be too resource hungry for CI

we wait until this gets worked out within chrome/chromedriver and continue to use PhantomJS in the meantime

What are your thoughts @stanhu, @grzesiek?

I'm inclined to try option 3 and see how it performs. If the performance or resource requirements are unacceptable we probably ought to wait for an official fix.

assigned to @grzesiek

unassigned @grzesiek

@mikegreiling I agree that we should try option 3, to see how it works. I guess that this limitation is going to be resolved soon by upstream maintainers, but we can benefit from knowing how it performs with xvfb right now.

@grzesiek The WebDriver sendKeys problem is related to lines like following:

page.find('.filtered-search').send_keys(:enter)

this will not work without xvfb for some reason, and we use this type of method to fill out form fields and trigger keyboard events in a lot of places in our test suite.

mentioned in issue gitlab-foss#34012 (moved)

added OKR-FE in GitLab FOSS label

Quick update on the status of this issue.

The sendKeys bug in ChromeDriver has been fixed and should be available whenever they release v2.31 https://bugs.chromium.org/p/chromedriver/issues/detail?id=1772

I think it is worth waiting for this rather than having a go at xvfb or working around this with a hack.

Marking this as blocked in GitLab FOSS until then.

added blocked in GitLab FOSS label

https://drivy.engineering/running-capybara-headless-chrome/ might be worth reading as well.

@mikegreiling since this is blocked, should we move this to %9.5 in gitlab-foss?

@ClemMakesApps yes, changing it now.

changed milestone to %9.5 in gitlab-foss

Just for kicks, I'm going to try compiling the latest chromedriver from source in Docker to see how well it plays with @stanhu's WIP.

Thanks @mikegreiling

@mikegreiling Someone in the community already created their own version of Chromedriver: https://github.com/davidthornton/chromedriver-2.31

Hey Folks, you might also be interested in https://github.com/joelgriffith/navalia (I'm the maintainer) as it's heavily geared towards integration/functional testing. I'd be happy to help work through any issues you might encounter.

If you're looking for some "recipes" of how to execute functional tests in it, check out this writeup (It mentions react, but can be used on any stack): https://codeburst.io/composable-end-to-end-tests-for-react-apps-2ec82170af62

Thanks!