Validate GitLab QA selectors in CE / EE
GitLab QA is black box integration testing framework. We use Capybara to click our way through GitLab, so we use DOM selectors a lot.
The problem is that selectors do change quite often, we shuffle a page structure, we remove pages, we add new ones. The ultimate solution is to run entire GitLab QA in CE / EE pipeline, which by itself would validate selectors. In case of GitLab QA incompatible change, pipeline would fail, and the author of the change would need to fix the problem.
This seems like a good solution, and we did a significant amount of work, described in #25 (closed), to make it happen. On the other hand, running entire GitLab QA pipeline for each commit in every merge request might not be realistic, as it depends on how long it is going to take to build a full Omnibus package.
Currently, it is hard to say if this is the way to go. Other options are:
- make GitLab QA a manual action, but somehow make it necessary to execute this action before merge request gets merged
- run some elements of GitLab QA test suite on the Rails app only (this is controversial, it might be troublesome as well)
- use persistent selectors, that we would tell people not to change, like
.qa-do-something-button
- add a new QA job which purpose would be to quickly validate all selector
I'm thinking about the last solution - adding a new GitLab QA job that would run for every commit in CE / EE repo.
We already have code that implements capybara actions in qa/
directory in CE / EE. We also have capybara actions grouped inside Pages
module.
We should be able to add abstraction that would make it possible to validate most of selectors on a Page
that we use in a real GitLab QA test pipeline.
The problem is that without performing real action, we won't be able to reach some selectors, but maybe the way to go is validate against app/views
?
This is merely an idea that we may want to consider.
We probably should start with the most simple solution - making it super fast to execute entire GitLab QA pipeline, including building a package.
/cc @rymai