Solve problem of the fragile tests
Updated content
By fragile tests in this context I mean that when some developer changes something in GitLab, and the change breaks GitLab QA (false negative) which is navigating through GitLab using selectors / buttons / menus (click-driven approach), the developer should be responsible for fixing GitLab in their merge request, and the change should not cause false negatives in nightly pipelines.
Previous content
Outdated - we have moved instance level integrations tests to CE / EE repositories
Keeping previous content here, because it shows what problems we had to solve, and where we are now.
We decided to keep integration tests in the separate repository, because of few important reasons:
- we would like to have a separate pipeline that leverages Docker and container registry intensely
- it can take significant amount of time to finish pipeline, especially when we will install omnibus packages to test full omnibus instalation
- it can take significant amount of time to test upgrades
- we would like to support test matrix for EE, CE and Staging
- we would like to provide a standalone QA tool for customers, so that they could run tests against their on-premises installation
But this has some implications, and the most important implication are fragile tests related to changes in CE and EE and synchronization of code in QA repository.
Problems we need to solve:
- how to make sure that developers see the pipeline status of QA test suite when they change something in the codebase
- how can we leverage multi project pipeline when we have it
- how can we prevent problem when QA pipeline will take a long time, and this does not make sense to attach this pipeline to pipeline in CE/EE repositories
- synchronize development on QA test suite with development of GitLab CE/EE (should we release at the same time?)