Skip to content

Add chaos test via toxiproxy

George Koltsov requested to merge jmd-praefect-chaos into master

What does this MR do and why?

Add toxiproxy as a component to run adjacent to the GitalyCluster scenario. With this in place we will be able to make use of toxiproxy to add a category of chaos tests, primarily focused on networking issues to the gitlab-qa suite.
The general concept for this would be to run the chaos tests in their own pipeline separately from the E2E suite, but by making use of the existing framework, we can build out a suite of tests without the need to maintain multiple sets of supporting test.

Adding chaos tests with this change in place should be relatively familiar to anyone who is familiar with our current E2E test framework and a sample of what the test could look like is available in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/98019/diffs for reference.

This first iteration is focused on the Gitaly Cluster components, but with this implemented it should be possible to further extend this, by modifying the configuration for the gitlab instance as well to route traffic to it's various components via the proxy in future iterations.

How to set up and validate locally

bundle exec exe/gitlab-qa Test::Integration::Chaos EE ...

To manually add 'toxics' refer to toxiproxy#http-api.
Alternatively consider using the toxiproxy-ruby client

curl --location --request POST 'http://localhost:8474/proxies/gitaly1/toxics' \
--header 'Content-Type: application/json' \
--data-raw '{
    "name": "gitaly1_latency",
    "type": "latency",
    "attributes": {
        "latency": 1000,
        "jitter": 250
    },
     "stream": "downstream"
}'

With this setup, navigate to the gitaly-cluster running, and note from the application behaviour that the 'toxics' you applied will have an impact on the operation of the application.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by George Koltsov

Merge request reports