Setup benchmark suite for CI
We currently don't have any benchmarks that are automatically run on the CI, this makes it hard to ensure performance problems aren't re-introduced. This also makes it harder to test performance using different databases as not everybody might be able to set up both PostgreSQL and MySQL.
To help alleviate these problems we should add a benchmark suite that:
- Is executed on the CI just like the tests
- Can specify performance requirements
- Output the performance (e.g. iterations/sec) so we can view it in CI output
- Can be skipped when you just want to run the tests
A basic setup would be to basically write RSpec tests but put them in bench/
instead of spec
, allowing you to run both independently. Coming up with a way to specify performance requirements however is a bit harder. For example, you might require a certain algorithm to run at N iterations per second. Different platforms might yield different iterations/sec, possibly leading to the benchmarks to fail even though the "relative" performance is still good enough. Just measuring how something scales (linear, exponential, etc) isn't enough. Something scaling linearly doesn't say anything about how fast it performs.
As a first step I'll do some digging to see if there are any existing solutions for this. If anybody has any suggestions, please do share them :)