Merge branch 'benchmark-suite' into 'master'

Basic RSpec/benchmark-ips powered benchmark suite

Corresponding issue: #2909, see the commit messages for more details.

A few things to note:

1. The current use of `subject` isn't exactly easy on the eyes due to them having to return a Proc, I'm not sure yet how (and if) we can work around this.
2. The maximum amount of iterations in the current `User.by_login` benchmark is arbitrary, we might have to adjust it once said method's performance has been improved.
3. Benchmarks currently take 2 seconds to warm up and 5 seconds to run (benchmark-ips defaults). 
4. The custom RSpec matcher file (`benchmark_matchers.rb`) is a bit messy, any feedback on this would be appreciated

Any comments/feedback on this would be greatly appreciated.

See merge request !1503

.gitlab-ci.yml

@@ -24,6 +24,14 @@ spec:api:
     - ruby
     - mysql
+spec:benchmark:
+  script:
+    - RAILS_ENV=test bundle exec rake spec:benchmark
+  tags:
+    - ruby
+    - mysql
+  allow_failure: true
+
 spec:other:
   script:
     - RAILS_ENV=test SIMPLECOV=true bundle exec rake spec:other

doc/development/README.md

@@ -8,3 +8,4 @@
 - [UI guide](ui_guide.md) for building GitLab with existing css styles and elements
 - [Migration Style Guide](migration_style_guide.md) for creating safe migrations
 - [How to dump production data to staging](dump_db.md)
+- [Benchmarking](benchmarking.md)

doc/development/benchmarking.md

+# Benchmarking
+
+GitLab CE comes with a set of benchmarks that are executed for every build. This
+makes it easier to measure performance of certain components over time.
+
+Benchmarks are written as RSpec tests using a few extra helpers. To write a
+benchmark, first tag the top-level `describe`:
+
+```ruby
+describe MaruTheCat, benchmark: true do
+
+end
+```
+
+This ensures the benchmark is executed separately from other test collections.
+It also exposes the various RSpec matchers used for writing benchmarks to the
+test group.
+
+Next, lets write the actual benchmark:
+
+```ruby
+describe MaruTheCat, benchmark: true do
+  let(:maru) { MaruTheChat.new }
+
+  describe '#jump_in_box' do
+    benchmark_subject { maru.jump_in_box }
+
+    it { is_expected.to iterate_per_second(9000) }
+  end
+end
+```
+
+Here `benchmark_subject` is a small wrapper around RSpec's `subject` method that
+makes it easier to specify the subject of a benchmark. Using RSpec's regular
+`subject` would require us to write the following instead:
+
+```ruby
+subject { -> { maru.jump_in_box } }
+```
+
+The `iterate_per_second` matcher defines the amount of times per second a
+subject should be executed. The higher the amount of iterations the better.
+
+By default the allowed standard deviation is a maximum of 30%. This can be
+adjusted by chaining the `with_maximum_stddev` on the `iterate_per_second`
+matcher:
+
+```ruby
+it { is_expected.to iterate_per_second(9000).with_maximum_stddev(50) }
+```
+
+This can be useful if the code in question depends on external resources of
+which the performance can vary a lot (e.g. physical HDDs, network calls, etc).
+However, in most cases 30% should be enough so only change this when really
+needed.
+
+## Benchmarks Location
+
+Benchmarks should be stored in `spec/benchmarks` and should follow the regular
+Rails specs structure. That is, model benchmarks go in `spec/benchmark/models`,
+benchmarks for code in the `lib` directory go in `spec/benchmarks/lib`, etc.
+
+## Underlying Technology
+
+The benchmark setup uses [benchmark-ips][benchmark-ips] which takes care of the
+heavy lifting such as warming up code, calculating iterations, standard
+deviation, etc.
+
+[benchmark-ips]: https://github.com/evanphx/benchmark-ips

lib/tasks/spec.rake

@@ -19,11 +19,20 @@ namespace :spec do
     run_commands(cmds)
   end
+  desc 'GitLab | Rspec | Run benchmark specs'
+  task :benchmark do
+    cmds = [
+      %W(rake gitlab:setup),
+      %W(rspec spec --tag @benchmark)
+    ]
+    run_commands(cmds)
+  end
+
   desc 'GitLab | Rspec | Run other specs'
   task :other do
     cmds = [
       %W(rake gitlab:setup),
-      %W(rspec spec --tag ~@api --tag ~@feature)
+      %W(rspec spec --tag ~@api --tag ~@feature --tag ~@benchmark)
     ]
     run_commands(cmds)
   end

spec/benchmarks/models/user_spec.rb

+require 'spec_helper'
+
+describe User, benchmark: true do
+  describe '.by_login' do
+    before do
+      %w{Alice Bob Eve}.each do |name|
+        create(:user,
+               email: "#{name}@gitlab.com",
+               username: name,
+               name: name)
+      end
+    end
+
+    let(:iterations) { 1000 }
+
+    describe 'using a capitalized username' do
+      benchmark_subject { User.by_login('Alice') }
+
+      it { is_expected.to iterate_per_second(iterations) }
+    end
+
+    describe 'using a lowercase username' do
+      benchmark_subject { User.by_login('alice') }
+
+      it { is_expected.to iterate_per_second(iterations) }
+    end
+
+    describe 'using a capitalized Email address' do
+      benchmark_subject { User.by_login('Alice@gitlab.com') }
+
+      it { is_expected.to iterate_per_second(iterations) }
+    end
+
+    describe 'using a lowercase Email address' do
+      benchmark_subject { User.by_login('alice@gitlab.com') }
+
+      it { is_expected.to iterate_per_second(iterations) }
+    end
+  end
+end

spec/spec_helper.rb

@@ -14,6 +14,7 @@ require File.expand_path("../../config/environment", __FILE__)
 require 'rspec/rails'
 require 'shoulda/matchers'
 require 'sidekiq/testing/inline'
+require 'benchmark/ips'
 # Requires supporting ruby files with custom matchers and macros, etc,
 # in spec/support/ and its subdirectories.
@@ -32,7 +33,7 @@ RSpec.configure do |config|
   config.include TestEnv
   config.include StubGitlabCalls
   config.include StubGitlabData
-
+  config.include BenchmarkMatchers, benchmark: true
   config.infer_spec_type_from_file_location!
   config.raise_errors_for_deprecations!

spec/support/matchers/benchmark_matchers.rb

+module BenchmarkMatchers
+  extend RSpec::Matchers::DSL
+
+  def self.included(into)
+    into.extend(ClassMethods)
+  end
+
+  matcher :iterate_per_second do |min_iterations|
+    supports_block_expectations
+
+    match do |block|
+      @max_stddev ||= 30
+
+      @entry = benchmark(&block)
+
+      expect(@entry.ips).to be >= min_iterations
+      expect(@entry.stddev_percentage).to be <= @max_stddev
+    end
+
+    chain :with_maximum_stddev do |value|
+      @max_stddev = value
+    end
+
+    description do
+      "run at least #{min_iterations} iterations per second"
+    end
+
+    failure_message do
+      ips    = @entry.ips.round(2)
+      stddev = @entry.stddev_percentage.round(2)
+
+      "expected at least #{min_iterations} iterations per second " \
+        "with a maximum stddev of #{@max_stddev}%, instead of " \
+        "#{ips} iterations per second with a stddev of #{stddev}%"
+    end
+  end
+
+  # Benchmarks the given block and returns a Benchmark::IPS::Report::Entry.
+  def benchmark(&block)
+    report = Benchmark.ips(quiet: true) do |bench|
+      bench.report(&block)
+    end
+
+    report.entries[0]
+  end
+
+  module ClassMethods
+    # Wraps around rspec's subject method so you can write:
+    #
+    #     benchmark_subject { SomeClass.some_method }
+    #
+    # instead of:
+    #
+    #     subject { -> { SomeClass.some_method } }
+    def benchmark_subject(&block)
+      subject { block }
+    end
+  end
+end

spec/support/setup_builds_storage.rb

@@ -10,8 +10,10 @@ RSpec.configure do |config|
   end
   config.after(:suite) do
-    Dir.chdir(builds_path) do
-      `ls | grep -v .gitkeep | xargs rm -r`
+    Dir[File.join(builds_path, '*')].each do |path|
+      next if File.basename(path) == '.gitkeep'
+
+      FileUtils.rm_rf(path)
     end
   end
 end