Skip to content

Bug when an exception occurs in after(:all) hook

gitlab-qa-bot requested to merge github/fork/myronmarston/error_in_after_all into master

Created by: myronmarston

When an exception occurs in an after(:all) hook, rspec doesn't handle it correctly. RSpec itself raises this exception:

/Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/example_group.rb:278:in `instance_eval_with_rescue': undefined method `set_exception' for nil:NilClass (NoMethodError)
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/hooks.rb:39:in `run_in'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/hooks.rb:74:in `run_all!'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/hooks.rb:109:in `run_hook!'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/example_group.rb:198:in `eval_after_alls'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/example_group.rb:223:in `run'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/command_line.rb:43:in `run_examples'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/configuration.rb:187:in `inject'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/command_line.rb:43:in `each'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/command_line.rb:43:in `inject'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/command_line.rb:43:in `run_examples'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/command_line.rb:26:in `run'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/reporter.rb:11:in `report'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/command_line.rb:23:in `run'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/runner.rb:55:in `run_in_process'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/runner.rb:46:in `run'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/runner.rb:10:in `autorun'
        from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bin/rspec:19

I started to look into fixing it, and there's not a clear "right way" to do it, so I just added a wip cuke and pending spec. Some thoughts of mine:

For consistency's sake, I think that an exception in an after(:all) hook should cause all the examples in the group to fail. This is symmetrical with what happens on an exception in a before(:all) hook. This desired behavior is what my cuke and spec describe. That said, I know it's not that simple--as I understand it, the rspec runner reports each example as passing, as it completes, before the after(:all) hooks run. This complicates things as we can't really tell the reporter "actually, those specs didn't pass...". Would it make sense for an example group that has an after(:all) hook to delay reporting the status of the specs until the entire group (including the after(:all) hook) has run, so that the success/failure of the hook can be taken into account in determining if each spec passes or fails?

I took a look at the behavior of rspec 1.3:


require 'rubygems'
gem 'rspec', '1.3.0'

require 'spec'
require 'spec/autorun'

describe "an error in after(:all)" do
  after(:all) do
    raise "Oops"
  end

  it "fails this example" do
  end

  it "fails this example, too" do
  end
end

ruby-1.8.7-p302 ➜  code  ruby rspec_1_example.rb --format nested
an error in after(:all)
  fails this example
  fails this example, too
  after(:all) (FAILED - 1)

1)
RuntimeError in 'an error in after(:all) after(:all)'
Oops
rspec_1_example.rb:9:
rspec_1_example.rb:7:

Finished in 0.009762 seconds

2 examples, 1 failure

This is interesting---it essentially dynamically reports after(:all) as an example and considers it a failure. I'm fine with this behavior if you want to preserve it, although I think the consistency/symmetry I've suggested would be a bit better. Overall, I think we just need to decide and document the behavior of this, and ensure RSpec 2 conforms to it.

Merge request reports