Bug when an exception occurs in after(:all) hook
Created by: myronmarston
When an exception occurs in an after(:all) hook, rspec doesn't handle it correctly. RSpec itself raises this exception:
/Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/example_group.rb:278:in `instance_eval_with_rescue': undefined method `set_exception' for nil:NilClass (NoMethodError) from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/hooks.rb:39:in `run_in' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/hooks.rb:74:in `run_all!' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/hooks.rb:109:in `run_hook!' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/example_group.rb:198:in `eval_after_alls' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/example_group.rb:223:in `run' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/command_line.rb:43:in `run_examples' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/configuration.rb:187:in `inject' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/command_line.rb:43:in `each' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/command_line.rb:43:in `inject' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/command_line.rb:43:in `run_examples' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/command_line.rb:26:in `run' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/reporter.rb:11:in `report' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/command_line.rb:23:in `run' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/runner.rb:55:in `run_in_process' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/runner.rb:46:in `run' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bundler/gems/rspec-core-cfa5b5707fd6/lib/rspec/core/runner.rb:10:in `autorun' from /Users/mmarston/.rvm/gems/ruby-1.8.7-p302/bin/rspec:19
I started to look into fixing it, and there's not a clear "right way" to do it, so I just added a wip cuke and pending spec. Some thoughts of mine:
For consistency's sake, I think that an exception in an after(:all) hook should cause all the examples in the group to fail. This is symmetrical with what happens on an exception in a before(:all) hook. This desired behavior is what my cuke and spec describe. That said, I know it's not that simple--as I understand it, the rspec runner reports each example as passing, as it completes, before the after(:all) hooks run. This complicates things as we can't really tell the reporter "actually, those specs didn't pass...". Would it make sense for an example group that has an after(:all) hook to delay reporting the status of the specs until the entire group (including the after(:all) hook) has run, so that the success/failure of the hook can be taken into account in determining if each spec passes or fails?
I took a look at the behavior of rspec 1.3:
require 'rubygems' gem 'rspec', '1.3.0' require 'spec' require 'spec/autorun' describe "an error in after(:all)" do after(:all) do raise "Oops" end it "fails this example" do end it "fails this example, too" do end end
ruby-1.8.7-p302 ➜ code ruby rspec_1_example.rb --format nested an error in after(:all) fails this example fails this example, too after(:all) (FAILED - 1) 1) RuntimeError in 'an error in after(:all) after(:all)' Oops rspec_1_example.rb:9: rspec_1_example.rb:7: Finished in 0.009762 seconds 2 examples, 1 failure
This is interesting---it essentially dynamically reports after(:all) as an example and considers it a failure. I'm fine with this behavior if you want to preserve it, although I think the consistency/symmetry I've suggested would be a bit better. Overall, I think we just need to decide and document the behavior of this, and ensure RSpec 2 conforms to it.