Execute pending examples and mark as failed if they succeed.

This is a backwards incompatible change that makes the behaviour of
pending consisent with the current behaviour for when it is called
inside an example with a block.

The old "never run this example" behaviour is still available via the
"x" family of methods, the `:skip` metadata option, or the new `skip`
method.

Implements #1208.

Changelog.md

@@ -24,6 +24,11 @@ Breaking Changes for 3.0.0:
 * Remove support for deprecated `--configure` CLI option. (Myron Marston)
 * Remove support for deprecated `RSpec::Core::RakeTask#spec_opts=`.
   (Myron Marston)
+* An example group level `pending` block or `:pending` metadata now behaves the
+  same as a `pending` block inside an example: it will be executed and cause
+  a failure if it passes, otherwise it will be pending if it fails. The old
+  "never run" behaviour is still used when prefixing example/it/specify with an
+  `x`, or via a new `skip` method or `:skip` metadata option. (Xavier Shay)
 
 Enhancements:
 
@@ -41,6 +46,8 @@ Enhancements:
   pending. (Myron Marston)
 * Add `fdescribe` and `fcontext` as shortcuts to focus an example group.
   (Myron Marston)
+* Add `skip` method to not run any more of an example. (This behaviour is
+  identical to the old `pending` behaviour from RSpec 2.) (Xavier Shay)
 
 Bug Fixes:
 

features/command_line/format_option.feature

@@ -34,7 +34,7 @@ Feature: --format option
         end
 
         it "does something that is pending", :pending => true do
-          expect(5).to be > 3
+          expect(5).to be < 3
         end
       end
       """

features/hooks/around_hooks.feature

@@ -230,6 +230,7 @@ Feature: around hooks
 
         it "should be detected as pending" do
           pending
+          fail
         end
       end
       """

features/pending/pending_examples.feature

@@ -77,7 +77,78 @@ Feature: pending examples
     And the output should contain "Expected pending 'something else getting finished' to fail. No Error was raised."
     And the output should contain "pending_with_passing_block_spec.rb:3"
 
-  Scenario: temporarily pending by prefixing `it`, `specify`, or `example` with an x
+  Scenario: pending for an example that is currently passing
+    Given a file named "pending_with_passing_block_spec.rb" with:
+      """ruby
+      describe "an example" do
+        pending("something else getting finished") do
+          expect(1).to eq(1)
+        end
+      end
+      """
+    When I run `rspec pending_with_passing_block_spec.rb`
+    Then the exit status should not be 0
+    And the output should contain "1 example, 1 failure"
+    And the output should contain "FIXED"
+    And the output should contain "Expected pending 'No reason given' to fail. No Error was raised."
+    And the output should contain "pending_with_passing_block_spec.rb:2"
+
+  Scenario: pending for an example that is currently passing with a reason
+    Given a file named "pending_with_passing_block_spec.rb" with:
+      """ruby
+      describe "an example" do
+        example("something else getting finished", :pending => 'unimplemented') do
+          expect(1).to eq(1)
+        end
+      end
+      """
+    When I run `rspec pending_with_passing_block_spec.rb`
+    Then the exit status should not be 0
+    And the output should contain "1 example, 1 failure"
+    And the output should contain "FIXED"
+    And the output should contain "Expected pending 'unimplemented' to fail. No Error was raised."
+    And the output should contain "pending_with_passing_block_spec.rb:2"
+
+  Scenario: skipping using `skip`
+    Given a file named "skipped_spec.rb" with:
+      """ruby
+      describe "an example" do
+        skip "is skipped" do
+        end
+      end
+      """
+    When I run `rspec skipped_spec.rb`
+    Then the exit status should be 0
+    And the output should contain "1 example, 0 failures, 1 pending"
+    And the output should contain:
+      """
+      Pending:
+        an example is skipped
+          # No reason given
+          # ./skipped_spec.rb:2
+      """
+
+  Scenario: skipping using `skip` inside an example
+    Given a file named "skipped_spec.rb" with:
+      """ruby
+      describe "an example" do
+        it "is skipped" do
+          skip
+        end
+      end
+      """
+    When I run `rspec skipped_spec.rb`
+    Then the exit status should be 0
+    And the output should contain "1 example, 0 failures, 1 pending"
+    And the output should contain:
+      """
+      Pending:
+        an example is skipped
+          # No reason given
+          # ./skipped_spec.rb:2
+      """
+
+  Scenario: temporarily skipping by prefixing `it`, `specify`, or `example` with an x
     Given a file named "temporarily_pending_spec.rb" with:
       """ruby
       describe "an example" do
@@ -98,13 +169,13 @@ Feature: pending examples
       """
       Pending:
         an example is pending using xit
-          # Temporarily disabled with xit
+          # Temporarily skipped with xit
           # ./temporarily_pending_spec.rb:2
         an example is pending using xspecify
-          # Temporarily disabled with xspecify
+          # Temporarily skipped with xspecify
           # ./temporarily_pending_spec.rb:5
         an example is pending using xexample
-          # Temporarily disabled with xexample
+          # Temporarily skipped with xexample
           # ./temporarily_pending_spec.rb:8
       """
 
@@ -117,6 +188,7 @@ Feature: pending examples
         end
         specify do
           pending
+          fail
         end
       end
       """
@@ -138,7 +210,7 @@ Feature: pending examples
           expect(3+4).to eq(7)
         end
         pending do
-          expect("string".reverse).to eq("gnirts")
+          fail
         end
       end
       """

lib/rspec/core/example.rb

@@ -41,15 +41,23 @@ def self.delegate_to_metadata(*keys)
         keys.each { |key| define_method(key) { @metadata[key] } }
       end
 
-      delegate_to_metadata :full_description, :execution_result, :file_path, :pending, :location
+      delegate_to_metadata \
+        :execution_result,
+        :file_path,
+        :full_description,
+        :location,
+        :pending,
+        :skip
 
       # Returns the string submitted to `example` or its aliases (e.g.
       # `specify`, `it`, etc).  If no string is submitted (e.g. `it { is_expected.to
       # do_something }`) it returns the message generated by the matcher if
       # there is one, otherwise returns a message including the location of the
       # example.
       def description
-        description = metadata[:description].to_s.empty? ? "example at #{location}" : metadata[:description]
+        description = metadata[:description].to_s.empty? ?
+          "example at #{location}" :
+          metadata[:description]
         RSpec.configuration.format_docstrings_block.call(description)
       end
 
@@ -84,7 +92,6 @@ def initialize(example_group_class, description, metadata, example_block=nil)
         @example_group_class, @options, @example_block = example_group_class, metadata, example_block
         @metadata  = @example_group_class.metadata.for_example(description, metadata)
         @example_group_instance = @exception = nil
-        @pending_declared_in_example = false
         @clock = RSpec::Core::Time
       end
 
@@ -100,6 +107,7 @@ def example_group
       end
 
       alias_method :pending?, :pending
+      alias_method :skipped?, :skip
 
       # @api private
       # instance_evals the block passed to the constructor in the context of
@@ -112,13 +120,21 @@ def run(example_group_instance, reporter)
         start(reporter)
 
         begin
-          unless pending || RSpec.configuration.dry_run?
+          unless skipped? || RSpec.configuration.dry_run?
             with_around_each_hooks do
               begin
                 run_before_each
                 @example_group_instance.instance_exec(self, &@example_block)
-              rescue Pending::PendingDeclaredInExample => e
-                @pending_declared_in_example = e.message
+
+                result = metadata[:execution_result]
+
+                if result[:pending_fixed] == false
+                  result[:pending_fixed] = true
+                  raise Pending::PendingExampleFixedError
+                end
+              rescue Pending::SkipDeclaredInExample
+                # no-op, required metadata has already been set by the `skip`
+                # method.
               rescue Exception => e
                 set_exception(e)
               ensure
@@ -257,16 +273,18 @@ def start(reporter)
       end
 
       def finish(reporter)
-        if @exception
+        pending_message = metadata[:execution_result][:pending_message]
+
+        if pending_message && !metadata[:execution_result][:pending_fixed]
+          record_finished 'pending', :pending_message => pending_message
+          reporter.example_pending self
+          true
+        elsif @exception
           record_finished 'failed', :exception => @exception
           reporter.example_failed self
           false
-        elsif @pending_declared_in_example
-          record_finished 'pending', :pending_message => @pending_declared_in_example
-          reporter.example_pending self
-          true
-        elsif pending
-          record_finished 'pending', :pending_message => String === pending ? pending : Pending::NO_REASON_GIVEN
+        elsif skipped?
+          record_finished 'pending', :pending_message => skip_message
           reporter.example_pending self
           true
         else
@@ -278,7 +296,11 @@ def finish(reporter)
 
       def record_finished(status, results={})
         finished_at = clock.now
-        record results.merge(:status => status, :finished_at => finished_at, :run_time => (finished_at - execution_result[:started_at]).to_f)
+        record results.merge(
+          :status      => status,
+          :finished_at => finished_at,
+          :run_time    => (finished_at - execution_result[:started_at]).to_f
+        )
       end
 
       def run_before_each
@@ -301,7 +323,6 @@ def verify_mocks
         if metadata[:execution_result][:pending_fixed]
           metadata[:execution_result][:pending_fixed] = false
           metadata[:pending] = true
-          @pending_declared_in_example = metadata[:execution_result][:pending_message]
           @exception = nil
         else
           set_exception(e, :dont_print)
@@ -321,6 +342,14 @@ def assign_generated_description
       def record(results={})
         execution_result.update(results)
       end
+
+      def skip_message
+        if String === skip
+          skip
+        else
+          Pending::NO_REASON_GIVEN
+        end
+      end
     end
   end
 end

lib/rspec/core/example_group.rb

@@ -59,13 +59,41 @@ def self.define_example_method(name, extra_options={})
           define_method(name) do |*all_args, &block|
             desc, *args = *all_args
             options = Metadata.build_hash_from(args)
-            options.update(:pending => RSpec::Core::Pending::NOT_YET_IMPLEMENTED) unless block
+            options.update(:skip => RSpec::Core::Pending::NOT_YET_IMPLEMENTED) unless block
             options.update(extra_options)
+
+            # Metadata inheritance normally happens in `Example#initialize`,
+            # but for `:pending` specifically we need it earlier.
+            pending_metadata = options[:pending] || metadata[:pending]
+
+            if pending_metadata
+              options, block = ExampleGroup.pending_metadata_and_block_for(
+                options.merge(:pending => pending_metadata),
+                block
+              )
+            end
+
             examples << RSpec::Core::Example.new(self, desc, options, block)
             examples.last
           end
         end
 
+        def pending_metadata_and_block_for(options, block)
+          if String === options[:pending]
+            reason = options[:pending]
+          else
+            options[:pending] = true
+            reason = RSpec::Core::Pending::NO_REASON_GIVEN
+          end
+
+          # This will fail if no block is provided, which is effectively the
+          # same as failing the example so it will be marked correctly as
+          # pending.
+          callback = Proc.new { pending(reason) { instance_exec(&block) } }
+
+          return options, callback
+        end
+
         # Defines an example within a group.
         # @example
         #   example do
@@ -100,18 +128,21 @@ def self.define_example_method(name, extra_options={})
         # @see example
         define_example_method :fit,     :focused => true, :focus => true
 
-        # Shortcut to define an example with :pending => true
+        # Shortcut to define an example with :skip => 'Temporarily skipped with xexample'
         # @see example
-        define_example_method :pending,  :pending => true
-        # Shortcut to define an example with :pending => 'Temporarily disabled with xexample'
+        define_example_method :xexample, :skip => 'Temporarily skipped with xexample'
+        # Shortcut to define an example with :skip => 'Temporarily skipped with xit'
         # @see example
-        define_example_method :xexample, :pending => 'Temporarily disabled with xexample'
-        # Shortcut to define an example with :pending => 'Temporarily disabled with xit'
+        define_example_method :xit,      :skip => 'Temporarily skipped with xit'
+        # Shortcut to define an example with :skip => 'Temporarily skipped with xspecify'
         # @see example
-        define_example_method :xit,      :pending => 'Temporarily disabled with xit'
-        # Shortcut to define an example with :pending => 'Temporarily disabled with xspecify'
+        define_example_method :xspecify, :skip => 'Temporarily skipped with xspecify'
+        # Shortcut to define an example with :skip => true
         # @see example
-        define_example_method :xspecify, :pending => 'Temporarily disabled with xspecify'
+        define_example_method :skip,     :skip => true
+        # Shortcut to define an example with :pending => true
+        # @see example
+        define_example_method :pending,  :pending => true
 
         # Works like `alias_method :name, :example` with the added benefit of
         # assigning default metadata to the generated example.
@@ -270,11 +301,11 @@ def self.example_group(*args, &example_group_block)
 
       # Shortcut to temporarily make an example group pending.
       # @see example_group
-      alias_example_group_to :xdescribe, :pending => "Temporarily disabled with xdescribe"
+      alias_example_group_to :xdescribe, :skip => "Temporarily skipped with xdescribe"
 
       # Shortcut to temporarily make an example group pending.
       # @see example_group
-      alias_example_group_to :xcontext,  :pending => "Temporarily disabled with xcontext"
+      alias_example_group_to :xcontext,  :skip => "Temporarily skipped with xcontext"
 
       # Shortcut to define an example group with `:focus` => true
       # @see example_group