gh-108885: Use subtests for doctest examples run by unittest

Run each example as a subtest in unit tests synthesized by
doctest.DocFileSuite() and doctest.DocTestSuite().

Author: serhiy-storchaka

Doc/library/doctest.rst

@@ -1043,12 +1043,15 @@ from text files and modules with doctests:
    Convert doctest tests from one or more text files to a
    :class:`unittest.TestSuite`.
 
-   The returned :class:`unittest.TestSuite` is to be run by the unittest framework
-   and runs the interactive examples in each file.  If an example in any file
-   fails, then the synthesized unit test fails, and a :exc:`failureException`
-   exception is raised showing the name of the file containing the test and a
-   (sometimes approximate) line number.  If all the examples in a file are
-   skipped, then the synthesized unit test is also marked as skipped.
+   The returned :class:`unittest.TestSuite` is to be run by the unittest
+   framework and runs the interactive examples in each file.
+   Each file is run as a separate unit test, and each example in a file
+   is run as a :ref:`subtest <subtests>`.
+   If any example in a file fails, then the synthesized unit test fails.
+   The traceback for failure or error contains the name of the file
+   containing the test and a (sometimes approximate) line number.
+   If all the examples in a file are skipped, then the synthesized unit
+   test is also marked as skipped.
 
    Pass one or more paths (as strings) to text files to be examined.
 
@@ -1078,12 +1081,12 @@ from text files and modules with doctests:
 
    Optional argument *setUp* specifies a set-up function for the test suite.
    This is called before running the tests in each file.  The *setUp* function
-   will be passed a :class:`DocTest` object.  The setUp function can access the
+   will be passed a :class:`DocTest` object.  The *setUp* function can access the
    test globals as the *globs* attribute of the test passed.
 
    Optional argument *tearDown* specifies a tear-down function for the test
    suite.  This is called after running the tests in each file.  The *tearDown*
-   function will be passed a :class:`DocTest` object.  The setUp function can
+   function will be passed a :class:`DocTest` object.  The *tearDown* function can
    access the test globals as the *globs* attribute of the test passed.
 
    Optional argument *globs* is a dictionary containing the initial global
@@ -1105,16 +1108,22 @@ from text files and modules with doctests:
    The global ``__file__`` is added to the globals provided to doctests loaded
    from a text file using :func:`DocFileSuite`.
 
+   .. versionchanged:: next
+      Run each example as a :ref:`subtest <subtests>`.
+
 
 .. function:: DocTestSuite(module=None, globs=None, extraglobs=None, test_finder=None, setUp=None, tearDown=None, optionflags=0, checker=None)
 
    Convert doctest tests for a module to a :class:`unittest.TestSuite`.
 
-   The returned :class:`unittest.TestSuite` is to be run by the unittest framework
-   and runs each doctest in the module.  If any of the doctests fail, then the
-   synthesized unit test fails, and a :exc:`failureException` exception is raised
-   showing the name of the file containing the test and a (sometimes approximate)
-   line number.  If all the examples in a docstring are skipped, then the
+   The returned :class:`unittest.TestSuite` is to be run by the unittest
+   framework and runs each doctest in the module.
+   Each docstring is run as a separate unit test, and each example in
+   a docstring is run as a :ref:`subtest <subtests>`.
+   If any of the doctests fail, then the synthesized unit test fails.
+   The traceback for failure or error contains the name of the file
+   containing the test and a (sometimes approximate) line number.
+   If all the examples in a docstring are skipped, then the
    synthesized unit test is also marked as skipped.
 
    Optional argument *module* provides the module to be tested.  It can be a module
@@ -1132,19 +1141,16 @@ from text files and modules with doctests:
    drop-in replacement) that is used to extract doctests from the module.
 
    Optional arguments *setUp*, *tearDown*, and *optionflags* are the same as for
-   function :func:`DocFileSuite` above.
+   function :func:`DocFileSuite` above, but they are called for each docstring.
 
    This function uses the same search technique as :func:`testmod`.
 
    .. versionchanged:: 3.5
       :func:`DocTestSuite` returns an empty :class:`unittest.TestSuite` if *module*
       contains no docstrings instead of raising :exc:`ValueError`.
 
-.. exception:: failureException
-
-   When doctests which have been converted to unit tests by :func:`DocFileSuite`
-   or :func:`DocTestSuite` fail, this exception is raised showing the name of
-   the file containing the test and a (sometimes approximate) line number.
+   .. versionchanged:: next
+      Run each example as a :ref:`subtest <subtests>`.
 
 Under the covers, :func:`DocTestSuite` creates a :class:`unittest.TestSuite` out
 of :class:`!doctest.DocTestCase` instances, and :class:`!DocTestCase` is a
@@ -1508,7 +1514,7 @@ DocTestRunner objects
    with strings that should be displayed.  It defaults to ``sys.stdout.write``.  If
    capturing the output is not sufficient, then the display output can be also
    customized by subclassing DocTestRunner, and overriding the methods
-   :meth:`report_start`, :meth:`report_success`,
+   :meth:`report_skip`, :meth:`report_start`, :meth:`report_success`,
    :meth:`report_unexpected_exception`, and :meth:`report_failure`.
 
    The optional keyword argument *checker* specifies the :class:`OutputChecker`
@@ -1533,14 +1539,27 @@ DocTestRunner objects
    :class:`DocTestRunner` defines the following methods:
 
 
+   .. method:: report_skip(out, test, example)
+
+      Report that the given example was skipped.  This method is provided to
+      allow subclasses of :class:`DocTestRunner` to customize their output; it
+      should not be called directly.
+
+      *example* is the example about to be processed.  *test* is the test
+      containing *example*.  *out* is the output function that was passed to
+      :meth:`DocTestRunner.run`.
+
+      .. versionadded:: next
+
+
    .. method:: report_start(out, test, example)
 
       Report that the test runner is about to process the given example. This method
       is provided to allow subclasses of :class:`DocTestRunner` to customize their
       output; it should not be called directly.
 
       *example* is the example about to be processed.  *test* is the test
-      *containing example*.  *out* is the output function that was passed to
+      containing *example*.  *out* is the output function that was passed to
       :meth:`DocTestRunner.run`.
 
 

Lib/doctest.py

@@ -101,15 +101,14 @@ def _test():
 import re
 import sys
 import traceback
+import types
 import unittest
 from io import StringIO, IncrementalNewlineDecoder
 from collections import namedtuple
 import _colorize  # Used in doctests
 from _colorize import ANSIColors, can_colorize
 
 
-__unittest = True
-
 class TestResults(namedtuple('TestResults', 'failed attempted')):
     def __new__(cls, failed, attempted, *, skipped=0):
         results = super().__new__(cls, failed, attempted)
@@ -387,7 +386,7 @@ def __init__(self, out):
         self.__out = out
         self.__debugger_used = False
         # do not play signal games in the pdb
-        pdb.Pdb.__init__(self, stdout=out, nosigint=True)
+        super().__init__(stdout=out, nosigint=True)
         # still use input() to get user input
         self.use_rawinput = 1
 
@@ -1280,6 +1279,11 @@ def __init__(self, checker=None, verbose=None, optionflags=0):
     # Reporting methods
     #/////////////////////////////////////////////////////////////////
 
+    def report_skip(self, out, test, example):
+        """
+        Report that the given example was skipped.
+        """
+
     def report_start(self, out, test, example):
         """
         Report that the test runner is about to process the given
@@ -1377,6 +1381,8 @@ def __run(self, test, compileflags, out):
 
             # If 'SKIP' is set, then skip this example.
             if self.optionflags & SKIP:
+                if not quiet:
+                    self.report_skip(out, test, example)
                 skips += 1
                 continue
 
@@ -2274,12 +2280,63 @@ def set_unittest_reportflags(flags):
     return old
 
 
+class _DocTestCaseRunner(DocTestRunner):
+
+    def __init__(self, *args, test_case, test_result, **kwargs):
+        super().__init__(*args, **kwargs)
+        self._test_case = test_case
+        self._test_result = test_result
+        self._examplenum = 0
+
+    def _subTest(self):
+        subtest = unittest.case._SubTest(self._test_case, str(self._examplenum), {})
+        self._examplenum += 1
+        return subtest
+
+    def report_skip(self, out, test, example):
+        unittest.case._addSkip(self._test_result, self._subTest(), '')
+
+    def report_success(self, out, test, example, got):
+        self._test_result.addSubTest(self._test_case, self._subTest(), None)
+
+    def report_unexpected_exception(self, out, test, example, exc_info):
+        tb = self._add_traceback(exc_info[2], test, example)
+        exc_info = (*exc_info[:2], tb)
+        self._test_result.addSubTest(self._test_case, self._subTest(), exc_info)
+
+    def report_failure(self, out, test, example, got):
+        msg = ('Failed example:\n' + _indent(example.source) +
+            self._checker.output_difference(example, got, self.optionflags).rstrip('\n'))
+        exc = self._test_case.failureException(msg)
+        tb = self._add_traceback(None, test, example)
+        exc_info = (type(exc), exc, tb)
+        self._test_result.addSubTest(self._test_case, self._subTest(), exc_info)
+
+    def _add_traceback(self, traceback, test, example):
+        if test.lineno is None or example.lineno is None:
+            lineno = None
+        else:
+            lineno = test.lineno + example.lineno + 1
+        return types.SimpleNamespace(
+            tb_frame = types.SimpleNamespace(
+                f_globals=test.globs,
+                f_code=types.SimpleNamespace(
+                    co_filename=test.filename,
+                    co_name=test.name,
+                ),
+            ),
+            tb_next = traceback,
+            tb_lasti = -1,
+            tb_lineno = lineno,
+        )
+
+
 class DocTestCase(unittest.TestCase):
 
     def __init__(self, test, optionflags=0, setUp=None, tearDown=None,
                  checker=None):
 
-        unittest.TestCase.__init__(self)
+        super().__init__()
         self._dt_optionflags = optionflags
         self._dt_checker = checker
         self._dt_test = test
@@ -2303,30 +2360,28 @@ def tearDown(self):
         test.globs.clear()
         test.globs.update(self._dt_globs)
 
+    def run(self, result=None):
+        self._test_result = result
+        return super().run(result)
+
     def runTest(self):
         test = self._dt_test
-        old = sys.stdout
-        new = StringIO()
         optionflags = self._dt_optionflags
+        result = self._test_result
 
         if not (optionflags & REPORTING_FLAGS):
             # The option flags don't include any reporting flags,
             # so add the default reporting flags
             optionflags |= _unittest_reportflags
+        if getattr(result, 'failfast', False):
+            optionflags |= FAIL_FAST
 
-        runner = DocTestRunner(optionflags=optionflags,
-                               checker=self._dt_checker, verbose=False)
-
-        try:
-            runner.DIVIDER = "-"*70
-            results = runner.run(test, out=new.write, clear_globs=False)
-            if results.skipped == results.attempted:
-                raise unittest.SkipTest("all examples were skipped")
-        finally:
-            sys.stdout = old
-
-        if results.failed:
-            raise self.failureException(self.format_failure(new.getvalue().rstrip('\n')))
+        runner = _DocTestCaseRunner(optionflags=optionflags,
+                               checker=self._dt_checker, verbose=False,
+                               test_case=self, test_result=result)
+        results = runner.run(test, clear_globs=False)
+        if results.skipped == results.attempted:
+            raise unittest.SkipTest("all examples were skipped")
 
     def format_failure(self, err):
         test = self._dt_test
@@ -2441,7 +2496,7 @@ def shortDescription(self):
 class SkipDocTestCase(DocTestCase):
     def __init__(self, module):
         self.module = module
-        DocTestCase.__init__(self, None)
+        super().__init__(None)
 
     def setUp(self):
         self.skipTest("DocTestSuite will not work with -O2 and above")