Terrible user experience if test_exceptions.ExceptionTests.test_recursion_normalizing_infinite_exception fails

[AlexWaygood]

Bug report

Bug description:

If test_exceptions.ExceptionTests.test_recursion_normalizing_infinite_exception fails, a huge amount of output is printed to the terminal, which is a pretty terrible user experience. The full output is here: exception_tb.txt.

To reproduce, run FORCE_COLOR ./python.exe -m unittest -v test.test_exceptions.ExceptionTests.test_recursion_normalizing_infinite_exception.

(Note that this issue isn't about the test failing when FORCE_COLOR is set. #117605 is about that. This issue is about the poor user experience if the test fails.)

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-117606: Truncate extremely long error message in test_exceptions #117670
• [3.12] gh-117606: Truncate extremely long error message in test_exceptions (GH-117670) #117745

[nineteendo]

Would this be acceptable? It will tell you RecursionError: maximum recursion depth exceeded is not in the error message, which is really all you need to know.

-self.assertIn(b'RecursionError: maximum recursion depth exceeded', err)
+# gh-117606: Don't use assertIn! The error message can be really long.
+self.assertTrue(b'RecursionError: maximum recursion depth exceeded' in err)

Example output:

import unittest

class MyTestCase(unittest.TestCase):
    def test(self):
        err = "good"
        self.assertTrue("bad" in err)

unittest.main()

F
======================================================================
FAIL: test (__main__.MyTestCase.test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\Users\wanne\Desktop\test.py", line 6, in test
    self.assertTrue("bad" in err)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

[AlexWaygood]

What about something like this, so that you can still see the traceback that you actually got, but it's truncated so that it can reasonably fit on your screen?

diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py
index 36fd89dbb8..000337d08f 100644
--- a/Lib/test/test_exceptions.py
+++ b/Lib/test/test_exceptions.py
@@ -1451,7 +1451,8 @@ def test_recursion_normalizing_infinite_exception(self):
         """
         rc, out, err = script_helper.assert_python_failure("-c", code)
         self.assertEqual(rc, 1)
-        self.assertIn(b'RecursionError: maximum recursion depth exceeded', err)
+        expected = b'RecursionError: maximum recursion depth exceeded'
+        self.assertTrue(expected in err, msg=f"{expected} not found in {err[:3_000]!r}... (truncated)")
         self.assertIn(b'Done.', out)

Output:

(main)⚡ % FORCE_COLOR=1 ./python.exe -m unittest -v test.test_exceptions.ExceptionTests.test_recursion_normalizing_infinite_exception
test_recursion_normalizing_infinite_exception (test.test_exceptions.ExceptionTests.test_recursion_normalizing_infinite_exception) ... FAIL

======================================================================
FAIL: test_recursion_normalizing_infinite_exception (test.test_exceptions.ExceptionTests.test_recursion_normalizing_infinite_exception)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/alexw/dev/cpython/Lib/test/test_exceptions.py", line 1455, in test_recursion_normalizing_infinite_exception
    self.assertTrue(expected in err, msg=f"{expected} not found in {err[:3_000]!r}... (truncated)")
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: False is not true : b'RecursionError: maximum recursion depth exceeded' not found in b'Traceback (most recent call last):\n  File \x1b[35m"<string>"\x1b[0m, line \x1b[35m4\x1b[0m, in \x1b[35m<module>\x1b[0m\n    raise _testcapi.RecursingInfinitelyError\n\x1b[1;35mRecursionError\x1b[0m: \x1b[35mmaximum recursion depth exceeded while calling a Python object\x1b[0m\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNormalization failed: type=RecursingInfinitelyError args=<NULL>\nNor'... (truncated)

----------------------------------------------------------------------
Ran 1 test in 0.046s

[nineteendo]

Yeah, that would be even better, and it doesn't need a comment to explain the use of self.assertTrue().
Do you make a pull request, or I?

[AlexWaygood]

Feel free to make a PR! :-)