PyTest internal error when using "except*" on an unhashable exception

[joseph-e-k]

Tl;dr: A strange exception thrown by except* from "line -1" causes PyTest to crash when trying to retrieve the line it came from. Code to reproduce:

class MyException(Exception):
    __hash__ = None


def test_broken():
    try:
        raise ExceptionGroup("Foo", [
            MyException("Bar")
        ])
    except* Exception:
        pass

And then run that test function (test_broken) with pytest.

Expected result: test succeeds.

Expected result once you know about the weird behavior of expect* (see below): test fails with a TypeError.

Actual result: pytest fails with the following traceback; test is marked as "not run".

Launching pytest with arguments test_broken.py::test_broken --no-header --no-summary -q in C:\Users\Josep\PycharmProjects\sloge\tests

============================= test session starts =============================
collecting ... collected 1 item

test_broken.py::test_broken 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\main.py", line 349, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\runner.py", line 112, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\runner.py", line 131, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\runner.py", line 222, in call_and_report
INTERNALERROR>     report: TestReport = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_callers.py", line 55, in _multicall
INTERNALERROR>     gen.send(outcome)
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\skipping.py", line 265, in pytest_runtest_makereport
INTERNALERROR>     rep = outcome.get_result()
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\pluggy\_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\runner.py", line 366, in pytest_runtest_makereport
INTERNALERROR>     return TestReport.from_item_and_call(item, call)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\reports.py", line 349, in from_item_and_call
INTERNALERROR>     longrepr = item.repr_failure(excinfo)
INTERNALERROR>                ^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\python.py", line 1823, in repr_failure
INTERNALERROR>     return self._repr_failure_py(excinfo, style=style)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\nodes.py", line 484, in _repr_failure_py
INTERNALERROR>     return excinfo.getrepr(
INTERNALERROR>            ^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\_code\code.py", line 669, in getrepr
INTERNALERROR>     return fmt.repr_excinfo(self)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\_code\code.py", line 944, in repr_excinfo
INTERNALERROR>     reprtraceback = self.repr_traceback(excinfo_)
INTERNALERROR>                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\_code\code.py", line 871, in repr_traceback
INTERNALERROR>     reprentry = self.repr_traceback_entry(entry, einfo)
INTERNALERROR>                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\_code\code.py", line 822, in repr_traceback_entry
INTERNALERROR>     s = self.get_source(source, line_index, excinfo, short=short)
INTERNALERROR>         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "C:\Users\Josep\PycharmProjects\sloge\venv\Lib\site-packages\_pytest\_code\code.py", line 755, in get_source
INTERNALERROR>     lines.append(self.flow_marker + "   " + source.lines[line_index])
INTERNALERROR>                                             ~~~~~~~~~~~~^^^^^^^^^^^^
INTERNALERROR> IndexError: list index out of range

============================ no tests ran in 0.08s ============================

---

I'm running CPython 3.11.0 for Windows (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)], and pytest 7.2.0. My complete pip list:

Package    Version
---------- -------
attrs      22.1.0
colorama   0.4.6
iniconfig  1.1.1
packaging  21.3
pip        22.3
pluggy     1.0.0
pyparsing  3.0.9
pytest     7.2.0
setuptools 65.5.0

---

I think the cause of the bug is as follows.

Python 3.11's new except* syntax has an odd quirk: it throws an exception (from within Python itself) if you use it on a try block that throws an unhashable exception. Normal try ... except blocks don't do this. For example, this works fine:

try:
    raise ExceptionGroup("Foo", [
        MyException("Bar")
    ])
except Exception:
    pass

But this:

class MyException(Exception):
    __hash__ = None

try:
    raise ExceptionGroup("Foo", [
        MyException("Bar")
    ])
except* Exception:
    pass

Produces the following unhandled exception traceback:

  + Exception Group Traceback (most recent call last):
  |   File "C:\Users\Josep\AppData\Roaming\JetBrains\PyCharmCE2022.2\scratches\scratch_61.py", line 6, in <module>
  |     raise ExceptionGroup("Foo", [
  | ExceptionGroup: Foo (1 sub-exception)
  +-+---------------- 1 ----------------
    | MyException: Bar
    +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Josep\AppData\Roaming\JetBrains\PyCharmCE2022.2\scratches\scratch_61.py", line -1, in <module>
TypeError: unhashable type: 'MyException'

I'm fairly sure this qualifies as an actual bug in the Python implementation I've got, as I couldn't find this behavior documented anywhere, but that's a ticket for a different repo.

Back to pytest. At the top of this bug report is a block of code reproducing this exception inside a test. Obviously the correct behavior would be for pytest to catch the TypeError, mark the test as failing due to an uncaught exception, and display it with full traceback - or, if that was completely impossible due to the "line -1" thing, then without one.

The most important thing is that the test should fail due to an uncaught exception, rather than pytest itself throwing an internal error and then marking the test as "not run".

[Zac-HD]

This looks like a straightforward bug in our get_source() function, which should not (anymore?) assume that line_index is in source.lines; we might need some fallback logic for the can't-find-source case. Thanks very much for the report!

[asottile]

I think you should report the TypeError to cpython -- and then there's nothing to fix here I believe

[Zac-HD]

Confirmed as an upstream bug - thanks again for the report @joseph-e-k! I'd guess that this will be fixed in Python 3.11.1 🙂

[joseph-e-k]

Thanks very much! I'll just work around it for now.