Allow pytest.raises cooperate with ExceptionGroups

[elupus]

What's the problem this feature will solve?

Let pytest.raises() handle ExceptionGroup unwrapping, similar to what an "except*" clause would do. The following is valid python code that will catch the ValueError exception:

try:
  raise ExceptionGroup("", [ValueError])
except* ValueError:
  pass

However pytest.raises will not manage this, and will fail with catching the ExceptionGroup instead.

with pytest.raises(ValueError):
  raise ExceptionGroup("", [ValueError])

Describe the solution you'd like

Assert some code throws an exception or an exception as part of an exception group.

anyio/trio will now always wrap exception in ExceptionGroups when raised from taskgroups. This leads to silly checks like: https://github.com/agronholm/anyio/blob/3f1eca1addcd782e2347350a6ddb2ad2b80c6354/tests/test_taskgroups.py#L279C1-L287C31 to unwrap the exceptiongroup.

A basic solution to handle this (without the bells and whistles) would be:

@contextlib.contextmanager
def raises_nested(exc):
    try:
        yield
    except* exc:
        pass
    else:
        raise AssertionError("Did not raise expected exception")

Alternative Solutions

Additional context

[Zac-HD]

Related to #10441 which was marked as fixed in #11424 (not yet released). The main blocker here is someone designing an interface which:

• is convenient to use, and simple in simple cases
• doesn't only handle simple cases
• doesn't encourage users to ignore e.g. the nesting depth of their exception groups or the possibility of multiple unrelated errors

We're generally happy to review suggestions, if you'd like to propose something!

[elupus]

I suggested something above. That test generally matches the semantics of except*

[elupus]

Okey it's simple, it does ignore nest level. But that is one very common need.

Ps You generally don't use raises () to test for exception chains with the old api. raise A() from b.

[jakkdl]

Related to #10441 which was marked as fixed in #11424 (not yet released). The main blocker here is someone designing an interface which:

• is convenient to use, and simple in simple cases

• doesn't only handle simple cases

• doesn't encourage users to ignore e.g. the nesting depth of their exception groups or the possibility of multiple unrelated errors

We're generally happy to review suggestions, if you'd like to propose something!

I'm not sure whether this should go in here, or in #10441 (which was maybe closed prematurely), or in a new issue - but when writing a helper for Trio I went through a couple ideas for ways to solve this - and I think my favorite one is introducing a class ExpectedExceptionGroup that pytest.raises can accept an instance of, in place of a type[Exception].

Example usage:

from pytest import ExpectedExceptionGroup

with pytest.raises(ExpectedExceptionGroup((ValueError,))):
  ...

supports nested structures:

with pytest.raises(ExpectedExceptionGroup(ExpectedExceptionGroup((KeyError,)), RuntimeError)):
  ...

and works with the current feature of passing a tuple to pytest.raises when you expect one of several errors - though maybe not in a super elegant way if you're nested several layers deep:

with pytest.raises(
  (
    ExpectedExceptionGroup(KeyError),
    ExpectedExceptionGroup(ValueError),
    RuntimeError,
  )
):
  ...

Can additionally work in keyword args for matching message and/or note.

with pytest.raises(ExpectedExceptionGroup((KeyError, ValueError), message="foo", note="bar")):
  ...

and possibly also accept instances of exceptions, to be able to match the message in sub-exceptions

with pytest.raises(ExpectedExceptionGroup((KeyError(".* very bad"), RuntimeError)):
  ...

This would fulfill all of the requested bullet points, be a minimal change to the API, and shouldn't be very complicated to implement. To experiment outside of pytest (as requested in #10441 (comment)) should be somewhat doable - if a bit messy*

*can duplicate the functionality of pytest.raises,
or monkey patch _pytest.python_api.RaisesContext.__exit__,
or possibly with https://docs.python.org/3/library/abc.html#abc.ABCMeta.__subclasshook__ to trick https://github.com/pytest-dev/pytest/blob/fdb8bbf15499fc3dfe3547822df7a38fcf103538/src/_pytest/python_api.py#L1017C3-L1017C3 )

[RonnyPfannschmidt]

A while back I discussed the idea of matchers with @asottile

I believe having a matcher as single argument beats making a ever growing complexity of the raises arguments

[jakkdl]

raises is quite messy, and would be more so with my suggestion (though most of the complexity would lie in ExpectedExceptionGroup - not in raises), but I think pytest needs support for exception groups sooner rather than later. I'm all for a clean and beautiful matcher argument, but if the above suggestion is acceptable I'll write up a PR.

[Zac-HD]

The extension to pytest.raises() mostly works for me - I like the idea of having a matcher object you pass to raises(), but the interface feels a little off to me.

We're passing inner exception instances, but interpreting their __str__ (which is usually but not always equal to the message string) as a regex pattern to search by, instead of the usual interpretation as a string. Could we have an alternative form where regex patterns are explicitly passed via match= arguments? I think that having one new keyword-only argument to raises() is going to be easier on users than introducing a new ExpectedExceptionGroup type. Concretely, this might look like:

with pytest.raises(
    group=(  # asserts that we raise a (subclass of) `BaseExceptionGroup`, with exceptions matching the sequence given:
        pytest.raises(ValueError, match="vmsg"),
        ..., # Ellipsis literal to indicate "one or more exceptions omitted"
        AssertionError,  # also valid, ignores the message
    ),
    match="message",  # interpreted as matching the message (and notes, if any) of the ExceptionGroup
):
    raise ExceptionGroup("message", [ValueError("vmsg"), KeyError("kmsg"), AssertionError()])

Thoughts?

[elupus]

How does this proposal handle the case of matching behaviour of except*, which does not care about nesting level of group?

try:
  raise ExceptionGroup([ExceptionGroup([ValueError(0])
except* ValueError:

try:
  raise ExceptionGroup([ValueError(0])
except* ValueError:

The nesting level is an implementation detail that an API user does not care about.

I would expect to be able to use raises to catch both above cases, same as you catch both cases with the same exception handler.

[RonnyPfannschmidt]

@Zac-HD i find the extension quite hard to grasp, unless we externalize the complexity id rather see a new entrypoint to exceptiongroups

the raises api already is overcomplicated

what you pass in as group is basically a matcher without being explicit or defined

i say NO to that type of compexitx, its hard to read, hard to grasp and hard to define propperly

[jakkdl]

How does this proposal handle the case of matching behaviour of except*, which does not care about nesting level of group?

good question - I was not thinking of that use case ~at all, since I was thinking about it from the POV of a library maintainer that cares about the nesting level. I'll see if I can add a kwarg that toggles this.
If we want to further mirror except* behaviour then

with pytest.raises(ExpectedExceptionGroup(ValueError)):
  raise ValueError

should maybe work as well.

.
Likewise my current implementation (which you can see at https://github.com/python-trio/trio/blob/84fb527cf9d9a97baf11c7437870f52c6b318992/src/trio/testing/_exceptiongroup_util.py) also checks that exactly the specified exceptions are encountered, and no others, unlike #11424 which will ignore any extraneous errors. I'll try to add a kwarg that toggles that behaviour as well - although I'm unsure if that is a good pattern - if your program is going from raising ExceptionGroup("", (ValueError,)) to raising ExceptionGroup("", (ValueError, SyntaxError)) that's probably something the test suite should make you aware of.

[jakkdl]

We're passing inner exception instances, but interpreting their __str__ (which is usually but not always equal to the message string) as a regex pattern to search by, instead of the usual interpretation as a string. Could we have an alternative form where regex patterns are explicitly passed via match= arguments?

Yeah this isn't great, I encountered problems when using it with an OSError - where str(OSError(5, "foo")) == '[Errno 5] foo' and those [s get interpreted as regex patterns. Checking args doesn't work great with regex's either.

Though a small wrapper around the exception, instead of instantiating it, seems like a good idea. Could be something like this:

with pytest.raises(
  ExpectedExceptionGroup(
    ValueError,
    pytest.matcher(ValueError, match=r'foo[bar]'),
    # can add a fully customizable kwarg that takes a callable
    pytest.matcher(OSError, check=lambda x: x.bar == 7),

    # it could even make the pos arg skippable
    # e.g. when you want to check `== type` instead of using `isinstance(..., type)`
    pytest.matcher(check=lambda x: type(x) == my_error_generator()),
  )
):

[jakkdl]

I think that having one new keyword-only argument to raises() is going to be easier on users than introducing a new ExpectedExceptionGroup type.

Concretely, this might look like:

with pytest.raises(
    group=(  # asserts that we raise a (subclass of) `BaseExceptionGroup`, with exceptions matching the sequence given:
        pytest.raises(ValueError, match="vmsg"),
        ..., # Ellipsis literal to indicate "one or more exceptions omitted"
        AssertionError,  # also valid, ignores the message
    ),
    match="message",  # interpreted as matching the message (and notes, if any) of the ExceptionGroup
):
    raise ExceptionGroup("message", [ValueError("vmsg"), KeyError("kmsg"), AssertionError()])

Thoughts?

Seems messy to specify nested exceptiongroups - would be something like this?

with pytest.raises(
  group=(
    pytest.raises(
      group=(
        pytest.raises(ValueError, match="vmsg"),
      )
    )
  )
):
  ...

[jakkdl]

If we want to step away from pytest.raises, and also not have a type, - an alternative could be something like

with pytest.groupraises(
  ValueError,
  pytest.groupraises(
    SyntaxError,
  )
):
  ...

or as a type

with pytest.RaisedGroup(
  ValueError,
  pytest.RaisedGroup(
    SyntaxError,
  )
):
  ...