Document that `with pytest.warns()` captures all warnings, not just those it tests were emitted

[Kojoley]

I have a test that ensures a specific warning type is produced by a function, but since I switched to pytest.warns I no longer see Python 3.10 compatibility warnings in Pytest warnings summary the code was emitting before switching to pytest.warns. Is it expected behavior (in which case it contradicts https://docs.pytest.org/en/6.2.x/warnings.html#deprecationwarning-and-pendingdeprecationwarning) and how I can opt-out of hiding DeprecationWarnings by pytest.warns (there is nothing about that in the documentation https://docs.pytest.org/en/6.2.x/reference.html#pytest-warns)?

MVCE:

import pytest
import warnings

def test_warning():
    with pytest.warns(DeprecationWarning):
        with pytest.warns(UserWarning):  # catches UserWarning, but also every other type too
            warnings.warn("my warning", UserWarning)
            warnings.warn("some deprecation warning", DeprecationWarning)

================================== FAILURES ===================================
________________________________ test_warning _________________________________

    def test_warning():
>       with pytest.warns(DeprecationWarning):
E       Failed: DID NOT WARN. No warnings of type (<class 'DeprecationWarning'>,) was emitted. The list of emitted warnings is: [].

[nicoddemus]

Hi @Kojoley,

This is the expected behavior, pytest.warns acts as a net which captures all warnings inside the block, ensuring the given warning is raised, or failing otherwise.

We might add a note to https://docs.pytest.org/en/6.2.x/warnings.html#deprecationwarning-and-pendingdeprecationwarning to mention that behavior does not apply for code inside pytest.warns, what do you think?

[Kojoley]

The pytest.warns documentation does not say it will filter everything (actually it does not say it will filter anything at all), so it should be fixed too. It preferably should say how to make pytest.warns to not capture everything or an alternative solution, but I still myself have no idea how to achieve that with tools Pytest provides, could you please suggest anything?

[nicoddemus]

The pytest.warns documentation does not say it will filter everything (actually it does not say it will filter anything at all), so it should be fixed too.

True, thanks!

I still myself have no idea how to achieve that with tools Pytest provides, could you please suggest anything?

Hmm I can't think of anything right now, pytest.warns uses warnings.catch_warnings under the covers, which by design captures all warnings.

[nicoddemus]

Created #9291 to track the docs change.

[Kojoley]

If I understand correctly pytest.deprecated_call hides all warnings by design too?

Also, https://docs.pytest.org/en/6.2.x/warnings.html#warns says You can check that code raises a particular warning using func:pytest.warns, which works in a similar manner to raises: but raises does not hide all exceptions, example:

def test_raises():
    with pytest.raises(RuntimeError):              # passes,
        with pytest.raises(NotImplementedError):   # because this will not catch and hide the exception
            raise RuntimeError("some exception")

[nicoddemus]

If I understand correctly pytest.deprecated_call hides all warnings by design too?

Yes. 👍

but raises does not hide all exceptions, example

Good point, we need to mention that difference in the docs too.