pytest.fail() within fixture with __tracebackhide__ produces confusing output

[okken]

• a detailed description of the bug or problem you are having

If a fixture has __tracebackhide__ = True and fails:

• Failing with pytest.fail() with pytrace=False -> works great
• Failing with assert -> works ok, but still produces traceback
• Failing with pytest.fail() -> produces traceback into def fail()

This is confusing.

In the assert case, the traceback is actually shown anyway, which isn't terrible but still strange.
Also, the message from assert doesn't show up in the summary report, which probably isn't good either:

ERROR test_bug.py::test_good <--- { no message }
ERROR test_bug.py::test_not_terrible - assert fail message
ERROR test_bug.py::test_bad - Failed: fail with pytrace True message

• test_good - Calling pytest.fail("message", pytrace=False)
  • ✔️ traceback avoided
  • ✔️ message shows up in summary
• test_not_terrible - Calling assert False, "message"
  • ❌ traceback not avoided
  • ✔️ traceback shows fixture, which is better than anything else, I guess.
  • ❌ message doesn't show up in summary
• test_bad - Calling pytest.fail("message", pytrace=True)
  • ❌ traceback not avoided
  • ❌ traceback traces pytest.fail() instead of test or fixture.
  • ✔️ message shows up in summary

I see that if I call pytest.fail() with pytrace=False, I get the expected output of no traceback.
But I think even with pytest=True (default), showing the tb of fail() is a bit bizarre.

• output of pip list from the virtual environment you are using

$ pip list
Package    Version
---------- -------
attrs      22.2.0
colorama   0.4.6
iniconfig  2.0.0
packaging  23.0
pip        23.0.1
pluggy     1.0.0
pytest     7.2.1
setuptools 65.5.0

• pytest and operating system versions
  pytest 7.2.1, windows 10

• minimal example if possible

test_bug.py

import pytest

@pytest.fixture()
def fail_with_assert():
    __tracebackhide__ = True
    assert False, "assert fail message"

@pytest.fixture()
def fail_pytrace_true():
    __tracebackhide__ = True
    pytest.fail("fail with pytrace True message")

@pytest.fixture()
def fail_pytrace_false():
    __tracebackhide__ = True
    pytest.fail("fail with pytrace False message", pytrace=False)

def test_good(fail_pytrace_false):
    ...

def test_not_terrible(fail_with_assert):
    ...

def test_bad(fail_pytrace_true):
    ...

results

$ pytest test_bug.py
============================= test session starts =============================
platform win32 -- Python 3.11.0, pytest-7.2.1, pluggy-1.0.0
rootdir: C:\Users\okken\projects\example
collected 3 items

test_bug.py EEE                                                          [100%]

=================================== ERRORS ====================================
_________________________ ERROR at setup of test_good _________________________
fail with pytrace False message
_____________________ ERROR at setup of test_not_terrible _____________________

    @pytest.fixture()
    def fail_with_assert():
        __tracebackhide__ = True
>       assert False, "assert fail message"
E       assert fail message
E       assert False

test_bug.py:7: AssertionError
_________________________ ERROR at setup of test_bad __________________________

reason = 'fail with pytrace True message', pytrace = True, msg = None

    @_with_exception(Failed)
    def fail(reason: str = "", pytrace: bool = True, msg: Optional[str] = None) -> NoReturn:
        """Explicitly fail an executing test with the given message.

        :param reason:
            The message to show the user as reason for the failure.

        :param pytrace:
            If False, msg represents the full failure information and no
            python traceback will be reported.

        :param msg:
            Same as ``reason``, but deprecated. Will be removed in a future version, use ``reason`` instead.
        """
        __tracebackhide__ = True
        reason = _resolve_msg_to_reason("fail", reason, msg)
>       raise Failed(msg=reason, pytrace=pytrace)
E       Failed: fail with pytrace True message

venv\Lib\site-packages\_pytest\outcomes.py:194: Failed
=========================== short test summary info ===========================
ERROR test_bug.py::test_good
ERROR test_bug.py::test_not_terrible - assert fail message
ERROR test_bug.py::test_bad - Failed: fail with pytrace True message
============================== 3 errors in 0.14s ==============================