Incorrect mixture of exit code and jUnit report when pytest.exit is called and a fixture teardown fails

[Hawk777]

Consider the following file:

import pytest

@pytest.fixture(scope="function")
def myfixture():
    print("enter fixture")
    yield
    assert False, "fixture fails in teardown"

def test_foo(myfixture):
    pytest.exit("foo")

Run this with pytest --junit-xml=junit.xml. The exit code is 1, which, per the documentation, means “some of the tests failed”. However, if you examine junit.xml, it does not record any failed tests! I’m not sure whether the exit code ought to be 2 (because of the pytest.exit call) or whether the test teardown ought to be recorded as a failure in junit.xml (because of the assertion failure, and the exit code remain 1 in this case), and I don’t really care that much which one happens, but it ought to be one or the other.

This happened on Pytest 6.2.4 under Linux.

[asottile]

pytest.exit is meant to be "uncleanly tear down the process" -- I'd be surprised if any useful junit is written in that case (I don't expect it to). pytest.exit isn't meant to be used in general (more for plugins, or extremely exceptional cases) -- pytest.fail is probably more appropriate or just assert.

[nicoddemus]

I debugged this, junitxml writes the file during pytest_sessionfinish, which is called even when pytest.exit() is executed.

However junitxml never receives the pytest_runtest_reportlog hook with the fixture assertion failure, because of the order pytest_sessionfinish is called: when executing the teardown procedure in wrap_session, first junitxml.pytest_sessionfinish is called, which writes out the XML file, and later runner.pytest_sessionfinish is called, which tear downs the fixture and issues the pytest_runtest_reportlog with the fixture failure.

pytest.exit isn't meant to be used in general (more for plugins, or extremely exceptional cases) -- pytest.fail is probably more appropriate or just assert.

User code can definitely call it (not only plugins), however as @asottile mentions, it is more of an "unclean exit" so some hooks might not get called.

I don't see a clean solution to this, I'm afraid.

[Hawk777]

I realize pytest.exit isn’t the nicest and cleanest thing to use, but in this specific case, I think it’s appropriate: I call it if a particular test fails in a particular way that leaves external hardware in a state where the rest of the scheduled tests can’t usefully run, and their results would be misleading if they were to do so.

It seems to me that an exit due to pytest.exit probably ought to be 2 rather than 1 in this case, since pytest.exit is more important than a fixture failing; however, I don’t know how easy it is to plumb the necessary information through to the place where the process exit code is calculated as I’m not very familiar with pytest’s internals.