`unittest` may incorrectly return exit code 5 when `setUpClass` raises an exception

[ueshin]

Bug report

Bug description:

In Python 3.12+, unittest returns exit code 5 when no tests are run and none are skipped. However, if a test class’s setUpClass() method raises any exception (except unittest.SkipTest; related to #106584), no tests are executed, and unittest may incorrectly return exit code 5.

This is misleading: the failure occurred due to an error in class-level setup, not because all tests were skipped.

import unittest

class BugTest(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        raise ValueError("Simulated setup failure")

    def test_example(self):
        self.assertTrue(True)

python -m unittest bug_test.py
echo $?

Actual result:

• Exit code is 5.

Expected result:

• Exit code should be 1, because an error occurred during test setup.

I think we should report the error first, then check whether no tests are run and none are skipped or not.

https://github.com/python/cpython/blob/3.12/Lib/unittest/main.py#L282-L288

Suggested:

        if self.exit:
            if not self.result.wasSuccessful():
                sys.exit(1)
            elif self.result.testsRun == 0 and len(self.result.skipped) == 0:
                sys.exit(_NO_TESTS_EXITCODE)
            else:
                sys.exit(0)

CPython versions tested on:

3.12

Operating systems tested on:

macOS

Linked PRs

• gh-136442: Fix unittest to return exit code 5 when setUpClass raises an exception #136487

[ZeroIntensity]

Sounds like a reasonable fix. Would you like to submit a PR?

[ueshin]

@ZeroIntensity Sure, I will submit a PR. Thanks!