JUnit report is broken, if markTestSkipped is used within setUpBeforeClass

[thirsch]

Q	A
PHPUnit version	10.1.2
PHP version	8.2.5
Installation Method	Composer

Summary

Mandatory JUnit-Report attribute tests in the first testsuite-Node is missing, if setUpBeforeClass uses e.g. self::markTestSkipped(). This is working fine with PHPUnit 9. As you can see in the examples, PHPUnit 9 handled the case also differently in the specific testsuite of the file containing the method call.

We are skipping the test class using setUpBeforeClass instead of using the setUp method, because there is a dataprovider generating many test cases that should not even be generated if skipped.

I've just changed our tests and just wanted to let you know what I've discovered.

Current behavior

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="">
    <testsuite name="Application Test Suite" tests="0" assertions="0" errors="0" failures="0" skipped="0" time="0.000000">
      <testsuite name="SimpleTest" file="/src/tests/SimpleTest.php" tests="0" assertions="0" errors="0" failures="0" skipped="0" time="0.000000"/>
    </testsuite>
  </testsuite>
</testsuites>

How to reproduce

Use a simple test with the self::markTestSkipped() method call, e.g.:

<?php 

class SimpleTest extends TestCase
{
    public static function setUpBeforeClass(): void
    {
        self::markTestSkipped('Skip all');
    }

    public function test(): void
    {
        self::assertTrue(true);
    }
}

Expected behavior

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
  <testsuite name="" tests="1" assertions="0" errors="0" warnings="0" failures="0" skipped="1" time="0.000000">
    <testsuite name="Application Test Suite" tests="1" assertions="0" errors="0" warnings="0" failures="0" skipped="1" time="0.000000">
      <testsuite name="SimpleTest" file="/src/tests/SimpleTest.php" tests="1" assertions="0" errors="0" warnings="0" failures="0" skipped="1" time="0.000000">
        <testcase name="test" class="SimpleTest" classname="SimpleTest" file="/src/tests/SimpleTest.php" line="12" assertions="0" time="0.000000">
          <skipped/>
        </testcase>
      </testsuite>
    </testsuite>
  </testsuite>
</testsuites>

[asika32764]

Teamcity has same issue. This breaks phpstorm's test results.

Test in PHP 8.4.1 / PhpUnit 11.4.4

[asika32764]

After a quick investigate, if skip in setupBeforeClass() phpunit will not call testSkipped() hook.

// TeamCityLogger

    // ...

    public function testSkipped(Skipped $event): void
    {
        // Only run if skip in test method
        // If skip in setupBeforeClass(), phpunit will not run this hook.


        
        if ($this->time === null) {
            $this->time = $event->telemetryInfo()->time();
        }

        $parameters = [
            'name'    => $event->test()->name(),
            'message' => $event->message(),
        ];

        // ...

[asika32764]

I think it because there are missing TestSuiteSkippedSubscriber, but I have no idea how to make it works.

[sebastianbergmann]

After a quick investigate, if skip in setupBeforeClass() phpunit will not call testSkipped() hook.

Which is correct as we never get to the point where an individual test method would be skipped. The TestSuiteSkipped event, which is intended to be emitted in this case, is emitted.

The loggers mentioned here, JUnit and TeamCity, need to also subscribe to this event and handle it accordingly.

[asika32764]

I compare with phpunit 9

    public static function setUpBeforeClass(): void
    {
        if (...) {
            self::markTestSkipped('XCache is not enabled on this system.');
        }
    }

The correct output should contains testIgnored for test class and every test methods, and a testSuiteFinished at class ending.

...

##teamcity[testSuiteStarted name='Windwalker\Session\Test\Handler\XcacheHandlerTest' locationHint='php_qn:///private/var/www/windwalker/ww3/src/Session/Test/Handler/XcacheHandlerTest.php::\Windwalker\Session\Test\Handler\XcacheHandlerTest' flowId='68871']

##teamcity[testStarted name='testIsSupported' locationHint='php_qn:///private/var/www/windwalker/ww3/src/Session/Test/Handler/XcacheHandlerTest.php::\Windwalker\Session\Test\Handler\XcacheHandlerTest::testIsSupported' flowId='68871']

##teamcity[testIgnored name='testIsSupported' message='XCache is not enabled on this system.' details=' /private/var/www/windwalker/ww3/src/Session/Test/Handler/XcacheHandlerTest.php:30|n ' duration='0' flowId='68871']

##teamcity[testFinished name='testIsSupported' duration='0' flowId='68871']

##teamcity[testStarted name='testRead' locationHint='php_qn:///private/var/www/windwalker/ww3/src/Session/Test/Handler/XcacheHandlerTest.php::\Windwalker\Session\Test\Handler\XcacheHandlerTest::testRead' flowId='68871']

##teamcity[testIgnored name='testRead' message='Test skipped because of an error in hook method' details=' ' duration='0' flowId='68871']

##teamcity[testFinished name='testRead' duration='0' flowId='68871']

... Repeat for every methods

##teamcity[testSuiteFinished name='Windwalker\Session\Test\Handler\XcacheHandlerTest' flowId='68871']

At phpunit >= 10, it only contains testSuiteStarted, no Finished log, so phpstorm output will broke.

[sebastianbergmann]

Any comparison with PHPUnit 9 is pointless as PHPUnit >= 10 works completely different internally.

[asika32764]

Yes but I'm not comparing about the code, I think the difference about the logging output is the main cause for this issue.