TestDox printer does not consider that issues can be suppressed by attribute, baseline, source location, or `@` operator

[Bilge]

Q	A
PHPUnit version	10.5.24
PHP version	8.2.15
Installation Method	Composer

Summary

Testdox prints seemingly-random warning icons (⚠) next to some tests when a test suite passes with 100% success. I could not find the documented meaning behind this anywhere, and searching the code for that emoji did not yield any results, so I'm assuming it's either a bug or undocumented feature, but either way it is very confusing and was not present in PHPUnit 9.

Current behavior

Testdox will print warnings next to some tests for no discernable reason whatsoever.

Patreon Link
 ✔ Patreon link with Follower·only
 ✔ Patreon link with Non-member
 ⚠ Patreon link with Tier·2
 ✔ Patreon link with Former·patron
 ✔ Patreon link with Tier·1
 ✔ Patreon link with Cancelled·but·active·member
 ⚠ Patreon link with T1·->·T2·upgrade
 ⚠ Patreon link with Tier·3

OK (131 tests, 511 assertions)

How to reproduce

Run some tests?

Expected behavior

If the summary shows 100% OK, I would not expect to see warning icons in the summary output. If there are warnings, I expect to see details of what those warnings are somewhere.

[sebastianbergmann]

PHPUnit 10 introduced a clear separation between the outcome (errored, failed, incomplete, skipped, or passed) of a test and the issues (considered risky, triggered a warning, ...) of a test.

Consider this example:

<?php declare(strict_types=1);
use PHPUnit\Framework\TestCase;

final class ExampleTest extends TestCase
{
    public function testSucceeds(): void
    {
        $this->assertTrue(true);
    }

    public function testFails(): void
    {
        $this->assertTrue(false);
    }

    public function testRisky(): void
    {
    }
}

Running the test shown above with PHPUnit 10 and the --testdox CLI option yields the output shown below:

❯ phpunit --no-progress --testdox ExampleTest.php
PHPUnit 10.5.24 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.8

Time: 00:00.008, Memory: 8.00 MB

Example
 ✔ Succeeds
 ✘ Fails
   │
   │ Failed asserting that false is true.
   │
   │ /home/sb/ExampleTest.php:13
   │
 ⚠ Risky

There was 1 risky test:

1) ExampleTest::testRisky
This test did not perform any assertions

/home/sb/ExampleTest.php:16

FAILURES!
Tests: 3, Assertions: 2, Failures: 1, Risky: 1.

In the example above, the ⚠ icon is used to indicate that the test in question was considered risky.

Running the same test with PHPUnit 9 and the --testdox CLI option yields the output shown below:

❯ phpunit --testdox ExampleTest.php
PHPUnit 9.6.19 by Sebastian Bergmann and contributors.

Example
 ✔ Succeeds
 ✘ Fails
   │
   │ Failed asserting that false is true.
   │
   │ /home/sb/ExampleTest.php:13
   │

 ☢ Risky

Time: 00:00.034, Memory: 6.00 MB

Summary of non-successful tests:

Example
 ✘ Fails
   │
   │ Failed asserting that false is true.
   │
   │ /home/sb/ExampleTest.php:13
   │

 ☢ Risky

FAILURES!
Tests: 3, Assertions: 2, Failures: 1, Risky: 1.

As you can see, PHPUnit 9 had the same TestDox output (it used ☢ instead of ⚠ to indicate risky tests, though).

I am closing this issue now as this works as intended. I have opened sebastianbergmann/phpunit-documentation-english#360 to track improvements to the documentation of TestDox output.

[Bilge]

This is not just a documentation issue. As you can see from the summary, there are no risky tests:

OK (131 tests, 511 assertions)

Moreover, there were no tests marked risky in PHPUnit 9. These warning icons suddenly appear in PHPUnit 10 without apparent cause.

[gartner]

I see this behavior too:

Plant Images Resource (App\Tests\Functional\PlantImagesResource)
✔ Anonymous cannot access
⚠ Cannot create new plant image with invalid data
⚠ Can create new plant image from url
⚠ Can create new plant image from base 64
⚠ Can create new plant image from base 64 without origin url
⚠ Create with existing origin url returns existing instead

Pricegroup (App\Tests\Functional\Pricegroup)
✔ Anonymus cannot create pricegroup
✔ Create pricegroup with All·valid·data
✔ Create pricegroup with 3-digit·color·code·is·valid
✔ Create pricegroup with 3-digit·color·code·without·#·is·valid
✔ Create pricegroup with 4-digit·color·code·is·invalid
✔ Create pricegroup with 5-digit·color·code·without·#·is·invalid
✔ Create pricegroup with Engros·as·float·is·invalid
✔ Create pricegroup with Detail·as·float·is·invalid
✔ Create pricegroup with Non-hex·digits·in·color·is·invalid

Print Text Label (App\Tests\Functional\PrintTextLabel)
✔ Cannot print without data
✔ Print with data

User (App\Tests\Functional\User)
✔ Default user can login

zettle Service (App\Tests\Service\zettleService)
✔ Get bearer

FAILURES!
Tests: 34, Assertions: 55, Failures: 2.

No risky tests, still an exclamation icon on some tests.

But it gets worse.
Immediately after running the above test, I ran it again. No change to code, or anything else. and now I get this:

Plant Images Resource (App\Tests\Functional\PlantImagesResource)
⚠ Anonymous cannot access
⚠ Cannot create new plant image with invalid data
⚠ Can create new plant image from url
⚠ Can create new plant image from base 64
⚠ Can create new plant image from base 64 without origin url
⚠ Create with existing origin url returns existing instead

Pricegroup (App\Tests\Functional\Pricegroup)
⚠ Anonymus cannot create pricegroup
⚠ Create pricegroup with All·valid·data
⚠ Create pricegroup with 3-digit·color·code·is·valid
⚠ Create pricegroup with 3-digit·color·code·without·#·is·valid
⚠ Create pricegroup with 4-digit·color·code·is·invalid
⚠ Create pricegroup with 5-digit·color·code·without·#·is·invalid
⚠ Create pricegroup with Engros·as·float·is·invalid
⚠ Create pricegroup with Detail·as·float·is·invalid
⚠ Create pricegroup with Non-hex·digits·in·color·is·invalid

Print Text Label (App\Tests\Functional\PrintTextLabel)
⚠ Cannot print without data
⚠ Print with data

User (App\Tests\Functional\User)
⚠ Default user can login

zettle Service (App\Tests\Service\zettleService)
✔ Get bearer

FAILURES!
Tests: 34, Assertions: 55, Failures: 2.

This is run using phpunit 10.5.27 in a Symfony app.

[Bilge]

In case it matters, mine is also a Symfony app. In any case, I think this issue needs to be reopened as it seems clear something is raising a risky event but nothing is forwarding the reason to the end-user, making it entirely opaque why these tests are being so marked. More importantly, only the testdox printer is sensitive to these events; the summary does NOT report any tests as being risky.

[marien-probesys]

I had the same issue: Testdox shows warnings, but I had no warnings in the summary. I dig a bit the issue and it appears that I used the @ symbol to silence the warnings from functions like mkdir and unlink. So for some reasons, warnings raised by these "silenced" functions are detected by Testdox, but not by the default output.

You can easily reproduce the issue with this test:

class WarningsTest extends PHPUnit\Framework\TestCase
{
    public function testWarning(): void
    {
        $folder = sys_get_temp_dir() . '/documents';
        if (is_dir($folder)) {
            // Just make sure to start with a clean state
            rmdir($folder);
        }
        mkdir($folder);

        // This will generate a PHP warning which will be silenced.
        // The default output shows nothing, while Testdox shows a warning but
        // cannot tell what it is. Removing the `@` shows the warning in both output.
        @mkdir($folder);

        $this->assertTrue(is_dir($folder));
    }
}

[jrfnl]

Came here to report the same as I noticed this on two different repos today. @sebastianbergmann Can this issue please be re-opened ?

I dig a bit the issue and it appears that I used the @ symbol to silence the warnings from functions like mkdir and unlink. So for some reasons, warnings raised by these "silenced" functions are detected by Testdox, but not by the default output.

I can confirm this. IF the error silencing operator is being used (either in the code under test or the test itself) AND a deprecation/notice/warning is being raised (and silenced), the warning icon appears.

However, if the test is then marked with #[WithoutErrorHandler], the icon disappears (but adding this attribute may not be without consequence).

I've also found another situation which triggers the appearance of the warning icon, where IMO it is inappropriate:
When expectUserDeprecationMessage() is called in a test, with or without the #[IgnoreDeprecations] attribute (makes no difference).

I've set up a relatively simple reproduction scenario, which can be checked-out here: https://github.com/jrfnl/bug-report-reproduction-scenarios/tree/phpunit/5884-demo-code

The different output of the reproduction scenarios on PHPUnit 9/10/11 can also be seen in the GH Actions workflow output: https://github.com/jrfnl/bug-report-reproduction-scenarios/actions/runs/10672517439

[Edit]: What I find most confusing about these warning symbols, is - as @Bilge has also pointed out - that there is seemingly no explanation for the appearance of the symbols (in the test output) and you have to go digging to find this issue to figure out why they appear.

[sebastianbergmann]

Thank you, Juliette, for the reproducing example. I now (finally) understand what the issue is. I am working on a fix for PHPUnit 10.5 and PHPUnit 11.3 right now.