Performance issue running coverage on Windows

[jrfnl]

Q	A
PHPUnit version	12.0.1
PHP version	8.4
Installation Method	PHAR (but probably Composer to)

Summary

I'm seeing a significant performance slow-down when using PHPUnit 12 for path/branch coverage versus PHPUnit 11 when running on Windows (10).

When running just one test class (filtered down), I see a difference of 3 vs 50 seconds.

The slow-down appears to be at the start, before progress is being shown.

Current behavior

$ phpunit11 -c phpunit10.xml --filter AssertObjectNotEquals

PHPUnit 11.5.7 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.4.1 with Xdebug 3.4.1
Configuration: path/to/PHPUnit-Polyfills/phpunit10.xml

.........S.....S.......                                           23 / 23 (100%)

Time: 00:03.563, Memory: 376.00 MB

OK, but some tests were skipped!
Tests: 23, Assertions: 48, Skipped: 2.

Generating code coverage report in HTML format ... done [00:00.741]


Code Coverage Report Summary:
  Classes:  4.55% (1/22)
  Methods:  4.41% (6/136)
  Paths:    5.94% (6/101)
  Branches:    5.00% (6/120)
  Lines:    1.80% (10/555)

$ phpunit12 -c phpunit10.xml --filter AssertObjectNotEquals

PHPUnit 12.0.1 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.4.1 with Xdebug 3.4.1
Configuration: path/to/PHPUnit-Polyfills/phpunit10.xml

.........S.....S.......                                           23 / 23 (100%)

Time: 00:50.153, Memory: 370.00 MB

OK, but some tests were skipped!
Tests: 23, Assertions: 48, Skipped: 2.

Generating code coverage report in HTML format ... done [00:00.422]


Code Coverage Report Summary:
  Classes:  4.55% (1/22)
  Methods:  4.41% (6/136)
  Paths:    7.32% (6/82)
  Branches:    6.19% (6/97)
  Lines:    1.80% (10/555)

How to reproduce

If I run with --debug on, it looks like the test suite "hangs" between "Test Passed" and "Test Finished"

Test Runner Execution Started (23 tests)
Test Suite Started (path/to/PHPUnit-Polyfills/phpunit10.xml, 23 tests)
Test Suite Started (PHPUnitPolyfills, 23 tests)
Test Suite Started (Yoast\PHPUnitPolyfills\Tests\Polyfills\AssertObjectNotEqualsTest, 21 tests)
Test Preparation Started (Yoast\PHPUnitPolyfills\Tests\Polyfills\AssertObjectNotEqualsTest::testAssertObjectNotEquals)
Test Prepared (Yoast\PHPUnitPolyfills\Tests\Polyfills\AssertObjectNotEqualsTest::testAssertObjectNotEquals)
Test Passed (Yoast\PHPUnitPolyfills\Tests\Polyfills\AssertObjectNotEqualsTest::testAssertObjectNotEquals)

....hanging....

Test Finished (Yoast\PHPUnitPolyfills\Tests\Polyfills\AssertObjectNotEqualsTest::testAssertObjectNotEquals)
Test Preparation Started (Yoast\PHPUnitPolyfills\Tests\Polyfills\AssertObjectNotEqualsTest::testAssertObjectNotEqualsCustomMethodName)

Reproducible via the https://github.com/Yoast/PHPUnit-Polyfills repo (though I'm still making some updates for PHPUnit 12 compatibility, which will be pulled soonish).

Expected behavior

No significant slow-down in performance.

[sebastianbergmann]

There have been no changes (that I am aware of) that are specific to branch/path coverage.

From your --debug output (thank you for that!) I assume that it always hangs after the execution of the first test has finished. After "Test Passed" and before "Test Finished" the collection of code coverage data is stopped and the collected data is processed. The first code coverage data is processed during the execution of a test suite there is now additional static analysis performed which could explain the "hanging". However, this static analysis is the same regardless of whether branch/path coverage is enabled or not. The implementation of https://github.com/sebastianbergmann/php-code-coverage/blob/main/src/Target/MapBuilder.php is not optimized and its result is not yet cached, I have this on my TODO for after the release anyway.

Before I go down the wrong rabbit hole, can you please check whether you also get the "hanging" when branch/path coverage is disabled?

[sebastianbergmann]

@staabm Can you have a look at https://github.com/sebastianbergmann/php-code-coverage/blob/main/src/Target/MapBuilder.php and see if/how this could be optimized? Thanks!

[staabm]

I can have a look after Juliette pushed the phpunit 12 changes for the reproducer

[sebastianbergmann]

I can have a look after Juliette pushed the phpunit 12 changes for the reproducer

I think any project can be used to look into the performance of MapBuilder. And the map it builds is the same for PCOV and Xdebug.

[staabm]

I was not yet able to reproduce the slow run on https://github.com/Yoast/PHPUnit-Polyfills (I have no experience with branch coverage):

➜  PHPUnit-Polyfills git:(3.x) ✗ vendor/bin/phpunit -c phpunit10.xml.dist --filter AssertObjectNotEquals --path-coverage
PHPUnit 12.0.1 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.16 with Xdebug 3.4.1
Configuration: /Users/staabm/workspace/PHPUnit-Polyfills/phpunit10.xml.dist

WWWWWWWWWSSWWSWSWWWWW..                                           23 / 23 (100%)

Time: 00:00.732, Memory: 158.00 MB

do you think we can trigger the slow path without xdebug beeing loaded (e.g. with pre-recorded information)?

running blackfire and xdebug in the same process usually doesn't work, as these step onto each others feet.

[sebastianbergmann]

I think we should wait until Juliette answered my "can you please check whether you also get the "hanging" when branch/path coverage is disabled?" question. I am almost certain that this is not related to branch/path coverage.

[jrfnl]

Before I go down the wrong rabbit hole, can you please check whether you also get the "hanging" when branch/path coverage is disabled?

@sebastianbergmann I've just checked and with only line coverage and filtered down to one test class, I see a small, but not a significant slow down.

---

Logs

$ phpunit11 -c phpunit10.xml --filter AssertObjectNotEquals

PHPUnit 11.5.7 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.4.1 with Xdebug 3.4.1
Configuration: path/to/PHPUnit-Polyfills/phpunit10.xml

.........S.....S.......                                           23 / 23 (100%)

Time: 00:00.534, Memory: 52.00 MB

OK, but some tests were skipped!
Tests: 23, Assertions: 48, Skipped: 2.

Generating code coverage report in HTML format ... done [00:00.473]


Code Coverage Report Summary:
  Classes:  4.55% (1/22)
  Methods:  4.41% (6/136)
  Lines:    1.80% (10/555)

$ phpunit12 -c phpunit10.xml --filter AssertObjectNotEquals

PHPUnit 12.0.1 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.4.1 with Xdebug 3.4.1
Configuration: path/to/PHPUnit-Polyfills/phpunit10.xml

.........S.....S.......                                           23 / 23 (100%)

Time: 00:01.452, Memory: 54.00 MB

OK, but some tests were skipped!
Tests: 23, Assertions: 48, Skipped: 2.

Generating code coverage report in HTML format ... done [00:00.196]


Code Coverage Report Summary:
  Classes:  4.55% (1/22)
  Methods:  4.41% (6/136)
  Lines:    1.80% (10/555)

---

When running the full suite with only line coverage, I see no significant difference in the performance.

---

Logs

$ phpunit11 -c phpunit10.xml

PHPUnit 11.5.7 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.4.1 with Xdebug 3.4.1
Configuration: path/to/PHPUnit-Polyfills/phpunit10.xml

...............................................................  63 / 478 ( 13%)
................................................SSSS....SSSS... 126 / 478 ( 26%)
.SSSSSSSS.................................SSSSSSSSSSSS......... 189 / 478 ( 39%)
............................................................... 252 / 478 ( 52%)
............................................................... 315 / 478 ( 65%)
......................S.....S..............S.....S......SSS.... 378 / 478 ( 79%)
SSS............................................................ 441 / 478 ( 92%)
..............................

Time: 00:03.360, Memory: 54.00 MB

OK, but some tests were skipped!
Tests: 471, Assertions: 862, Skipped: 39.

Generating code coverage report in HTML format ... done [00:00.447]


Code Coverage Report Summary:
  Classes:  9.09% (2/22)
  Methods:  5.15% (7/136)
  Lines:    2.88% (16/555)

$ phpunit12 -c phpunit10.xml

PHPUnit 12.0.1 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.4.1 with Xdebug 3.4.1
Configuration: path/to/PHPUnit-Polyfills/phpunit10.xml

...............................................................  63 / 478 ( 13%)
................................................SSSS....SSSS... 126 / 478 ( 26%)
.SSSSSSSS.................................SSSSSSSSSSSS......... 189 / 478 ( 39%)
............................................................... 252 / 478 ( 52%)
............................................................... 315 / 478 ( 65%)
......................S.....S..............S.....S......SSS.... 378 / 478 ( 79%)
SSS............................................................ 441 / 478 ( 92%)
..............................

Time: 00:03.985, Memory: 54.00 MB

OK, but some tests were skipped!
Tests: 471, Assertions: 862, Skipped: 39.

Generating code coverage report in HTML format ... done [00:00.203]


Code Coverage Report Summary:
  Classes:  9.09% (2/22)
  Methods:  5.15% (7/136)
  Lines:    2.88% (16/555)

---

So, for argument's sake, I checked the performance of path/branch coverage when running the full test suite as well, and the slow down is back, so that confirms it's the path/branch coverage which is the source of the slow-down, not the test filtering.

Running the full test suite vs a filtered down test doesn't seem to make a difference in the amount of slow down though.

---

Logs

$ phpunit11 -c phpunit10.xml

PHPUnit 11.5.7 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.4.1 with Xdebug 3.4.1
Configuration: D:\000_GitHub\WordPress_SEO_by_Yoast\PHPUnit-Polyfills\phpunit10.xml

...............................................................  63 / 478 ( 13%)
................................................SSSS....SSSS... 126 / 478 ( 26%)
.SSSSSSSS.................................SSSSSSSSSSSS......... 189 / 478 ( 39%)
............................................................... 252 / 478 ( 52%)
............................................................... 315 / 478 ( 65%)
......................S.....S..............S.....S......SSS.... 378 / 478 ( 79%)
SSS............................................................ 441 / 478 ( 92%)
..............................

Time: 00:46.100, Memory: 376.00 MB

OK, but some tests were skipped!
Tests: 471, Assertions: 862, Skipped: 39.

Generating code coverage report in HTML format ... done [00:00.724]


Code Coverage Report Summary:
  Classes:  9.09% (2/22)
  Methods:  5.15% (7/136)
  Paths:    6.82% (9/132)
  Branches:    8.12% (13/160)
  Lines:    2.88% (16/555)

$ phpunit12 -c phpunit10.xml

PHPUnit 12.0.1 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.4.1 with Xdebug 3.4.1
Configuration: D:\000_GitHub\WordPress_SEO_by_Yoast\PHPUnit-Polyfills\phpunit10.xml

...............................................................  63 / 478 ( 13%)
................................................SSSS....SSSS... 126 / 478 ( 26%)
.SSSSSSSS.................................SSSSSSSSSSSS......... 189 / 478 ( 39%)
............................................................... 252 / 478 ( 52%)
............................................................... 315 / 478 ( 65%)
......................S.....S..............S.....S......SSS.... 378 / 478 ( 79%)
SSS............................................................ 441 / 478 ( 92%)
..............................

Time: 01:43.787, Memory: 370.00 MB

OK, but some tests were skipped!
Tests: 471, Assertions: 862, Skipped: 39.

Generating code coverage report in HTML format ... done [00:00.457]


Code Coverage Report Summary:
  Classes:  9.09% (2/22)
  Methods:  5.15% (7/136)
  Paths:    7.96% (9/113)
  Branches:    9.49% (13/137)
  Lines:    2.88% (16/555)

---

I can have a look after Juliette pushed the phpunit 12 changes for the reproducer

@staabm The WIP branch I'm using is available, though not yet merged: https://github.com/Yoast/PHPUnit-Polyfills/commits/feature/composer-ci-allow-for-phpunit-12/

Just FYI/I'll figure out some hacky solution - The thing I'm running into is something which was fixed in PHPUnit 11.3.? (non-existent traits not being ignored, but causing a fatal). As this package polyfills a PHPUnit 11.2.0 assertion (assertObjectNotEquals()), the tests need to run on PHPUnit < 11.2 + > 11.2. What with 12 now enforcing CoversTrait for traits and the CoversTrait attribute not being available in PHPUnit < 11.2 AND PHPUnit < 11.3.? not ignoring unknown attributes... the tests won't run on PHPUnit < 11.2, even when not running with code coverage.

I suspect I'll end up creating PHPUnit version specific wrappers for the tests + adding another config file. Nothing to worry about though as this is an edge-case which should not affect "normal" packages.

[jrfnl]

I was not yet able to reproduce the slow run on https://github.com/Yoast/PHPUnit-Polyfills (I have no experience with branch coverage):

@staabm In case it helps, this is the exact config I used locally:

<?xml version="1.0" encoding="UTF-8"?>
<phpunit
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/10.5/phpunit.xsd"
    backupGlobals="true"
    bootstrap="./tests/bootstrap.php"
    beStrictAboutOutputDuringTests="true"
    beStrictAboutTestsThatDoNotTestAnything="true"
    colors="true"
    cacheDirectory=".cache/phpunit.cache"
    requireCoverageMetadata="true"
    displayDetailsOnTestsThatTriggerErrors="true"
    displayDetailsOnTestsThatTriggerWarnings="true"
    displayDetailsOnTestsThatTriggerNotices="true"
    displayDetailsOnTestsThatTriggerDeprecations="true"
    displayDetailsOnPhpunitDeprecations="true"
    failOnWarning="true"
    failOnNotice="true"
    failOnDeprecation="true"
    failOnPhpunitDeprecation="false"
    >
    <testsuites>
        <testsuite name="PHPUnitPolyfills">
            <directory>./tests/</directory>
        </testsuite>
    </testsuites>

    <source>
        <include>
            <file>phpunitpolyfills-autoload.php</file>
            <directory suffix=".php">./src/</directory>
        </include>
        <exclude>
            <file>src/Polyfills/AssertClosedResource_Empty.php</file>
        </exclude>
    </source>

    <coverage pathCoverage="true">
        <report>
            <text outputFile="php://stdout" showOnlySummary="true"/>
            <html outputDirectory="build/coverage-html" lowUpperBound="35" highLowerBound="90"/>
        </report>
    </coverage>
</phpunit>

[staabm]

which xdebug.mode do you use?

I am still unable to reproduce with

php -v
PHP 8.3.16 (cli) (built: Jan 14 2025 18:25:29) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.3.16, Copyright (c) Zend Technologies
    with Xdebug v3.4.1, Copyright (c) 2002-2025, by Derick Rethans

using vendor/bin/phpunit -c phpunit10.xml.dist --filter AssertObjectNotEquals and your given config

[jrfnl]

which xdebug.mode do you use?

@staabm This the config I'm using for PHP 8.4.1 (which I used for the above tests):

[xdebug]
zend_extension="C:/wamp/bin/php/ext/xdebug/php_xdebug-3.4.1-8.4-ts-vs17-x86_64.dll"

;xdebug.mode allowed are : off develop coverage debug gcstats profile trace
xdebug.mode=coverage
xdebug.output_dir="C:/wamp/tmp"
xdebug.show_local_vars=0
xdebug.log="C:/wamp/logs/xdebug.log"
;xdebug.log_level : 0 Criticals, 1 Connection, 3 Warnings, 5 Communication, 7 Information, 10 Debug	Breakpoint
xdebug.log_level=7
xdebug.profiler_output_name=trace.%H.%t.%p.cgrind
xdebug.use_compression=false