No warning when CLI options are used multiple times

[ondrejmirtes]

Hi, right now it seems from my testing that passing --filter multiple times rewrites the previous occurrence so the first --filter option is silently discarded.

vendor/bin/phpunit --filter 'PHPStan\\Type\\UnionTypeTest::testSorting' --filter 'PHPStan\\Type\\UnionTypeTest::testAccepts#7' --verbose --debug
PHPUnit 9.5.23 #StandWithUkraine

Runtime:       PHP 8.3.1
Configuration: /Users/ondrej/Development/phpstan/phpunit.xml
Warning:       No code coverage driver available

Test 'PHPStan\Type\UnionTypeTest::testAccepts with data set #7 (PHPStan\Type\UnionType Object (...), PHPStan\Type\ObjectType Object (...), PHPStan\TrinaryLogic Object (...))' started
Test 'PHPStan\Type\UnionTypeTest::testAccepts with data set #7 (PHPStan\Type\UnionType Object (...), PHPStan\Type\ObjectType Object (...), PHPStan\TrinaryLogic Object (...))' ended

Time: 00:00.117, Memory: 570.00 MB

OK (1 test, 1 assertion)

I'd like to use --filter to be interpreted in a way - "I want to run tests that pass this filter and also tests that pass this other filter". So in this case PHPUnit should run the two tests I specified.

Even if you don't want to interpret multiple --filter options that way, I think something could be done about the silent discarding.

Thank you.

[sebastianbergmann]

May I kindly ask you to check whether PHPUnit 10.5 and PHPUnit 11.0 also silently ignore multiple --filter options? Thanks!

[sebastianbergmann]

Nevermind, I can confirm that --filter (as well as --exclude-filter) can be used only once and that when it is used multiple times the "last one parsed wins", silently ignoring the previously parsed ones. I consider this a bug that should be fixed for PHPUnit 10 and PHPUnit 11 (PHPUnit 9 no longer receives bug fixes).

I do not, however, think that multiple --filter options should be allowed.

[ondrejmirtes]

I do not, however, think that multiple --filter options should be allowed.

Yeah, I got a little bit obsessed recently with passing the same option multiple times after watching https://www.youtube.com/watch?v=aolI_Rz0ZqY (great talk btw) because git blame does different things whether you pass -C once, twice, or thrice. So I thought it could apply here but of course I'm fine if it doesn't do anything useful.

[sebastianbergmann]

All long options supported by PHPUnit's CLI test runner can only be used once. For instance, when --filter foo --filter bar is used then only --filter bar is used as it overwrites any --filter option that was given before.

As a first step, PHPUnit's CLI test runner will be changed (for PHPUnit >= 10.5) to emit a warning when an option is used multiple times. Future work will be done (for PHPUnit >= 11.1) to allow multiple --group and --exclude-group options, for example.

[ondrejmirtes]

Awesome, thanks!

[gnutix]

@sebastianbergmann Is does not seem to be true of the -d option, yet I started having a warning about it.
We've used it for years in our test pipeline to pass arguments to our tests. Our commands look like this :

phpunit -c phpunit-something.xml --testsuite=something \
  -d memory_limit=1G \
  -d app.tests.generate=1 \
  -d app.tests.temporal=2024 \
  -d app.tests.xxx=yyy \
  ...

And all arguments have always been properly passed, and still do on PHPUnit 10.5.11 (just tested right now to be sure).

However, we do have (crappy) custom code in our bootstrap.php file to support this, so maybe that's why it still works ?

// https://github.com/sebastianbergmann/phpunit/issues/634
// The upstream issue about passing custom parameters to PHPUnit was closed.
// Let's simply abuse `-d` for now and scan argvs.
/** @var string $arg */
foreach ($_SERVER['argv'] as $arg) {
    if (preg_match('/app\.tests\.generate=1/', $arg)) {
        $_SERVER['TESTS_GENERATE'] = '1';
    }
    if (preg_match('/app\.tests\.temporal=([0-9A-Z:\-\/]+)/', $arg)) {
        $_SERVER['TESTS_TEMPORAL'] = $matches[1];
    }
    // ... and on it goes ...
}

And then we use $_SERVER['TESTS_XXX'] in our test code to trigger different behaviors.

[sebastianbergmann]

Is does not seem to be true of the -d option, yet I started having a warning about it.

I meant to write "long options" instead of just "options" in #5704 (comment). I totally forgot about -d.