Assert no Exception happened

[guilhermeblanco]

Until PHPUnit 6, a successful execution could be left without any assertion, while faulty situations were captured using Exceptions. Now on PHPUnit 6, successful executions are being considered risky (R), and sometimes no assertion at all is needed.

We should be able to flag that no exceptions happened during the execution of the method, which would be a much better replacement than the current alternative self::assertTrue(true);.

[sebastianbergmann]

PHPUnit is now strict about useless tests by default. If you do not want this then simply run your tests using the --dont-report-useless-tests commandline option or beStrictAboutTestsThatDoNotTestAnything=false in your phpunit.xml.

[guilhermeblanco]

@sebastianbergmann that is the problem... I want the risky test report to be turned on, but I also want a meaningful way of correcting my test cases instead of useless true===true assertions.

[kubawerlos]

Once I also felt the need to checking if no exception was thrown.

But then I made a small deduction:

• if a function returns something I can test that
• if a function has an argument I can mock it and made some expectations
• if a function isn't returning anything and has no arguments can I REALLY test it? Sure, I can write a test, but what if I remove whole body of this function? Test will still pass, proving itself being completely useless.

[sebastianbergmann]

PHPUnit 6.1 will always display details about risky tests. PHPUnit 6.0 requires --verbose or verbose="true".

[sebastianbergmann]

The @doesNotPerformAssertions annotation can be used to exclude a test from the risky test check in question.

[marcospassos]

The problem now is that tests with no assertion are not considered covered in the coverage report.

[andybeak]

My particular use case is a regression test. The condition used to throw an exception, but now that I've fixed the problem I want to assert that the code runs without an exception being raised. I'd prefer to do this explicitly rather than making a dummy assertion as above and commenting in code.

[bigfoot90]

@sebastianbergmann This sounds like a hack.
Any chance to have an explicit method in future versions?

[keradus]

@marcospassos

When I use the annotation, I do have the coverage.

First, I have investigated the code and found no reason why it would affect coverage, then I executed sample test class using annotation for every single test, coverage result is the same as without annotation:

ker@dus:~/github/PHP-CS-Fixer$ git st
On branch master
nothing to commit, working tree clean
ker@dus:~/github/PHP-CS-Fixer$ phpdbg -qrr vendor/bin/phpunit --coverage-text tests/DocBlock/AnnotationTest.php 
PHPUnit 6.5.5 by Sebastian Bergmann and contributors.

Runtime:       PHPDBG 7.2.1-1+ubuntu17.04.1+deb.sury.org+1
Configuration: /home/keradus/github/PHP-CS-Fixer/phpunit.xml.dist

Testing PhpCsFixer\Tests\DocBlock\AnnotationTest
.....................................................                                                                                                                                                                                                                53 / 53 (100%)

Time: 92 ms, Memory: 8.00MB

OK (53 tests, 100 assertions)


Code Coverage Report:       
  2018-01-12 12:45:44       
                            
 Summary:                   
  Classes:  0.00% (0/286)   
  Methods:  0.71% (12/1702) 
  Lines:    0.20% (43/21130)

\PhpCsFixer\DocBlock::PhpCsFixer\DocBlock\Annotation
  Methods:  92.31% (12/13)   Lines:  97.73% ( 43/ 44)

ker@dus:~/github/PHP-CS-Fixer$ git stash pop
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   tests/DocBlock/AnnotationTest.php

no changes added to commit (use "git add" and/or "git commit -a")
Dropped refs/stash@{0} (550bd9664b6af31b65c65646674b6068f10bfb6d)
ker@dus:~/github/PHP-CS-Fixer$ git du
diff --git a/tests/DocBlock/AnnotationTest.php b/tests/DocBlock/AnnotationTest.php
index 26343f1c..b5801b1c 100644
--- a/tests/DocBlock/AnnotationTest.php
+++ b/tests/DocBlock/AnnotationTest.php
@@ -88,6 +88,7 @@
      * @param string $content
      *
      * @dataProvider provideGetContentCases
+     * @doesNotPerformAssertions
      */
     public function testGetContent($index, $content)
     {
@@ -114,6 +115,7 @@ public function provideGetContentCases()
      * @param int $start
      *
      * @dataProvider provideStartCases
+     * @doesNotPerformAssertions
      */
     public function testStart($index, $start)
     {
@@ -139,6 +141,7 @@ public function provideStartCases()
      * @param int $end
      *
      * @dataProvider provideEndCases
+     * @doesNotPerformAssertions
      */
     public function testEnd($index, $end)
     {
@@ -164,6 +167,7 @@ public function provideEndCases()
      * @param string $tag
      *
      * @dataProvider provideGetTagCases
+     * @doesNotPerformAssertions
      */
     public function testGetTag($index, $tag)
     {
@@ -190,6 +194,7 @@ public function provideGetTagCases()
      * @param int $end
      *
      * @dataProvider provideRemoveCases
+     * @doesNotPerformAssertions
      */
     public function testRemove($index, $start, $end)
     {
@@ -218,6 +223,7 @@ public function provideRemoveCases()
      * @param string[] $expected
      *
      * @dataProvider provideTypeParsingCases
+     * @doesNotPerformAssertions
      */
     public function testTypeParsing($input, array $expected)
     {
@@ -315,6 +321,7 @@ public function provideTypeParsingCases()
      * @param string   $output
      *
      * @dataProvider provideTypesCases
+     * @doesNotPerformAssertions
      */
     public function testTypes($expected, $new, $input, $output)
     {
@@ -342,6 +349,9 @@ public function provideTypesCases()
         ];
     }
 
+    /**
+     * @doesNotPerformAssertions
+     */
     public function testGetTypesOnBadTag()
     {
         $this->expectException(\RuntimeException::class);
@@ -352,6 +362,9 @@ public function testGetTypesOnBadTag()
         $tag->getTypes();
     }
 
+    /**
+     * @doesNotPerformAssertions
+     */
     public function testSetTypesOnBadTag()
     {
         $this->expectException(\RuntimeException::class);
@@ -362,6 +375,9 @@ public function testSetTypesOnBadTag()
         $tag->setTypes(['string']);
     }
 
+    /**
+     * @doesNotPerformAssertions
+     */
     public function testGetTagsWithTypes()
     {
         $tags = Annotation::getTagsWithTypes();
ker@dus:~/github/PHP-CS-Fixer$ phpdbg -qrr vendor/bin/phpunit --coverage-text tests/DocBlock/AnnotationTest.php 
PHPUnit 6.5.5 by Sebastian Bergmann and contributors.

Runtime:       PHPDBG 7.2.1-1+ubuntu17.04.1+deb.sury.org+1
Configuration: /home/keradus/github/PHP-CS-Fixer/phpunit.xml.dist

Testing PhpCsFixer\Tests\DocBlock\AnnotationTest
.....................................................                                                                                                                                                                                                                53 / 53 (100%)

Time: 97 ms, Memory: 8.00MB

OK (53 tests, 100 assertions)


Code Coverage Report:       
  2018-01-12 12:46:02       
                            
 Summary:                   
  Classes:  0.00% (0/286)   
  Methods:  0.71% (12/1702) 
  Lines:    0.20% (43/21130)

\PhpCsFixer\DocBlock::PhpCsFixer\DocBlock\Annotation
  Methods:  92.31% (12/13)   Lines:  97.73% ( 43/ 44)
ker@dus:~/github/PHP-CS-Fixer$ 

without annotation:
Methods: 92.31% (12/13) Lines: 97.73% ( 43/ 44)
with annotation:
Methods: 92.31% (12/13) Lines: 97.73% ( 43/ 44)