Add `--fail-under` option

[saroad2]

Description

While generating coverage reports is super important, in order to use coverage in CI workflows, sometimes you want to enforce a certain coverage percentage.

Python's coverage library do that using the --fail-under flag which cause the coverage method to fail if coverage percentage is under than expected.

API

Simply add the --fail-under flag to the coverage execution with the desired coverage percentage. For example, --fail-under=90 will enforce at least 90% coverage and --fail-under=100 will enfore total coverage (100%).

[Victowolf]

@saroad2 @mosuem @liamappelbe
Hi,

I was trying to do this implementation, but got stuck in the last.

This is all what I did:

• Added --fail-under flag to ArgParser ( parsing CLI arguments) this function is present in bin/format_coverage.dart
• Added this logic to extract and validate --fail-under after parsing.

      final failUnder = args['fail-under'] as String?;
      int? failUnderThreshold;
  
      if (failUnder != null) {
        failUnderThreshold = int.tryParse(failUnder);
        if (failUnderThreshold == null || failUnderThreshold < 0 || failUnderThreshold > 100) {
           fail('Invalid value for --fail-under. It must be a number between 0 and 100.');
          }
      }

• Modified the Environment constructor and included failUnderThreshold so that it can be accessed later. then passed failUnderThreshold in parseArgs return statement.
• I found that coverage calculation is happening inside the formatLcov function, which is present in lib/src/formatter.dart and added this logic to check fail-under:

     buf.write('LF:${lineHits.length}\n');
     buf.write('LH:${lineHits.values.where((v) => v > 0).length}\n');
  
     int totalLines = lineHits.length;
     int coveredLines = lineHits.values.where((v) => v > 0).length;
     double coveragePercentage = totalLines == 0 ? 0.0 : (coveredLines / totalLines) * 100;
  
     //Adds the coverage percentage to output.
     buf.write('Coverage: ${coveragePercentage.toStringAsFixed(2)}%\n');
  
     //Fail-under to check
     if (failUnderThreshold != null && coveragePercentage < failUnderThreshold) {
         throw Exception(
            'Coverage check failed! Coverage (${coveragePercentage.toStringAsFixed(2)}%) '
            'is below the required threshold ($failUnderThreshold%).'
          );
     }

Testing:

• Ran dart test --coverage=coverage and gathered coverage data. data was successfully gathered with all tests passed.
• Converted the coverage data to .json file by running: dart run coverage:format_coverage -i coverage/test -o coverage/coverage.json --report-on=lib
• Then tested the --fail-under by running: dart bin/format_coverage.dart -i coverage/coverage.json --fail-under=90
• Test was conducted in VScode terminal (Powershell).

Result:

• no output was recevied

     PS C:\Users\HP\tools\pkgs\coverage> dart bin/format_coverage.dart -i coverage/coverage.json --fail-under=90
     PS C:\Users\HP\tools\pkgs\coverage>

Conclusion:

Not able to find the cause for which output is empty.

Possible cause:

As the fail under check is one file (lib/src/formatter.dart) and the parser is in another file (bin/format_coverage.dart), it is not getting executed when I run: dart bin/format_coverage.dart -i coverage/coverage.json --fail-under=90

[Victowolf]

@saroad2 @mosuem @liamappelbe
Hi,

Following the possible cause of previous comment.
I tried to shift the coveragePercentage calculation to bin/format_coverage.dart to get it executed.

This is what I did:

• Declared a global variable double coveragePercentage = 0.0;, in bin/format_coverage.dart.
• Defined a new function to calculated the coverage and called this function just after the hitmap calculation calculateCoverage(hitmap); :

    // Compute overall coverage percentage
    void calculateCoverage(Map<String, HitMap> hitmap) {
      int totalLines = 0;
      int coveredLines = 0;

      for (var entry in hitmap.values) {
        totalLines += entry.lineHits.length;
        coveredLines += entry.lineHits.values.where((v) => v > 0).length;
     }

      coveragePercentage = totalLines == 0 ? 0.0 : (coveredLines / totalLines) * 100;
      print('Overall Coverage: ${coveragePercentage.toStringAsFixed(2)}%');
   }

• Added fail-under check inside bin/format_coverage.dart instead of adding it in lib/src/formatter.dart.

 // fail-under check.
 if (failUnderThreshold != null && coveragePercentage < failUnderThreshold) {
   print(
      'Coverage check failed! Coverage (${coveragePercentage.toStringAsFixed(2)}%) '
      'is below the required threshold ($failUnderThreshold%).');
   exit(1); // Fail the CI/CD pipeline
  }

Testing:

• Ran dart test --coverage=coverage and gathered coverage data. data was successfully gathered with all tests passed.
• Converted the coverage data to .json file by running: dart run coverage:format_coverage -i coverage/test -o coverage/coverage.json --report-on=lib
• Then tested the --fail-under by running: dart bin/format_coverage.dart -i coverage/coverage.json --fail-under=90
• Test was conducted in VScode terminal (Powershell).

Result:

• Output was received

   PS C:\Users\HP\tools\pkgs\coverage> dart bin/format_coverage.dart -i coverage/coverage.json --fail-under=90
   Parsed failUnderThreshold: 90
   Coverage check failed! Coverage (0.00%) is below the required threshold (90%).
   PS C:\Users\HP\tools\pkgs\coverage>

• CoveragePercentage was not calculated, the debuging statemenets that i had added in calculateCoverage function didnt get executed. It used predefinded value double coveragePercentage = 0.0; .

Conclusion:

• calculateCoverage function didnt get executed.
• failunder check used predefined value of coveragePercentage to check and return output.

Possible cause:

Still trying to figure it out.
Any suggestions, feedback will be helpfull.

[mosuem]

@Victowolf Please file a Draft PR to make it easier to track your changes. Also no need to tag me (or others) on every comment :)

[vvs-personalstash]

Hello @mosuem ,
I am Viral Verma and I wanted to contribute to this issue. I have been following this issue for a while and since it has been inactive for a while, I have opened a PR regarding this issue with the implementation I had in mind.I tested it locally on some packages and it seems to be working for me as intended. I request you to please take a look at it.

cc: @liamappelbe