coveralls in github actions and parallel builds always report

[johanneswilm]

Hey,
I am migration from Travis to Github Actions. Everything seems to be working, but for some reason coveralls now reports percentages for individual jobs rather than just the final aggregated number. I am running several different jobs, each only covering parts of the code and so only the aggregated report will show a combined score of 91%. The individual jobs have percentages like "53%" which then cause the commit to be marked as failed on github.

The setting "ONLY SEND AGGREGATE COVERAGE UPDATES TO SCM?" seems to not make any difference.

See https://coveralls.io/builds/35153395

[afinetooth]

@johanneswilm briefly, the addition of the flag_name attribute to your POSTs, vis-a-vis COVERALLS_FLAG_NAME in your GH Action workflow, here, tells coveralls to send a separate check for each named job. If you remove that, you'll only get one check for the aggregate coverage. That should do it, but let me know if you have any further issues.

[casperdcl]

as mentioned in #1513 (comment), the proposed solution is wrong. This is still an issue.

Basically COVERALLS_FLAG_NAME & COVERALLS_PARALLEL mean multiple reports despite "Only send aggregate coverage updates to SCM" being enabled.

Here's an example:

Codecov correctly aggregates into one check, while coveralls floods reports.

[afinetooth]

Yes, @johanneswilm @casperdcl, @casperdcl's comment here made me realize I incorrectly answered this one.

The issue here is related to this issue in that both will be addressed by two (2) proposed feature change requests, described in the other issue here.

Adding those changes here for convenience:

I’ve submitted two feature change requests:

1. Only apply coverage threshold checks to overall coverage
2. Make the use of flag-name optional for parallel builds using the Coveralls GitHub Action.

Re-opened this issue.

---

@casperdcl FWIW, job checks are optional for most other CI setups, but they're "required" with the Coveralls Github Action by virtue of the flag_name currently being required, per its description in Usage > Input.

[jrfnl]

I'm running into this as well and find it very problematic.

I'm running four parallel builds with different environments and only the combined results of these builds gives a clear and relevant picture of the coverage status.

The aggregate result is a required status and I'd want the aggregate result to be able to mark a PR as failed/not mergable if the coverage is below a certain (high) threshold or has decreased by more than 0.5%.

However, as the four parallel builds are also showing and would mark a PR as failed as the coverage threshold and decrease threshold are also applied to them, all PRs in this particular repo would always show as failed, giving the wrong impression and making it easy to miss a real failed build.

Alternatively, the coverage threshold and decrease thresholds would have to be brought down significantly to allow the parallel builds to show as passed, but then coverage is effectively no longer safeguarded at all.

Basically, IMO, the individual builds in a parallel build set-up should never show in the status checks and the thresholds should only ever be applied to the aggregate results (which should show in the status checks).

Anything other than that makes the coverage check useless.

[afinetooth]

Thanks, @jrfnl.

I'm acquainted with your issue from this issue.

As I asked there, can you please share your coveralls URL so we can reproduce. If it's private (I presume it is), you can send it to support@coveralls.io and just reference this issue (or the other).

Since I can't see your PRs for a private repo, the ideal information would be:

1. Your coveralls repo URL
2. The coveralls build URL for a problematic build
3. A screenshot of the problematic, related PR
4. Your Ci config (GHA workflow file) or relevant segment

Even more ideal would be a full, verbose CI build log, which can be achieved by adding the following env var to your workflow file:
NODE_COVERALLS_DEBUG=1

[jrfnl]

@afinetooth You're in luck as I can actually post the details for a public repo ;-)

Here goes:

1. https://coveralls.io/github/PHPCSStandards/PHPCSUtils
2. All builds run into the same problem.
3. If you look at the recently merged PRs, you can see that I have been fiddling around with the threshold percentages as those red checkmarks next to each PR were making me very nervous.
   https://github.com/PHPCSStandards/PHPCSUtils/pulls?q=is%3Apr+is%3Aclosed
   Here's screenshot from a PR before I started fiddling:
   
   Ideally, I'd like to be able to return the settings to the values I had before, 95% for minimum coverage and 0.5% as a decrease threshold - both only applied to the aggregate result, but as things stand, that is not currently possible.
4. https://github.com/PHPCSStandards/PHPCSUtils/blob/6cf56c37c75603543ba4c8f668ff3a0625660239/.github/workflows/test.yml#L185-L276

Even more ideal would be a full, verbose CI build log, which can be achieved by adding the following env var to your workflow file:
NODE_COVERALLS_DEBUG=1

Done ;-) I've run a slimmed-down workflow with only the coverage builds: https://github.com/PHPCSStandards/PHPCSUtils/actions/runs/843942219
I'm not sure the addition of NODE_COVERALLS_DEBUG=1 is making any difference though as I don't see any debug information....
You can see the changes I made to the workflow script here: PHPCSStandards/PHPCSUtils@702408b (removing of the other non-coverage test builds and adding of the environment variable).

Please let me know if you need more information or want me to do some additional test runs. Happy to help debug this.

[afinetooth]

Hi @jrfnl,

Sorry for the delay. Thanks for the very clear and detailed issue report.

So, I'm looking at some of your recent builds:

• https://coveralls.io/builds/39653971
• https://coveralls.io/builds/39710162

And it looks like you've arrived at something closer to acceptable with 85% / 1% as your threshold settings:

Just curious how you're feeling about that? As a workaround is it workable? Or is your workflow still suffering from the inability to safeguard coverage at the desired target of 95%?

I noticed something interesting about your GHA workflow:
https://github.com/PHPCSStandards/PHPCSUtils/blob/6cf56c37c75603543ba4c8f668ff3a0625660239/.github/workflows/test.yml#L185-L276

You only seem to be using the Coveralls Github Action for the second of two required workflow steps:

1. Send one job for a parallel build (x4)
2. Close the parallel build

From L247 to END:

      # Uploading the results with PHP Coveralls v1 won't work from GH Actions, so switch the PHP version.
      - name: Switch to PHP 7.4
        if: ${{ success() && matrix.php != '7.4' }}
        uses: shivammathur/setup-php@v2
        with:
          php-version: 7.4
          coverage: none

      - name: Install Coveralls
        if: ${{ success() }}
        run: composer require php-coveralls/php-coveralls:"^2.4.2"

      - name: Upload coverage results to Coveralls
        if: ${{ success() }}
        env:
          COVERALLS_REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          COVERALLS_PARALLEL: true
          COVERALLS_FLAG_NAME: php-${{ matrix.php }}-phpcs-${{ matrix.phpcs_version }}
        run: vendor/bin/php-coveralls -v -x build/logs/clover.xml

  coveralls-finish:
    needs: coverage
    runs-on: ubuntu-latest

    steps:
      - name: Coveralls Finished
        uses: coverallsapp/github-action@master
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          parallel-finished: true

Since you're only using the Coveralls Github Action for Step (2) (Close your parallel build), and not for Step (1), where you instead seem to be using shivammathur/setup-php, you might be able to get around the main issue here, which is that the Coveralls Github Action requires flag-name be included for each job.

Have you tried removing COVERALLS_FLAG_NAME from your workflow file to see if coveralls stops sending status checks for each job in your build? This is at least how it used to work (before we added the ONLY SEND AGGREGATE COVERAGE... setting):

From our API Reference:

flag_name (String OPTIONAL): If this is set, the job being reported will be named in the view and have it’s own independent status reported to your VCS provider

Another thing that might help is changing the service_name parameter.

Incoming JSON to the Coveralls API at https://coveralls.io/api/v1/jobs triggers unique behavior for some values, including "github-actions". A value of service_name: github-actions tells the API to treat the incoming job as if it was coming through the Coveralls Github Action.

Using a different value for service_name, like "github", could resolve the issue, as the API should then process your job as if it was coming from any generic CI/integration.

I say "could" because, while I don't think there will be a conflict with your using the Coveralls Github Action for Step (2) (Close the parallel build), in my capacity as support engineer, rather than developer, there might be something I'm not aware of that would prevent that.

You can change service_name by adding this line to your coveralls.yml:

service_name: github

I hope those two changes will help improve your outcome.

---

P.S.
The reason NODE_COVERALLS_DEBUG=1 didn't work is because it's not applicable in your case. NODE_COVERALLS_DEBUG=1 is the debug env var for node-coveralls, which the Coveralls Github Action uses under-the-hood. It's not applicable because you're not using the Coveralls Github Action for Step (1) (Sending your jobs).

You'll need to try the debug / verbose option for php-coveralls/php-coveralls, which I believe is --verbose per the CLI options here.

We're looking for a coveralls-specific debugger, which adds the JSON you're actually POSTing to the Coveralls API to the output of your CI build log.

[jrfnl]

@afinetooth Hi James, thanks for your response.

I'll try and answer your suggestions and questions one by one. Please let me know if you need more clarification or if I'm overlooking something.

Just curious how you're feeling about that? As a workaround is it workable? Or is your workflow still suffering from the inability to safeguard coverage at the desired target of 95%?

I'm definitely not happy with this workaround as it no longer safeguards anything, but those red crosses were driving me crazy and giving a wrong impression to anyone else looking at/reviewing the PRs in the repo.

Having to remember to manually open up the checks fold-out and verify the actual aggregate coverage result has not gone down, is cumbersome, easy to forget and completely against the principle of having required status checks which will fail when they are not complied with, so this is definitely not an acceptable work-around. It is a form of "making do" while this issue is unresolved.

Since you're only using the Coveralls Github Action for Step (2) (Close your parallel build), and not for Step (1), where you instead seem to be using shivammathur/setup-php, you might be able to get around the main issue here, which is that the Coveralls Github Action requires flag-name be included for each job.

Just to clarify as I guess you may not be that familiar with a PHP project setup:

• Step 1 (sending the results of individual builds) is done via the php-coveralls/php-coveralls package.
  This is needed to translate the results from the PHPUnit clover format to the format expected by Coveralls.
• Step 2 (sending the "parallel finished") is done via the Coveralls GitHub Action as that seemed to be the easiest way to get that working.
  The docs in the php-coveralls/php-coveralls repo also explicitly state that step is still needed: https://github.com/php-coveralls/php-coveralls#parallel-builds

The shivammathur/setup-php package has nothing to do with this setup, other than providing the environment in which I run the tests and run the php-coveralls/php-coveralls command.

Have you tried removing COVERALLS_FLAG_NAME from your workflow file to see if coveralls stops sending status checks for each job in your build? This is at least how it used to work (before we added the ONLY SEND AGGREGATE COVERAGE... setting):

I did a lot of trial-and-error test runs when I made the switch from Travis to GH Actions and initially I didn't have the COVERALLS_FLAG_NAME in the coverage set-up (step 1).
If I remember correctly, this resulted in four separate builds for each PR and no aggregate build being created, so I could never get an accurate impression of the real code coverage, which is why I ended up adding the COVERALLS_FLAG_NAME.

Using a different value for service_name, like "github", could resolve the issue, as the API should then process your job as if it was coming from any generic CI/integration.

I say "could" because, while I don't think there will be a conflict with your using the Coveralls Github Action for Step (2) (Close the parallel build), in my capacity as support engineer, rather than developer, there might be something I'm not aware of that would prevent that.

I've run a test run with that.

• I've rebased and updated the test branch I uploaded previously - see:
  https://github.com/PHPCSStandards/PHPCSUtils/commits/TEST/coveralls-debug / PHPCSStandards/PHPCSUtils@541a30b
• I've opened a draft PR to see the reported statuses: Test/coveralls debug PHPCSStandards/PHPCSUtils#272

End result: when I initially opened the PR, the original four extra statuses were showing, but now the build has (sort of) finished running, they seem to have disappeared. Not sure what's going on there.

You'll need to try the debug / verbose option for php-coveralls/php-coveralls, which I believe is --verbose per the CLI options here.

We're looking for a coveralls-specific debugger, which adds the JSON you're actually POSTing to the Coveralls API to the output of your CI build log.

I'm already running php-coveralls/php-coveralls in verbose mode.

As per the docs: --verbose (-v): Used to show logs.

And the command in my GH Actions workflow: run: vendor/bin/php-coveralls -v -x build/logs/clover.xml <= the -v in this command.

[afinetooth]

Thanks @jrfnl. Will try to keep this as concise / efficient as possible:

[...] definitely not happy with this workaround [...] It is a form of "making do" while this issue is unresolved.

Understood. Thank you.

Just to clarify as [...] you may not be [...] familiar with a PHP project setup:

Correct, not as familiar with PHP projects, but shivammathur/setup-php was the only piece I hadn't encountered before. php-coveralls/php-coveralls makes sense for Step 1. Thanks for the details.

Have you tried removing COVERALLS_FLAG_NAME

If I remember correctly, this resulted in four separate builds for each PR and no aggregate build being created, so I could never get an accurate impression of the real code coverage, which is why I ended up adding the COVERALLS_FLAG_NAME.

My apologies ahead of time if that remains the result of removing COVERALLS_FLAG_NAME from your config, but in terms of workarounds for current functionality, it's the last one I can think of and could be worth a try. I'm under the impression this could be the only block to the recently-added ONLY SEND AGGREGATE COVERAGE UPDATES TO SCM setting working as intended. The fastest way to to find out, including from my end, will be a test.

End result: when I initially opened the PR, the original four extra statuses were showing, but now the build has (sort of) finished running, they seem to have disappeared. Not sure what's going on there.

Thanks. Using github instead of github-actions remains my recommendation here as we're trying to get away from the locked-down, status-per-job functionality of the action.

I'm not sure what would have happened there either except with a possible cause related to how you'll see one status each for the pull_request build and the associated push (merge) build on the Github side. On the coveralls side though, the statuses should be coming from one build (the PR build).

In any case, the final result looks like what's expected, with the disappointing fact of a status for each job, each constrained by the high and low coverage thresholds.

Again, it's my hope that you might avoid that by removing COVERALLS_FLAG_NAME, which would be great, but stands as the last recommendation for a workaround at this time.

I'm already running php-coveralls/php-coveralls in verbose mode.

That's the relevant debug flag for Step 1, so I assume JSON output is not included in that debug output.

Thanks for your patience, I understand not much bang for your buck at this point, unless the flag name piece works.