Feedback about Test Analytics and Flaky Test Reporting

[rohan-at-sentry]

Thanks for dropping by! 👋

We've recently released a whole new feature around Test analytics and are working on reporting Test Flakes on your PR ❄️ .

We'd love to hear feedback on

• How your setup experience was.
• How easy/useful the PR comment is

This issue is intended to share and collect feedback about the tool. If you have support needs or questions, please let us know!

[houserx-jmcc]

• How your setup experience was.
  • Very easy, worked on the first go after adding the new GitHub action step.
• How easy/useful the PR comment is
  • I like that it is combined with the existing coverage comment versus being a separate one.

In the event this action is not used in the context of a pull request (i.e. a scheduled test, pre-release test, etc.), it would be nice to be able to also display the results in GitHub's Job Summary in lieu of a PR comment. We currently use https://github.com/phoenix-actions/test-reporting to do this and it works nicely. When sharing test results, permalinking to the job summary works better due to the bigger screen size it can occupy. If this feature was added, we'd switch over completely!

[houserx-jmcc]

An update: we're also seeing the below error intermittently that we have been unfortunately seeing in v4 of the standard codecov-action (codecov/codecov-action#1280):

Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:160:15)
    at writeGeneric (node:internal/stream_base_commons:151:3)
    at Socket._writeGeneric (node:net:952:11)
    at Socket._write (node:net:964:8)
    at writeOrBuffer (node:internal/streams/writable:447:12)
    at _write (node:internal/streams/writable:389:10)
    at Socket.Writable.end (node:internal/streams/writable:665:17)
    at Socket.end (node:net:7[22](https://github.com/houserx/platform/actions/runs/8513226507/job/23316490519?pr=8922#step:12:23):31)
    at module.exports (/runner/_work/_actions/codecov/test-results-action/v1/node_modules/gpg/lib/spawnGPG.js:50:1)
    at Object.call (/runner/_work/_actions/codecov/test-results-action/v1/node_modules/gpg/lib/gpg.js:28:1)

[rohan-at-sentry]

Thanks for your feedback @houserx-jmcc - we're looking into adding support for reporting on GH Job summary as well

Re the issue with EPIPE - yeah it's something we've noticed. We're still working on a fix for this

[rohan-at-sentry]

@houserx-jmcc out of curiosity what non PR usecases are you using https://github.com/phoenix-actions/test-reporting for?

Off the top of my head, I'd guess something along the lines of running tests before deploy (of a production image/release) etc. Are there other usecases that you can share?

[houserx-jmcc]

Off the top of my head, I'd guess something along the lines of running tests before deploy (of a production image/release) etc. Are there other usecases that you can share?

Yup! That is one. Others include: running the tests on a stable branch while investigating suspected flakey tests, browser tests we run after deployments to increase confidence, and scheduled tests that periodically perform data quality checks against our stable environment databases. We use Jest for all these use cases and thus can easily pipe the XML into the test action reporter.

And for viewing historical test results on a pull request, the job summary is tied to each workflow execution. The PR comment is auto-updated though, so checking on past results becomes a bit more obtuse.

[rohan-at-sentry]

@houserx-jmcc - is this directionally what you had in mind? (still early days!)

https://github.com/joseph-sentry/codecov-cli/actions/runs/8560172298