Automated way to report flaky test

[anasalkouz]

Is your feature request related to a problem? Please describe.
Currently, Developers are reporting flaky test issue while working on submitting PRs and testing features.

Describe the solution you'd like
Having an automated way to detect flaky failed tests on gradle check, then report this on github issues.

[andrross]

re: detecting flaky tests

The test-retry plugin shows examples of reporting flaky tests: https://github.com/gradle/test-retry-gradle-plugin#gradle-enterprise

Assuming we get retries working properly (#5239) then flaky tests will have both a failed and a passed outcome in the test report. If we could surface flaky tests directly into the PR (probably in the post where the test outcome is shown) it would be a great help for identifying flaky tests.

[dblock]

@andrross I had put up #5200 but it was a hacky-ish way to do it; want to attempt something cleaner?

[andrross]

Now that we have implemented #5200, I believe the flaky test reporting is working well on PRs. Authors and reviews are notified about flaky tests in a very obvious way and folks are creating issues to track flakiness where appropriate. We could potentially implement more automation here, but I'm curious if that is worth the investment?

However, I think we still potentially have a gap where we're not taking any action on failures are instability in the gradle check that runs after the code is pushed (the gradle-check github action runs on both PRs and push events). Recent commits show that we have pretty good reliability here now (with retries preventing one-off flaky test failures from failing the build). I propose that we add a step here to automatically create an issue (and potentially assign/notify the author and maintainer that merged the commit) any time there is a failure. This would quickly surface build failures in case broken code is committed, and prevent hard-to-reproduce failures from going unnoticed.

[dblock]

+1 @andrross for an [AUTOCUT] issue

[mch2]

related - #5799, #5979

[mch2]

Just merged #5979 to autocut after a check fails post-merge. If there is anything else we see missing related to this issue please feel free to re-open.