Add Mechanism to prevent adding new unstable flaky to test

[anasalkouz]

Is your feature request related to a problem? Please describe.
Recently, We observe more flakiness on our tests because of newly created tests or updates on existing tests, like the following 2 examples:
#5189
#5031

Describe the solution you'd like
I am looking for a mechanism to avoid this from happening again, this could be by manual or an automated process, like maintainers should not approve PR that has new test without providing 100+ successful run with no failures.

Looking to for better alternatives

[anasalkouz]

@pranikum @imRishN @xuezhou25
Would like to know your thoughts on this?

[andrross]

This PR in the Gradle test-retry plugin is an interesting idea. The general approach is to identify tests that are added or changed as a part of a commit, then re-run those tests many times as a part of the build. The idea is to catch flakiness immediately so the developer that already has the context can fix them right away. This functionality doesn't exist in the test-retry plugin yet but it could be added.

[reta]

@andrross a bit different approach is taken by Jenkins (if I am not mistaken): the tests diff is taken from JUnit XML reports (we also have them). So basically the idea would be:

• fetch the reports from upstream branch (main, 2.x, ..)
• compare with the report for current pull request (there could be new test cases in existing suites or completely new test suites)

The question is: when should be run such check? We probably should not be running it on each change in pull request (however it could be useful sometimes), but certainly before the pull request gets merged.

[dblock]

I propose new tests be rerun with N random seeds as part of gradle check (or a new gradle check for new tests).

note Jenkins seems to know what’s new because it marks existing failing tests as regressions and marked builds unstable

[andrross]

While I do think this is an interesting idea, our quick approach of surfacing flakiness directly in a PR comment (#5200) seems to be working well for surfacing flakiness. It does require manual work from authors and reviewers to notice the failure, create an issue, and (for recently introduced flakiness) notify the original author, but I do believe this is working reasonably well now. I'm inclined to close this issue because I don't think anybody is going to take up this work in the near term to improve the automation here. @dblock @reta what do you think?

[dblock]

Let's close.

I'd a lot of things, like a Jenkins plugin that manages flaky test GitHub issues, but maybe we don't need to do that much engineering ;)