Mechanism to ensure newly added tests are not flaky

[dreamer-89]

Today, there exists multiple flaky tests which often results in unstable builds or gradle check failure. The later is more frustrating as it blocks the PR and needs explicit retry in the hope of passing gradle check. In order to ensure, we are not adding more flaky tests with new development, we can employ below two mechanisms to gate flaky tests from merge.

1. Guide

• Update pull_request_template.md to add/update line item to run newly added tests locally. Run integration tests at least 100 times & unit test 1000 times without failure.
• Update developer guide with recommendation to run tests locally before submitting PR.

2. Github workflow

• Add new github workflow which runs newly added tests in repetition. This supplements step 1 above.

[owaiskazi19]

Great request @dreamer-89! I would prefer to have only the second option to add a GHA which runs the newly added tests in repetition and fails if the any test is flaky.
This will reduce the load on the devs to run tests 100 times locally and the pain to raise a PR.