[chore]: fix go routine leaks in tests

[odubajDT]

Description:

• removing re-running parameter from gotestsum
• fixing tests/code with go routine leaks
• resolving race conditions (mostly caused by parallel tests)
• placing goleak ignorers for go routine leaks from external libraries

Link to tracking Issue: #34495

[crobert-1]

Failing unit test appears to be unrelated to this change, I've filed #34983 to resolve it.

[mx-psi]

Why is this option removed?

[odubajDT]

enabling the option led to re-running the failed tests (which mostly failed to to go routine leaks and race conditions), which always led to a passed test. This behavior was from some reason blocking the actual problems which were shown after it was removed

[mx-psi]

I added this in #31253 because we have a problem with flaky tests and not enough resources to deal with them. I feel like this is still the case and that if we remove this we will get a more flaky CI which is detrimental to contributors. Is there any way we can detect leaks while still re-running other unit tests?

[odubajDT]

I am not aware of any possibility here. In fact it seems very strange to me that all of the failing tests become passed after re-running them for the 2. time.

We can leave the re-run option in place until we find the root cause why the re-executed tests have very deterministic behavior in terms of go-leaks and data-races, but we should at least consider merging the fixed tests as part of this PR.

Maybe it can be a caching issue in the gotestsum wrapper, not sure here...

[mx-psi]

Can you report this upstream on the gotestsum repository then to see if it's actually an issue with them?

[odubajDT]

reported here

But there seems to be a bigger problem with the --rerun-fails option, as I see multiple issues created on the upstream related to it https://github.com/gotestyourself/gotestsum/issues