[Feature] Fail suite on retried fail

[dimkin-eu]

As for now, there are 2 options available for retries and fast fails
--retries=1 --max-failures=1

If the test is flaky even 1st fail will stop the suite. setting --max-failures=2, will fail on 2 flaky tests failures, which retry would pass.

Could be nice if --max-failures=1 would fail on hard fail, i.e. flaky test failed also in retry

[pavelfeldman]

I started looking into implementing it and stumbled upon something: I was thinking of moving retries to the very end - this strategy worked well for us in Chromium:

• run all the tests
• retry failures 1
• retry remaining failures 2
• retry remaining failures 3

That way all the flaky tests are retried in the very end of the run. Proposed semantics of max-failures would make it useless in case of that strategy.

So I'm going back to understanding your use case better. Could you share more why you need it?

[dimkin-eu]

--retries=1 makes sure, that after 1 retry test will pass
--max-failures=1 makes sure, that after 1 test failed, the suite fails

as for now, each failure counts, even if the test will pass after retry

so if n tests will fail, but be passing after retries, there must be 0 failures
if 1 test from that n will not pass after all configured retries- this must be +1 to the failures count
( with some extra option, for sure like --countHardFails=true or something)

[pavelfeldman]

I understand the rationale, but I'd like to understand the use case as well. Let's say we introduce --max-hard-failures=1, but will only retry tests in the end of the test run, would that help solving your use case?

[dimkin-eu]

kinda fail fast, but slower :)
retry is at the end of the spec, but anyway in the middle ( or even start) of the suite
as specs are running in parallel with limited workers it will anyway save some time

ah, yes, this will work even for 1 serial spec for many flaky tests - it will save time for retrying the rest if at least one is failed :)

(and yes, I'm not a big fan of retries at all. but e2e could depend on 3rd party, sadly )

[pavelfeldman]

Hang on, with the new strategy, we will run 1000 tests first (no fail fast yet), then we will re-run failed 10 once (no fail fast) and then we re-run failed 5 third time (now you'll get fail fast-ish). So is you say --max-hard-failures you end up running 10010 tests instead of 10015, which does not sounds like a fail fast to me.

[dimkin-eu]

from another perspective -
10 specs, with 10 tests in each, running using 10 browsers -> 1k tests
with --retries=1 it passes in 99.9999%, but if it fails in 1st spec, the suite will run till the end - all 1k tests ( in place of 10-ish)
with --max-failures=1 suite will be failed immediately, (but if the failed test would be retried the whole suite would pass)

I understand, this is a try-not-to-shoot-in-own-leg feature :)

[pavelfeldman]

We had a good conversation offline, arriving at setting max-failures to something like 20 - that is a handful of flaky tests that have failed. It would still allow early recovery from the broken infrastructure and would allow us to push retries into the end of the run.

[pavelfeldman]

Closing as per above, please feel free to open a new issue if this does not cover your use case.

[cgetzen]

We had a good conversation offline, arriving at setting max-failures to something like 20 - that is a handful of flaky tests that have failed. It would still allow early recovery from the broken infrastructure and would allow us to push retries into the end of the run.

Hi @pavelfeldman, I don't understand how this helps when there is a "true" test failure and the total tests failure count is < 20: the suite will fail, but run to completion (no "fail-fast").

[debasisj]

just a question -max-failures will overwrite any retries?
For example in my test suite --max-failures=1 set as 1 but i want to retry a single test multiple times .. but as soon we have first failure - no retries happens