[Feature] Flaky Tests - retry all tests at end of the test run

[mstepin]

It would be ideal to have a configuration option where flaky tests are retried at the end of the run once all other tests are finished. I see it was discussed here, but that one was closed and not sure the concept was since revisited.

This becomes a lot more relevant when using worker-scoped fixtures, since the thread gets torn down and all the auto: true worker-scoped fixtures are retriggered for the new thread. Since many of these startup actions are expensive both in time and resource, it makes more sense for the failed tests to share new threads at the end of the run.

My personal vision of a solution would be a config setting in playwright.config file eg retry-strategy: 'tailing'. Doing it this way would also open the opportunity for other configurations such as not starting a new thread on failure, or max number of hard fails, conditional retries etc.

[Vovencia]

You can use postinstall to replace one line in the file:

You need to replace this._queue.unshift(result.newJob) with this._queue.push(result.newJob)

Source file: https://github.com/microsoft/playwright/blob/release-1.40/packages/playwright/src/runner/dispatcher.ts#L126
Compiled file (in your project): /node_modules/@playwright/test/lib/runner/dispatcher.js

---

Можно с помощью postinstall заменить в файле одну строчку:

Нужно заменить this._queue.unshift(result.newJob) на this._queue.push(result.newJob)

Исходный файл: https://github.com/microsoft/playwright/blob/release-1.40/packages/playwright/src/runner/dispatcher.ts#L126
Скомпилированный файл (в вашем проекте): /node_modules/@playwright/test/lib/runner/dispatcher.js

[zargham-leanix]

+1

[joeflan]

This would be pretty useful for our team as well

[olekach]

+1. It's useful for the case when there is an env issue and all subsequent retries will fail but if they are postponed for a bit later, chances are the test will succeed. Would be good to dynamically manage this.

[AndiBMGitHub]

+1 I would like this to be implemented :)

[DerTimonius]

+1
This would be really useful when resetting the test data only once in the global setup and the flaky test now does not have a clean state it had on the initial run

[marcusNumminen]

+1 one for this

[philipfong]

+1 I'd like to see this too. This is also useful for when some specific tests may fail when sharded and the system is under heavy load, and then may have a better chance of succeeding when the number of shards tail off.

I know this all speaks to generally poor systems under test, but that's kind of the reality that we have to test with much of the time.

[TuxThunderbird]

+1 for this.