-
-
Notifications
You must be signed in to change notification settings - Fork 86
Labels
Description
We need a mechanism which would automatically retry failing selenium tests a couple of times before flagging them as failed.
With the help of ChatGPT I generated a prototype for this:
import unittest
import time
import traceback
import sys
class SeleniumTestMixin:
retry_max = 2
retry_delay = 0 # seconds
def run(self, result=None):
"""
Override unittest.TestCase.run to retry flaky tests.
"""
original_result = result
test_name = self.id()
for attempt in range(self.retry_max + 1):
temp_result = self.defaultTestResult()
super().run(temp_result)
if temp_result.wasSuccessful():
if original_result is not None:
original_result.addSuccess(self)
return
if attempt < self.retry_max:
print("-" * 80)
print(
f"[Retry] Test '{test_name}' failed on attempt {attempt + 1}. Retrying..."
)
print("-" * 80)
if self.retry_delay:
time.sleep(self.retry_delay)
continue
else:
for failed_test, err in temp_result.failures + temp_result.errors:
print("-" * 80)
print(f"[Traceback for '{failed_test.id()}']:\n")
print(err.strip())
print("-" * 80)
if original_result is not None:
for failure in temp_result.failures:
original_result.addFailure(*failure)
for error in temp_result.errors:
original_result.addError(*error)We need to update the docs to state that selenium tests are automatically retried and document retry_max and retry_delay, plus do some manual testing to ensure that the messages printed when flaky tests are retried are visible and readable.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Done
Status
Done