Skip to content

[tests] Retry failing selenium tests #464

@nemesifier

Description

@nemesifier

We need a mechanism which would automatically retry failing selenium tests a couple of times before flagging them as failed.

With the help of ChatGPT I generated a prototype for this:

import unittest
import time
import traceback
import sys


class SeleniumTestMixin:
    retry_max = 2
    retry_delay = 0  # seconds

    def run(self, result=None):
        """
        Override unittest.TestCase.run to retry flaky tests.
        """
        original_result = result
        test_name = self.id()

        for attempt in range(self.retry_max + 1):
            temp_result = self.defaultTestResult()
            super().run(temp_result)

            if temp_result.wasSuccessful():
                if original_result is not None:
                    original_result.addSuccess(self)
                return

            if attempt < self.retry_max:
                print("-" * 80)
                print(
                    f"[Retry] Test '{test_name}' failed on attempt {attempt + 1}. Retrying..."
                )
                print("-" * 80)
                if self.retry_delay:
                    time.sleep(self.retry_delay)
                continue
            else:
                for failed_test, err in temp_result.failures + temp_result.errors:
                    print("-" * 80)
                    print(f"[Traceback for '{failed_test.id()}']:\n")
                    print(err.strip())
                    print("-" * 80)

                if original_result is not None:
                    for failure in temp_result.failures:
                        original_result.addFailure(*failure)
                    for error in temp_result.errors:
                        original_result.addError(*error)

We need to update the docs to state that selenium tests are automatically retried and document retry_max and retry_delay, plus do some manual testing to ensure that the messages printed when flaky tests are retried are visible and readable.

Metadata

Metadata

Assignees

No one assigned

    Projects

    Status

    Done

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions