Skip to content

Feat/retry for api calls #66

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
AenEnlil wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Feat/retry for api calls #66

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
589795b
implemented retry logic
AenEnlil Sep 30, 2025
9ea2a41
apply retry logic for api methods
AenEnlil Sep 30, 2025
a75dc54
apply retry logic for get tests, updated early exit logic when test r…
AenEnlil Oct 1, 2025
d6836e2
updated unit tests to cover retry logic
AenEnlil Oct 1, 2025
1a3a6d6
updated README.md
AenEnlil Oct 1, 2025
f9d0e42
bump: version 2.10.2 -> 2.10.3b0
AenEnlil Oct 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,14 +175,16 @@ Note: Test id should be started from letter "T"
You can use environment variable to control certain features of testomat.io

#### Basic configuration
| Env variable | What it does | Examples |
|--------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------|
| TESTOMATIO | Provides token for pytestomatio to access and push data to testomat.io. Required for **sync** and **report** commands | TESTOMATIO=tstmt_***** pytest --testomatio sync |
| Env variable | What it does | Examples |
|--------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------|
| TESTOMATIO | Provides token for pytestomatio to access and push data to testomat.io. Required for **sync** and **report** commands | TESTOMATIO=tstmt_***** pytest --testomatio sync |
| TESTOMATIO_SYNC_LABELS | Assign labels to a test case when you synchronise test from code with testomat.io. Labels must exist in project and their scope must be enabled for tests | TESTOMATIO_SYNC_LABELS="number:1,list:one,standalone" pytest --testomatio report |
| TESTOMATIO_CODE_STYLE | Code parsing style for test synchronization. If you are not sure, don't set this variable. Default value is 'default' | TESTOMATIO_CODE_STYLE=pep8 pytest --testomatio sync |
| TESTOMATIO_CI_DOWNSTREAM | If set, pytestomatio will not set or update build url for a test run. This is useful in scenarios where build url is already set in the test run by Testomat.io for test runs that a created directly on Testomat.io. | TESTOMATIO_CI_DOWNSTREAM=true pytest --testomatio report |
| TESTOMATIO_URL | Customize testomat.io url | TESTOMATIO_URL=https://custom.com/ pytest --testomatio report |
| BUILD_URL | Overrides build url run tests | BUILD_URL=http://custom.com/ pytest --testomatio report |
| TESTOMATIO_CODE_STYLE | Code parsing style for test synchronization. If you are not sure, don't set this variable. Default value is 'default' | TESTOMATIO_CODE_STYLE=pep8 pytest --testomatio sync |
| TESTOMATIO_CI_DOWNSTREAM | If set, pytestomatio will not set or update build url for a test run. This is useful in scenarios where build url is already set in the test run by Testomat.io for test runs that a created directly on Testomat.io. | TESTOMATIO_CI_DOWNSTREAM=true pytest --testomatio report |
| TESTOMATIO_URL | Customize testomat.io url | TESTOMATIO_URL=https://custom.com/ pytest --testomatio report |
| BUILD_URL | Overrides build url run tests | BUILD_URL=http://custom.com/ pytest --testomatio report |
| TESTOMATIO_MAX_REQUEST_FAILURES | Sets the max number of attempts to send a request to the Testomat.io API. Default is 5 attempts. | TESTOMATIO_MAX_REQUEST_FAILURES=10 pytest --testomatio report |
| TESTOMATIO_REQUEST_INTERVAL | Sets the interval between API requests in seconds. Default is 5 sec. | TESTOMATIO_REQUEST_INTERVAL=2 pytest --testomatio report |


#### Test Run configuration
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ version_provider = "pep621"
update_changelog_on_bump = false
[project]
name = "pytestomatio"
version = "2.10.2"
version = "2.10.3b0"


dependencies = [
Expand Down
135 changes: 89 additions & 46 deletions pytestomatio/connect/connector.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,21 +1,31 @@
import os

import requests
from requests.exceptions import HTTPError, ConnectionError
import logging
from os.path import join, normpath
from os import getenv

from pytestomatio.connect.exception import MaxRetriesException
from pytestomatio.utils.helper import safe_string_list
from pytestomatio.testing.testItem import TestItem
import time

log = logging.getLogger('pytestomatio')
MAX_RETRIES_DEFAULT = 5
RETRY_INTERVAL_DEFAULT = 5


class Connector:
def __init__(self, base_url: str = '', api_key: str = None):
max_retries = os.environ.get('TESTOMATIO_MAX_REQUEST_FAILURES', '')
retry_interval = os.environ.get('TESTOMATIO_REQUEST_INTERVAL', '')
self.base_url = base_url
self._session = requests.Session()
self.jwt: str = ''
self.api_key = api_key
self.max_retries = int(max_retries) if max_retries.isdigit() else MAX_RETRIES_DEFAULT
self.retry_interval = int(retry_interval) if retry_interval.isdigit() else RETRY_INTERVAL_DEFAULT

@property
def session(self):
Expand Down Expand Up @@ -66,6 +76,47 @@ def _test_proxy_connection(self, test_url="https://api.ipify.org?format=json", t
log.error("Internet connection check timed out after %d seconds.", timeout)
return False

def _should_retry(self, response: requests.Response) -> bool:
"""Checks if request should be retried.
Skipped status codes explanation:
400 - Bad request(probably wrong API key)
404 - Resource not found. No point to retry request.
429 - Limit exceeded
500 - Internal server error
"""
if response.status_code in (400, 404, 429, 500):
return False
return response.status_code >= 401

def _send_request_with_retry(self, method: str, url: str, **kwargs):
"""Send HTTP request with retry logic"""
for attempt in range(self.max_retries):
log.debug(f'Trying to send request to {self.base_url}. Attempt {attempt+1}/{self.max_retries}')
try:
request_func = getattr(self.session, method)
response = request_func(url, **kwargs)

if self._should_retry(response):
if attempt < self.max_retries:
log.error(f'Request attempt failed. Response code: {response.status_code}. '
f'Retrying in {self.retry_interval} seconds')
time.sleep(self.retry_interval)
continue

return response
except ConnectionError as ce:
log.error(f'Failed to connect to {self.base_url}: {ce}')
raise
except HTTPError as he:
log.error(f'HTTP error occurred while connecting to {self.base_url}: {he}')
raise
except Exception as e:
log.error(f'An unexpected exception occurred. Please report an issue: {e}')
raise

log.error(f'Retries attempts exceeded.')
raise MaxRetriesException()

def load_tests(
self,
tests: list[TestItem],
Expand All @@ -75,6 +126,7 @@ def load_tests(
create: bool = False,
directory: str = None
):
url = f'{self.base_url}/api/load?api_key={self.api_key}'
request = {
"framework": "pytest",
"language": "python",
Expand All @@ -98,16 +150,11 @@ def load_tests(
"labels": safe_string_list(getenv('TESTOMATIO_SYNC_LABELS')),
})

log.info(f'Starting tests loading to {self.base_url}')
try:
response = self.session.post(f'{self.base_url}/api/load?api_key={self.api_key}', json=request)
except ConnectionError as ce:
log.error(f'Failed to connect to {self.base_url}: {ce}')
return
except HTTPError as he:
log.error(f'HTTP error occurred while connecting to {self.base_url}: {he}')
return
response = self._send_request_with_retry('post', url, json=request)
except Exception as e:
log.error(f'An unexpected exception occurred. Please report an issue: {e}')
log.error(f'Failed to load tests to {self.base_url}')
return

if response.status_code < 400:
Expand All @@ -116,9 +163,18 @@ def load_tests(
log.error(f'Failed to load tests to {self.base_url}. Status code: {response.status_code}')

def get_tests(self, test_metadata: list[TestItem]) -> dict:
# with safe_request('Failed to get test ids from testomat.io'):
response = self.session.get(f'{self.base_url}/api/test_data?api_key={self.api_key}')
return response.json()
log.info('Trying to receive test ids from testomat.io')
url = f'{self.base_url}/api/test_data?api_key={self.api_key}'
try:
response = self._send_request_with_retry('get', url)
if response.status_code < 400:
log.info('Test ids received')
return response.json()
else:
log.error('Failed to get test ids from testomat.io')
except Exception as e:
log.error('Failed to get test ids from testomat.io')


def create_test_run(self, access_event: str, title: str, group_title, env: str, label: str, shared_run: bool, shared_run_timeout: str,
parallel, ci_build_url: str) -> dict | None:
Expand All @@ -135,21 +191,19 @@ def create_test_run(self, access_event: str, title: str, group_title, env: str,
"shared_run_timeout": shared_run_timeout,
}
filtered_request = {k: v for k, v in request.items() if v is not None}
url = f'{self.base_url}/api/reporter'
log.info('Creating test run')
try:
response = self.session.post(f'{self.base_url}/api/reporter', json=filtered_request)
except ConnectionError as ce:
log.error(f'Failed to connect to {self.base_url}: {ce}')
return
except HTTPError as he:
log.error(f'HTTP error occurred while connecting to {self.base_url}: {he}')
return
response = self._send_request_with_retry('post', url, json=filtered_request)
except Exception as e:
log.error(f'An unexpected exception occurred. Please report an issue: {e}')
log.error(f'Failed to create test run')
return

if response.status_code == 200:
log.info(f'Test run created {response.json()["uid"]}')
return response.json()
else:
log.error('Failed to create test run')

def update_test_run(self, id: str, access_event: str, title: str, group_title,
env: str, label: str, shared_run: bool, shared_run_timeout: str, parallel, ci_build_url: str) -> dict | None:
Expand All @@ -166,22 +220,19 @@ def update_test_run(self, id: str, access_event: str, title: str, group_title,
"shared_run_timeout": shared_run_timeout
}
filtered_request = {k: v for k, v in request.items() if v is not None}

url = f'{self.base_url}/api/reporter/{id}'
log.info(f'Updating test run. Run Id: {id}')
try:
response = self.session.put(f'{self.base_url}/api/reporter/{id}', json=filtered_request)
except ConnectionError as ce:
log.error(f'Failed to connect to {self.base_url}: {ce}')
return
except HTTPError as he:
log.error(f'HTTP error occurred while connecting to {self.base_url}: {he}')
return
response = self._send_request_with_retry('put', url, json=filtered_request)
except Exception as e:
log.error(f'An unexpected exception occurred. Please report an issue: {e}')
log.error(f'Failed to update test run')
return

if response.status_code == 200:
log.info(f'Test run updated {response.json()["uid"]}')
return response.json()
else:
log.error('Failed to update test_run')

def update_test_status(self, run_id: str,
rid: str,
Expand All @@ -200,6 +251,7 @@ def update_test_status(self, run_id: str,
overwrite: bool | None,
meta: dict) -> None:

log.info(f'Reporting test. Id: {test_id}. Title: {title}')
request = {
"status": status, # Enum: "passed" "failed" "skipped"
"title": title,
Expand All @@ -218,35 +270,26 @@ def update_test_status(self, run_id: str,
"meta": meta
}
filtered_request = {k: v for k, v in request.items() if v is not None}
url = f'{self.base_url}/api/reporter/{run_id}/testrun?api_key={self.api_key}'
try:
response = self.session.post(f'{self.base_url}/api/reporter/{run_id}/testrun?api_key={self.api_key}',
json=filtered_request)
except ConnectionError as ce:
log.error(f'Failed to connect to {self.base_url}: {ce}')
return
except HTTPError as he:
log.error(f'HTTP error occurred while connecting to {self.base_url}: {he}')
return
response = self._send_request_with_retry('post', url, json=filtered_request)
except Exception as e:
log.error(f'An unexpected exception occurred. Please report an issue: {e}')
log.error(f'Failed to report test')
return
if response.status_code == 200:
log.info('Test status updated')
else:
log.error('Failed to report test')

# TODO: I guess this class should be just an API client and used within testRun (testRunConfig)
def finish_test_run(self, run_id: str, is_final=False) -> None:
log.info(f'Finishing test run. Run id: {run_id}')
status_event = 'finish_parallel' if is_final else 'finish'
url = f'{self.base_url}/api/reporter/{run_id}?api_key={self.api_key}'
try:
self.session.put(f'{self.base_url}/api/reporter/{run_id}?api_key={self.api_key}',
json={"status_event": status_event})
except ConnectionError as ce:
log.error(f'Failed to connect to {self.base_url}: {ce}')
return
except HTTPError as he:
log.error(f'HTTP error occurred while connecting to {self.base_url}: {he}')
return
self._send_request_with_retry('put', url, json={"status_event": status_event})
except Exception as e:
log.error(f'An unexpected exception occurred. Please report an issue: {e}')
log.error(f'Failed to finish test run')
return

def disconnect(self):
Expand Down
4 changes: 4 additions & 0 deletions pytestomatio/connect/exception.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@


class MaxRetriesException(Exception):
pass
6 changes: 4 additions & 2 deletions pytestomatio/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,8 @@ def pytest_collection_modifyitems(session: Session, config: Config, items: list[
directory=config.getoption('directory')
)
testomatio_tests = pytest.testomatio.connector.get_tests(meta)
if not testomatio_tests:
pytest.exit('Failed to update tests ids')
add_and_enrich_tests(meta, test_files, test_names, testomatio_tests, decorator_name)
pytest.exit('Sync completed without test execution')
case 'remove':
Expand All @@ -136,8 +138,8 @@ def pytest_collection_modifyitems(session: Session, config: Config, items: list[
run_details = pytest.testomatio.connector.update_test_run(**run.to_dict())

if run_details is None:
log.error('Test run failed to create. Reporting skipped')
return
log.error('Test run not found. Reporting skipped')
pytest.exit('Reporting skipped')

s3_details = read_env_s3_keys(run_details)

Expand Down
Loading
Loading