Skip to content

Conversation

@vizath
Copy link

@vizath vizath commented May 27, 2025

Recently, we had received a few 503 from axiom logging.
When the Client receives a 503 after few retries, it raise an Exception.
That's problematic because it also crashes the whole app.

How to reproduce in a production python app:

  • update your logger with a 503 error: axiom_py.Client(token=token, url_base="https://httpstat.us/503")
  • Have a FastAPI endpoint creating a log
  • See the logging crashing the route

Another solution would be to move the try/except in the flush function of logging.py.

I'm not super fluent in python, feel free to propose changes.

Some related stack trace:

Exception in thread Thread-8:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.12/threading.py", line 1433, in run
    self.function(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.12/site-packages/axiom_py/logging.py", line 67, in flush
    self.client.ingest_events(self.dataset, local_buffer)
  File "/usr/local/lib/python3.12/site-packages/axiom_py/client.py", line 262, in ingest_events
    return self.ingest(
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/axiom_py/client.py", line 239, in ingest
    res = self.session.post(
          ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests_toolbelt/sessions.py", line 76, in request
    return super(BaseUrlSession, self).request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 710, in send
    r = dispatch_hook("response", hooks, r, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/hooks.py", line 30, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/axiom_py/client.py", line 182, in <lambda>
    "response": lambda r, *args, **kwargs: raise_response_error(r)
                                           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/axiom_py/client.py", line 145, in raise_response_error
    raise AxiomError(res.status_code, error_res)
axiom_py.client.AxiomError: API error 404: Not Found

@vizath
Copy link
Author

vizath commented Jun 26, 2025

Any wish to have this reviewed? Axiom ingestion was down again last night between 2:28am UTC - 3:20am UTC, propagating the downtime to our endpoints.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant