oss_13_http_exporter_retry_after_backoff_cap #18

Activity

1. ocelotl commented on Jul 22, 2026

   

   ocelotl

   on Jul 22, 2026

   OwnerAuthor

   More actions

   🔒 Internal (dash0) — not for upstream.

   Tracking for the OTLP HTTP exporter retry fix.

   • Linear: https://linear.app/dash0/issue/OSS-13/x2-x3-http-exporter-honor-429retry-after-add-max-backoff-cap (findings X2/X3)
   • Slug/branch/worktree: oss_13_http_exporter_retry_after_backoff_cap
   • Commit: c29c774bc · PR: oss_13_http_exporter_retry_after_backoff_cap #27
   • Validation: HTTP suite 73 passed, gRPC suite 67 passed; scope otlp http+grpc + changelog.
2. ocelotl commented on Jul 22, 2026

   

   ocelotl

   on Jul 22, 2026

   OwnerAuthor

   More actions

   📣 Public-facing draft — to be opened upstream in open-telemetry/opentelemetry-python. No internal references; copy verbatim.

   Title: OTLP HTTP exporter ignores Retry-After, does not retry on 429, and has no backoff cap

   Description

   The OTLP HTTP exporters (trace, metric, logs) do not conform to the OTLP exporter spec's retry requirements in three ways.

   1. HTTP 429 Too Many Requests is not retried

   exporter/opentelemetry-exporter-otlp-proto-http/.../_common/__init__.py, _is_retryable() only retries 408 and 5xx:

   def _is_retryable(resp):
       if resp.status_code == 408:
           return True
       if resp.status_code >= 500 and resp.status_code <= 599:
           return True
       return False

   So 429 is treated as permanent and the batch is dropped. The OTLP spec (specification/protocol/exporter.md) requires retrying on 429; the gRPC sibling already honors throttling via the google.rpc.retryinfo-bin trailer.

   2. The Retry-After response header is never read

   The retry loops in trace_exporter/__init__.py, _log_exporter/__init__.py, and metric_exporter/__init__.py compute the delay solely from exponential backoff and never inspect Retry-After, which servers send with 429/503. The spec requires honoring it; Go and Java both parse it (delay-seconds and HTTP-date forms).

   3. The exponential backoff is uncapped

   Each exporter computes backoff = 2**retry_num * random.uniform(0.8, 1.2) with _MAX_RETRYS = 6, reaching ~32s with no clamp; the gRPC exporter has the same uncapped pattern and applies the server retry_delay with no ceiling. Go and Java cap the interval.

   Proposed fix

   • Add 429 to the retryable set in _is_retryable.
   • Add a Retry-After parser (delay-seconds integer and HTTP-date) and use its value as the next-retry delay across the trace/metric/log HTTP exporters.
   • Add a MAX_BACKOFF clamp (min(backoff, MAX_BACKOFF)) to the HTTP exporters and the gRPC exporter.
3. ocelotl commented on Sep 9, 2026

   

   ocelotl

   on Sep 9, 2026

   OwnerAuthor

   More actions

   Fixed by open-telemetry#5252 open-telemetry#5389
4. ocelotl

   closed this as completedon Sep 9, 2026