Skip to content

Report event post queue failures and bound payload decompression - #2646

Open
ejsmith wants to merge 1 commit into
mainfrom
issue/event-post-accept-correctness
Open

ejsmith wants to merge 1 commit into
mainfrom
issue/event-post-accept-correctness

Conversation

@ejsmith

@ejsmith ejsmith commented Oct 7, 2026

Copy link
Copy Markdown
Member

Summary

PR 1 of the ingestion plan in #2368 (plan): make the existing accept path honest about failures and tell clients when to retry.

  • No more silent loss on queue failures. When the payload could not be saved or the queue entry could not be created, POST and GET event submissions used to return 202/200, and the events were never processed. They now return 503 with Retry-After: 30, and the saved files are removed. EventPostService.SaveAndEnqueueAsync also turns storage and queue exceptions into the same failure result instead of leaving orphaned files.
  • Retry-After on every retryable response:
    • 503 when event submission is disabled: 300 seconds.
    • Throttling 429: the seconds until the throttle window resets.
  • Bounded decompression in EventPostsJob. Payloads are decompressed with the uncompressed size limit applied while reading. Previously the whole payload was expanded into memory before the size check, so a highly compressed post could exhaust job memory.
  • EventPostsJob fixes:
    • Posts over the file size limit now have their payload deleted when the entry completes.
    • The uncompressed size metric now records the decompressed length.
  • Stale payload cleanup. CleanupDataJob deletes queued event post files (q/) older than seven days. Those files are left behind when a queue entry is dead-lettered. Seven days keeps the window for an administrator to requeue them; events older than three days are discarded by the pipeline anyway.

Compatibility

  • The only API change is the new 503 response, documented on every event submission endpoint, plus Retry-After headers. The OpenAPI snapshot diff contains only those 503 entries.
  • The official clients already treat 503 as retryable and keep the batch. The .NET client suspends for 5 minutes; the JavaScript client suspends until the next 15-minute boundary.

Verification

  • dotnet build Exceptionless.slnx: 0 warnings, 0 errors.
  • Full backend suite: 3,162 passed, 0 failed, 3 skipped.
  • New tests:
    • Queue returns no entry, and queue throws: the result is a failure and no files remain.
    • The throttle 429 carries Retry-After, and the disabled-submission 503 carries Retry-After: 300.
    • Bounded gzip/deflate decompression.
    • An oversized payload file is deleted.
    • Queued files are kept while recent and deleted once stale; archive files are untouched.

- Return 503 with Retry-After when an event post cannot be queued instead
  of acknowledging events that will never be processed, and remove the
  saved payload files.
- Send Retry-After when event submission is disabled and when a request
  is throttled.
- Decompress queued payloads with a size limit so a highly compressed
  post cannot expand into unbounded memory before its size is checked.
- Delete the payload when a post exceeds the size limit, and record the
  uncompressed size metric from the decompressed payload.
- Delete queued event post files older than seven days, which are left
  behind when a queue entry is dead-lettered.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-07T13:14:21.972314Z 2aa8e7e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2aa8e7ed43

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

.Produces<WorkInProgressResult>(StatusCodes.Status202Accepted)
.ProducesProblem(StatusCodes.Status400BadRequest)
.ProducesProblem(StatusCodes.Status404NotFound)
.ProducesProblem(StatusCodes.Status503ServiceUnavailable)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove the spurious 503 response from event deletion

Remove this response declaration and its snapshot entry: DeleteEvents only reads and removes stored events and never invokes the event-post queue or EventQueueUnavailable, so this change incorrectly advertises a new public response that the endpoint cannot produce. Generated clients will consequently model a queue-unavailable failure for an unrelated delete operation.

AGENTS.md reference: AGENTS.md:L72-L72

Useful? React with 👍 / 👎.

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown

Code Coverage

Package Line Rate Branch Rate Complexity Health
Exceptionless.AppHost 23% 23% 128 ❌
Exceptionless.Core 77% 68% 10854 ✔
Exceptionless.Insulation 51% 43% 370 ➖
Exceptionless.Web 86% 70% 9178 ✔
Summary 80% (27899 / 34966) 69% (13780 / 20098) 20530 ✔

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant