Make worker errors visible off-host by reporting them to Sentry

[mihow]

Summary

When the Antenna worker fails to process a job, the error currently goes only to the log file on the worker machine itself. Nobody sees it unless they SSH in and read the log. In a recent production incident, a GPU licensing problem on a worker host caused every ML job to fail for several days before anyone noticed, because the failures were invisible from anywhere but the machine.

This change reports those errors to Sentry, where the team already monitors the other AMI services, so worker failures show up off-host within minutes instead of days.

List of Changes

1. Errors that make a whole job attempt fail (for example a CUDA or model-loading problem) are now reported to Sentry — previously they were only written to the local log by the claim-loop handler.
2. Errors that make a single batch fail during processing are now also reported to Sentry — this handler already reports the error back to the Antenna API per task, and now the exception itself is captured too.

Technical note

The worker logs through structlog, which does not route records through the stdlib logging module. Sentry's default logging integration only listens to stdlib logging, so logger.error(...) calls never reached Sentry even though sentry_sdk.init() runs at import time. The fix is an explicit sentry_sdk.capture_exception(e) at the two points where exceptions are caught and swallowed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DkHC7gGrP9sumqawjHLeLP

Summary by CodeRabbit

• Bug Fixes
  • Improved error monitoring by capturing worker and batch-processing failures for diagnostic reporting.
  • Existing error handling and processing behavior remains unchanged.

[coderabbitai]

📝 Walkthrough

Walkthrough

The worker now imports Sentry and captures exceptions in both job-level and batch-level error handlers before continuing existing logging and error-response behavior.

Changes

Sentry exception reporting

Layer / File(s)	Summary
Worker exception capture trapdata/antenna/worker.py	Adds the Sentry import and reports exceptions from job processing and batch processing handlers before existing error handling continues.

Estimated code review effort: 1 (Trivial) | ~5 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: worker errors are now reported to Sentry so they're visible off-host.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/worker-sentry-capture

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

trapdata/antenna/worker.py (1)

120-123: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add regression coverage for both Sentry capture paths.

The supplied worker tests verify Antenna error responses, but do not assert that job-level and batch-level exceptions are reported to Sentry. Mock sentry_sdk.capture_exception and verify each handler invokes it once while preserving the existing response behavior.

Also applies to: 367-369

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@trapdata/antenna/worker.py` around lines 120 - 123, Add regression tests for
both the job-level and batch-level exception handlers in the worker, mocking
sentry_sdk.capture_exception and asserting each handler calls it exactly once
with the raised exception while preserving the existing Antenna error responses.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@trapdata/antenna/worker.py`:
- Around line 120-123: Add regression tests for both the job-level and
batch-level exception handlers in the worker, mocking
sentry_sdk.capture_exception and asserting each handler calls it exactly once
with the raised exception while preserving the existing Antenna error responses.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 09a402a8-546c-4eb4-bca9-24cd70a6ed60

📥 Commits

Reviewing files that changed from the base of the PR and between a33746a and 3af89e5.

📒 Files selected for processing (1)

• trapdata/antenna/worker.py