fix(server): record what the provider says happened, not just handoff - #321
Merged
Merged
Conversation
Part of #311. Stacked on #319 (fix/313-permanent-send-errors). `notifications.status = 'sent'` has only ever meant "the provider's API returned 2xx" — a handoff receipt, not a delivery receipt. Twilio answers `201 queued` and then fails a message asynchronously (63007 unknown sender, 63016 outside the WhatsApp session window, 30034 unregistered 10DLC). None of it reached us: the row read `sent`, `attempts=1`, `last_error` NULL, and /health said the notifier was healthy while a heir's practice-drill invite sat "sent" for six days having never existed (mainnet notification id 40). Adds the inbound half: - migration: provider_message_id, delivery_status, delivery_detail, delivery_updated_at on notifications, plus their indexes - send_twilio attaches a StatusCallback and returns the Message SID; send_email returns the SMTP queue reply, the only correlation handle an SMTP send offers. Both are recorded at handoff. - POST /webhooks/twilio/status, authenticated by Twilio's HMAC-SHA1 request signature. The signed URL is rebuilt from the same status_callback_url() the sender used, NOT from request headers — Host and X-Forwarded-Proto are attacker-influenced, and trusting them turns the check into decoration. - an undelivered/failed verdict writes a notification_undelivered event, so it lands in the owner's activity feed instead of a log line nobody reads - /health gains notifications_undelivered, counted separately because those rows still say status='sent' WorkerOutcome::Sent now carries the provider id, and the log line says "handed to provider" rather than "sent". No new secret and no Twilio console setup: StatusCallback is per message and the signature uses the existing auth token. The callback is only attached when GHOSTKEY_PUBLIC_BASE_URL is https, since Twilio fetches it from the public internet. hmac + sha1 become direct dependencies; both were already in the lock file transitively, so this adds no supply-chain surface. DEPLOY.md's Twilio section was stale on three counts (a 555 example sender that #317 now rejects, "all four are required together" which sender independence ended, and "stay queued" which #282 ended). Tests: HMAC-SHA1 pinned to RFC 2202 case 1; a full signature worked example whose expected value was computed independently in Python, not copied from this implementation; order-independence and token-dependence; undelivered claim link recorded AND surfaced as an event; delivered recorded without raising one; unknown SID ignored rather than erroring. Stubbing is_negative_verdict to false fails the surfacing tests. NOT covered here: Resend bounce/complaint webhooks (issue #311 item 3). That needs a webhook secret configured on the deployment and an id-matching strategy verified against a real Resend 250 reply, which the SMTP path can't give us blind. Filed separately. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #311. Stacked on #319 — base is
fix/313-permanent-send-errors, so review that first. Retarget tomainonce #319 merges.The bug
notifications.status = 'sent'has only ever meant "the provider's API returned 2xx". That is a handoff receipt, not a delivery receipt.Twilio answers
201 queuedand then fails a message asynchronously — 63007 (sender not a known channel), 63016 (free-form outside the 24h WhatsApp session window), 30034 (unregistered 10DLC). None of it reached us. The row readsent,attempts=1,last_errorNULL, and/healthreported the notifier healthy the whole time.That is how a heir's practice-drill invite sat "sent" for six days having never existed (mainnet notification id 40, the drill that never completed).
What this adds
provider_message_id,delivery_status,delivery_detail,delivery_updated_atonnotifications, plus indexes for the webhook lookup and the health count.send_twilioreturns the Message SID;send_emailreturns the SMTP queue reply, the only correlation handle an SMTP send offers.WorkerOutcome::Sentcarries it, and the log line now reads "handed to provider" rather than "sent".POST /webhooks/twilio/status— one callback per status transition, authenticated by Twilio's HMAC-SHA1 request signature.undelivered/failedverdict writes anotification_undeliveredevent, so it lands in the vault's activity feed. Issue item 4 was explicit that this must not be a log-only signal./healthgainsnotifications_undelivered, counted separately because those rows still saystatus='sent'and so are invisible tonotifications_failed.Security note on the signature
The signed payload includes the URL Twilio called, which has to match byte for byte. It is rebuilt from the same
status_callback_url()the sender used — deliberately not fromHost/X-Forwarded-Proto, which are attacker-influenced. Getting that wrong makes the check decoration.Signature comparison goes through
subtle's constant-time equality.Unknown SIDs answer 2xx and are ignored: the Twilio account may be shared with another deployment, and a non-2xx would make Twilio retry forever.
Operational cost: none
No new secret, no Twilio console setup.
StatusCallbackis per-message and the signature uses the existing auth token. The callback is only attached whenGHOSTKEY_PUBLIC_BASE_URLishttps://, since Twilio fetches it from the public internet — local deployments still send, they just get no verdicts.hmacandsha1become direct dependencies. Both were already inCargo.locktransitively, so this adds no supply-chain surface.Docs
DEPLOY.md's Twilio section was stale on three counts: a
555example sender that #317 now rejects, "all four are required together" which sender independence ended, and "notifications stay queued" which #282 ended.Tests
deliveredrecorded without raising an eventStubbing
is_negative_verdicttofalsefails the surfacing tests.237 passed, 0 failed. clippy
-D warningsclean,cargo fmt --checkclean.Not covered here
Resend bounce/complaint webhooks (issue item 3). That needs a webhook secret configured on the deployment and an id-matching strategy verified against a real Resend
250reply, which I can't confirm without sending through the live account. This PR stores the SMTP reply so that correlation is possible later. Worth a follow-up issue before closing #311.🤖 Generated with Claude Code