Skip to content

fix(server): record what the provider says happened, not just handoff - #321

Merged
Jolah1 merged 1 commit into
fix/313-permanent-send-errorsfrom
fix/311-delivery-status
Jul 29, 2026
Merged

fix(server): record what the provider says happened, not just handoff#321
Jolah1 merged 1 commit into
fix/313-permanent-send-errorsfrom
fix/311-delivery-status

Conversation

@Jolah1

@Jolah1 Jolah1 commented Jul 29, 2026

Copy link
Copy Markdown
Owner

Part of #311. Stacked on #319 — base is fix/313-permanent-send-errors, so review that first. Retarget to main once #319 merges.

The bug

notifications.status = 'sent' has only ever meant "the provider's API returned 2xx". That is a handoff receipt, not a delivery receipt.

Twilio answers 201 queued and then fails a message asynchronously — 63007 (sender not a known channel), 63016 (free-form outside the 24h WhatsApp session window), 30034 (unregistered 10DLC). None of it reached us. The row read sent, attempts=1, last_error NULL, and /health reported the notifier healthy the whole time.

That is how a heir's practice-drill invite sat "sent" for six days having never existed (mainnet notification id 40, the drill that never completed).

What this adds

  • Migrationprovider_message_id, delivery_status, delivery_detail, delivery_updated_at on notifications, plus indexes for the webhook lookup and the health count.
  • Handoff idssend_twilio returns the Message SID; send_email returns the SMTP queue reply, the only correlation handle an SMTP send offers. WorkerOutcome::Sent carries it, and the log line now reads "handed to provider" rather than "sent".
  • POST /webhooks/twilio/status — one callback per status transition, authenticated by Twilio's HMAC-SHA1 request signature.
  • Surfacing — an undelivered/failed verdict writes a notification_undelivered event, so it lands in the vault's activity feed. Issue item 4 was explicit that this must not be a log-only signal.
  • /health gains notifications_undelivered, counted separately because those rows still say status='sent' and so are invisible to notifications_failed.

Security note on the signature

The signed payload includes the URL Twilio called, which has to match byte for byte. It is rebuilt from the same status_callback_url() the sender used — deliberately not from Host / X-Forwarded-Proto, which are attacker-influenced. Getting that wrong makes the check decoration.

Signature comparison goes through subtle's constant-time equality.

Unknown SIDs answer 2xx and are ignored: the Twilio account may be shared with another deployment, and a non-2xx would make Twilio retry forever.

Operational cost: none

No new secret, no Twilio console setup. StatusCallback is per-message and the signature uses the existing auth token. The callback is only attached when GHOSTKEY_PUBLIC_BASE_URL is https://, since Twilio fetches it from the public internet — local deployments still send, they just get no verdicts.

hmac and sha1 become direct dependencies. Both were already in Cargo.lock transitively, so this adds no supply-chain surface.

Docs

DEPLOY.md's Twilio section was stale on three counts: a 555 example sender that #317 now rejects, "all four are required together" which sender independence ended, and "notifications stay queued" which #282 ended.

Tests

  • HMAC-SHA1 pinned to RFC 2202 case 1, so a wrong digest can't hide
  • a full signature worked example whose expected value was computed independently in Python, not copied from this implementation — it pins payload assembly, not today's behaviour
  • order-independence and token-dependence of the signature
  • an undelivered claim link is recorded and reaches the activity feed with the provider's reason code
  • delivered recorded without raising an event
  • unknown SID ignored rather than erroring

Stubbing is_negative_verdict to false fails the surfacing tests.

237 passed, 0 failed. clippy -D warnings clean, cargo fmt --check clean.

Not covered here

Resend bounce/complaint webhooks (issue item 3). That needs a webhook secret configured on the deployment and an id-matching strategy verified against a real Resend 250 reply, which I can't confirm without sending through the live account. This PR stores the SMTP reply so that correlation is possible later. Worth a follow-up issue before closing #311.

🤖 Generated with Claude Code

Part of #311. Stacked on #319 (fix/313-permanent-send-errors).

`notifications.status = 'sent'` has only ever meant "the provider's
API returned 2xx" — a handoff receipt, not a delivery receipt. Twilio
answers `201 queued` and then fails a message asynchronously (63007
unknown sender, 63016 outside the WhatsApp session window, 30034
unregistered 10DLC). None of it reached us: the row read `sent`,
`attempts=1`, `last_error` NULL, and /health said the notifier was
healthy while a heir's practice-drill invite sat "sent" for six days
having never existed (mainnet notification id 40).

Adds the inbound half:

  - migration: provider_message_id, delivery_status, delivery_detail,
    delivery_updated_at on notifications, plus their indexes
  - send_twilio attaches a StatusCallback and returns the Message SID;
    send_email returns the SMTP queue reply, the only correlation
    handle an SMTP send offers. Both are recorded at handoff.
  - POST /webhooks/twilio/status, authenticated by Twilio's HMAC-SHA1
    request signature. The signed URL is rebuilt from the same
    status_callback_url() the sender used, NOT from request headers —
    Host and X-Forwarded-Proto are attacker-influenced, and trusting
    them turns the check into decoration.
  - an undelivered/failed verdict writes a notification_undelivered
    event, so it lands in the owner's activity feed instead of a log
    line nobody reads
  - /health gains notifications_undelivered, counted separately
    because those rows still say status='sent'

WorkerOutcome::Sent now carries the provider id, and the log line says
"handed to provider" rather than "sent".

No new secret and no Twilio console setup: StatusCallback is per
message and the signature uses the existing auth token. The callback
is only attached when GHOSTKEY_PUBLIC_BASE_URL is https, since Twilio
fetches it from the public internet.

hmac + sha1 become direct dependencies; both were already in the lock
file transitively, so this adds no supply-chain surface.

DEPLOY.md's Twilio section was stale on three counts (a 555 example
sender that #317 now rejects, "all four are required together" which
sender independence ended, and "stay queued" which #282 ended).

Tests: HMAC-SHA1 pinned to RFC 2202 case 1; a full signature worked
example whose expected value was computed independently in Python,
not copied from this implementation; order-independence and
token-dependence; undelivered claim link recorded AND surfaced as an
event; delivered recorded without raising one; unknown SID ignored
rather than erroring. Stubbing is_negative_verdict to false fails the
surfacing tests.

NOT covered here: Resend bounce/complaint webhooks (issue #311 item
3). That needs a webhook secret configured on the deployment and an
id-matching strategy verified against a real Resend 250 reply, which
the SMTP path can't give us blind. Filed separately.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
ghost-key Ready Ready Preview, Comment Jul 29, 2026 1:02am

@Jolah1
Jolah1 merged commit 41c0a21 into fix/313-permanent-send-errors Jul 29, 2026
2 checks passed
@Jolah1
Jolah1 deleted the fix/311-delivery-status branch August 10, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant