Repository navigation
Webhook ingest: duplicate notifications under concurrent requests (KV read-modify-write is not atomic) #38
Description
Activity
Second symptom from review (cubic): lost update across different sites
Same root cause (non-atomic KV read-modify-write of the whole summary), distinct symptom:
Two concurrent webhooks for different sites A and B both read the pre-update summary, then each rewrites the whole summary — A writes {A=new, B=old(fallback)}, B writes {A=old(fallback), B=new}. Last-writer-wins, so one site's real-time update is lost until the next cron tick (or next webhook) reconciles from D1.
Unlike the same-site duplicate-notification case, here the intermediate KV state is briefly inconsistent (not just a duplicate alert). Still self-corrected by cron, so low-harm, but it strengthens the case for the fix.
Fix options (superset)
- Durable Object per page/site to serialize read-modify-write + change detection (covers both symptoms cleanly).
- Per-site summary KV entries instead of one whole-summary blob, so concurrent writes to different sites don't clobber (the page read path would merge per-site keys).
- Short-lived KV lock key (TTL) or last-notified-status conditional write to dedupe notifications (covers the duplicate-alert symptom only).
Raised independently by both Greptile and cubic during #33 review.
Follow-up from review of #33.
Problem
The webhook ingest pipeline (
apps/worker/src/ingest.ts) reads the previous summary from KV, detects a status change, and writes the new summary — a read-modify-write that is not atomic. The cron path is serialized by the scheduler so it's unaffected, but the new webhook path can receive multiple in-flight requests simultaneously.If two rapid-fire webhooks land for the same site (or Statuspage retries quickly), both requests can read the same prior status, both pass the
prior.status !== r.statusgate, and both calldispatchNotifications→ subscribers receive duplicate alerts.Note the final KV state stays consistent (both writes carry the same new status); only notification delivery is affected — i.e. current semantics are at-least-once, and cron reconciles the summary regardless.
Options
dispatchNotificationswith an idempotency key (dedupe byslug + from→to + coarse timestamp). This also aligns with the existing note innotify.ts/wrangler.jsoncabout putting a Queue between producer and dispatch for retries/backpressure.Scope
Deferred from #33 (status-only webhook ingest). Low harm today (duplicate alert, not data corruption), so tracking rather than blocking.