Repository navigation
bug(ingest): dedupe marks the event id before the NATS publish — a failed publish + client retry permanently drops the event #384
Description
Activity
🔗 Related PRs
#122 - fix(ingest): break infinite-Nak loop on permanent delete errors [merged]
#182 - refactor: full api --> ingest --> clickhouse --> dlq refactor [closed]
#364 - feat(dedupe): observe or reject ingest rows missing the id_field [merged]
🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:issue_enrichment: auto_enrich: enabled: false
💬 Have feedback or questions? Drop into our discord!
- added a commit that references this issue
on Sep 26, 2026
Summary
processRecorddurably records the dedupe id before publishing to NATS, so any publish failure followed by the sanctioned client retry silently loses the event forever.Detail
internal/api/ingest.go::419h.Dedup.CheckAndMark(ctx, eventID)— durably writes the id to Pebble (internal/dedupe/embedded.go:44,pebble.Sync, survives restart).:450h.Publisher.Publish(...)runs after. On "maximum bytes exceeded" it returns503 + Retry-After: 30(the documented backpressure contract, design decision ci: bump actions/checkout from 4 to 6 #4); on marshal/other failure a 500.When the client retries as instructed, the retry hits
dup == trueat:424and is skipped. The event is never in NATS, never in ClickHouse, never in the DLQ. There is no unmark/rollback. Same for a mid-batch client abort after some records were marked.Impact
Silent, permanent data loss triggered by exactly the documented "NATS full → retry" path.
Fix direction
Mark only after a successful publish (mark-after-publish), or make the mark provisional and confirm on publish success / roll back on failure.
Found in a repo-wide audit; independently confirmed by two auditors. Distinct from #220/#221/#222/#370.