fix(ingest): retry ClickHouse outages instead of dead-lettering

[EricAndrechek]

Part of #613 (workstream A).

What changes

A failed batch insert used to go through row-by-row isolation whatever the failure, so a ClickHouse that was down, overloaded or read-only failed every row a second time and parked the whole batch on the DLQ. Now the worker asks what the failure was first:

• ClickHouse rejected the row (chconn.Rejected): unchanged. Row-by-row isolation, the rows rejected again go to the DLQ, or stay unacked with dlq.enabled off.
• ClickHouse refused a multi-row batch for its size (TOO_MANY_PARTS, MEMORY_LIMIT_EXCEEDED: chconn.Splittable): the same row-by-row isolation first. TOO_MANY_PARTS is also what ClickHouse answers for one INSERT spanning more than max_partitions_per_insert_block partitions: measured on 26.6.3.62, 150 rows over 150 day-partitions fail with 252, and each row inserts on its own. Retried whole, such a batch re-forms the same way on redelivery and never lands. If a row fails any way but Rejected (typically the first row failing the same way, because the server is the problem after all), isolation stops there and that row and the rest are retried under the backoff below; in the typical case that costs one extra request. ClickHouse's own Distributed async inserts split a batch on the same codes (isSplittableErrorCode).
• ClickHouse could not take it (Unavailable, Denied, Unknown): no isolation, no DLQ. The batch goes back to the queue with a delayed nak (mq.Message.NakWithDelay, new), under a backoff per ClickHouse pool (URL + user + database). The backoff starts at 1 s, doubles to a 30 s cap, and each delay is jittered down to as little as half. While the window runs, every table on that pool is turned away with no request, and a row that arrives is handed straight back rather than buffered. When the window ends, one flush probes. Any answer that is not an outage closes the backoff, and that includes a rejected row. While a probe is out, arriving rows are handed back with a delay of at least 0.5 s, so a slow probe does not cycle the backlog. A failure of one table (TABLE_IS_READ_ONLY, TABLE_IS_PERMANENTLY_READ_ONLY, TOO_MANY_PARTS, TOO_MANY_MUTATIONS, ACCESS_DENIED: chconn.TableScoped) backs off that table alone: its neighbours keep inserting until its waiting rows fill the tenant's maxAckPending (see below), and their successes don't close its backoff. The retries are counted by wavehouse_ingest_retries_total{table, reason}. An outage is logged at WARN when it starts, at most every 30 s while it lasts, and at INFO when ClickHouse takes inserts again.
• ClickHouse goes away mid-isolation: isolation stops at that row. The rows already inserted stay acked and the rows already rejected stay parked. The row that hit the outage, and every row after it, go back to the queue.

The classifier lives in internal/chconn/errclass.go: Classify(err) Class, ClassOfCode, ExceptionCode, and HTTPError/NewHTTPError for the HTTP interface. It reads *clickhouse.Exception and *clickhouse.HTTPError from the driver, the HTTP interface's X-ClickHouse-Exception-Code / Code: NNN. body, the clickhouse-go sentinels (ErrAcquireConnTimeout, ErrConnectionClosed, driver.ErrBadConn) and the network errors under them. So the query path can reuse it.

How the ambiguous cases are classed, and why

The line is drawn at the exception code.

• Unlisted exception code → Rejected (DLQ). A code means ClickHouse was up and read the request. Nearly all of its several hundred codes are verdicts on what it read, so the availability codes are the enumerated exception. The cost of each wrong guess settles it. If an availability code is missing from the list, its rows get parked: nothing is lost, and they wait on the DLQ (reading them back is feat(api): admin DLQ replay endpoint (re-validate + re-publish dead-lettered events) #237). If a genuine bad row were retried forever, it would pin the ingest stream's ack floor and a share of maxAckPending for every table behind it, and nothing would clear it. A code added by a future ClickHouse falls into the same bucket.
• No code, and not a recognisable transport failure → Unknown (retried). Examples: a bare 500 from something that is not ClickHouse, or a TLS setup error. Nothing says a row was judged, so isolating the batch would only multiply the requests, and dead-lettering it would park good rows.
• Schema drift: UNKNOWN_TABLE, NO_SUCH_COLUMN_IN_TABLE, UNKNOWN_DATABASE → Rejected. The row cannot insert into the table as it now is, and the DLQ keeps it rather than retrying it forever. TestDLQ_PopulatedOnIngestWorkerFailure still pins this.
• Auth: AUTHENTICATION_FAILED, ACCESS_DENIED, UNKNOWN_USER, WRONG_PASSWORD, REQUIRED_PASSWORD, IP_ADDRESS_NOT_ALLOWED, DATABASE_ACCESS_DENIED, USER_EXPIRED → Denied (retried). These are the connection's configuration, not the rows. A grant, a corrected clickhouse.username, or WH_CH_PASSWORD and a restart makes the whole batch insert as it is. ACCESS_DENIED is usually a grant missing on one table, so it backs off that table alone (see below). Parking would move every row of every table on that pool to the DLQ.
• Codes classed Unavailable (names checked with errorCodeToName on 26.8, and against system.errors on 26.6.3.62, the pinned test image): 95, 96, 159, 160, 164, 173, 201, 202, 203, 209, 210, 225, 236, 241, 242, 243, 244, 252, 254, 265, 279, 285, 286, 289, 297, 319, 364, 369, 394, 415, 425, 439, 499, 574, 692, 700, 735, 745, 749, 762, 774, 776, 777, 778, 904, 999, 1000. TABLE_IS_PERMANENTLY_READ_ONLY (774) is here even though it does not clear by itself: its rows are not at fault, and an operator fixes it the way one fixes a grant. UNKNOWN_STATUS_OF_INSERT (319) is retried: a duplicate is the lesser harm. ClickHouse's insert deduplication will not catch it, because the retried rows come back regrouped with newer ones; the docs now say delivery after an ambiguous failure is at-least-once. SERVER_OVERLOADED (745) was missing from the first cut: the CPU-overload check at the top of every query throws it, and so does the workload scheduler's max_waiting_queries.
• HTTP status with no code (a proxy answering for ClickHouse): 408/429/502/503/504 → Unavailable. 401/403/407 → Denied. 413 → Rejected, because isolation shrinks the body.

Unchanged on purpose: a batch whose tenant has no ClickHouse connection (parkBatch) still meets its DLQ switch whole. That rule exists so that a tenant no longer served does not pin the shared ingest queue.

What an operator sees

Rows waiting out an outage stay unacked in the ingest stream. They count toward maxAckPending, and the sweeper cannot purge past them. A long outage therefore fills the stream to mq.max_bytes_gb, and ingest then answers 503: backpressure, with nothing lost and nothing parked. The DLQ no longer fills up. NakWithDelay keeps the message on the consumer's pending list (verified in nats-server processNak), so the queue holds the backlog, not the worker.

A failure of one table counts toward the same budget. One that lasts (a missing grant, a permanently read-only table) suspends delivery for all of that tenant's tables once its waiting rows reach maxAckPending, and the tenant's ingest backs up to 503 as in an outage. A retry also does not keep arrival order; that matters only to a ReplacingMergeTree without a version column or a CollapsingMergeTree, and the docs say so.

Tests

• internal/chconn/errclass_test.go: a table of classifier cases covering a real refused dial and a real client timeout through net/http, *clickhouse.Exception, *clickhouse.HTTPError, the driver sentinels, header- and body-coded HTTP answers, and codeless proxy statuses. Also pins that the two code lists are disjoint, that every splittable code is a retried one, and NewHTTPError parsing.
• internal/ingest/backoff_test.go: escalation to the cap, jitter bounds, no escalation from late reports, one probe at a time, rate-limited outage logging, and one backoff per pool.
• internal/ingest/worker_test.go:
  • (a) an availability failure, in eleven shapes, never dead-letters and naks with a delay: one request, or two for a splittable code (the split's first row fails the same way);
  • a batch refused for its size (TOO_MANY_PARTS as too many partitions, MEMORY_LIMIT_EXCEEDED) is split, every row inserts, and no backoff opens; a lone row is not split again;
  • (b) a CANNOT_PARSE_NUMBER row is still isolated and parked;
  • (c) ClickHouse going down mid-isolation stops isolation and retries the unsettled rows;
  • an outage stops later column groups;
  • tables on a down pool back off together, and one probe closes the backoff;
  • another pool is unaffected;
  • the batcher buffers nothing while its pool backs off, or while a probe is out;
  • a read-only table backs off alone, and a healthy neighbour's success does not close its backoff.
• tests/integration/ingest_outage_test.go: stops a real ClickHouse container under a running worker and publishes three rows. It checks that none is parked for 12 s, then restarts ClickHouse and waits until all three land, still with none parked. Run against the old worker.go, it fails on both assertions.

Review

The pre-push-reviewer and docs-reviewer subagents (opus) ran in fresh context against this worktree, four rounds:

• Round 1 (a131fb95): both iterate. Code review: table-scoped codes flapped the shared pool breaker, and the delay while a probe was out was near zero. Docs review: an overclaim that the query handlers already use Classify, NakWithDelay missing from the mq surface list, an incomplete Denied list. Fixed in 8ae9614b.

• Round 2 (8ae9614b): both iterate. Code review: ACCESS_DENIED is usually one table's grant, so it should be table-scoped; a stale bound in the backoff map comment. Docs review: the docs pointed operators at a ClickHouse password in config.json (it is WH_CH_PASSWORD, boot config), and overstated the pool key. Fixed in 356125f3.

• Round 3 (356125f3): code ship_it; docs iterate, because the README, landing page, why page and architecture diagram still said failed inserts go to the DLQ. Fixed in e97edc8b.

• Round 4 (e97edc8b): both ship_it.

• Round 5 (c70d3324..d1ef1302): the size split for TOO_MANY_PARTS/MEMORY_LIMIT_EXCEEDED, SERVER_OVERLOADED, and doc corrections. Code review of c70d3324: ship_it, no findings. Docs review took nine passes to reach ship_it on d1ef1302. Along the way it fixed:

  • a drain runbook that described this build's retry signals for a drain run on the earlier build (which dead-letters an outage, so ClickHouse must stay healthy for the whole drain);
  • the split exception missing from two summaries;
  • a lock-free claim the shared backoffs broke;
  • duplicates after an ambiguous insert failure, which neither ClickHouse's insert deduplication nor dedupe.enabled catches;
  • version-column and FINAL advice where a user chooses an engine;
  • a set of missed copies of "a failed insert goes to the DLQ".

  Markers for round 5: the SubagentStop hook wrote none. The reports came back through the subagent hand-back, so the payload it reads evidently carried no VERDICT: line; fed the report text, the hook writes a marker (tested in a scratch repo). Both verdicts are recorded with scripts/skip-pre-push-review.sh, and each reason names the run. docs-reviewer ran on HEAD. pre-push-reviewer ran on c70d3324; everything after it is docs prose and two doc-comment rewordings.

Review markers (#454): the review-marker.sh SubagentStop hook writes to $CLAUDE_PROJECT_DIR/tmp, keyed to that checkout's HEAD (the main checkout, on main), so no tmp/<reviewer>-passed-e97edc8b… marker exists for this branch. The verdicts above are the record; no marker was hand-written or skipped.

Follow-ups (not in this PR)

• query API maps ClickHouse caller-side errors (e.g. ACCESS_DENIED) to HTTP 502 instead of 4xx #403 / bug(api): request-caused query errors return HTTP 500 + retryable:true (should be 4xx, non-retryable) #271: map /v1/query and /v1/ops/query failures through chconn.Classify and ExceptionCode. Rejected → 4xx (ACCESS_DENIED → 403), Unavailable → 503/502 with retryable: true. The classifier is exported for that.
• epic(distributed): shared backends and standalone workers for multi-node deployments #613 C (distributed workers): see the seams below.

Seams left for workstream C

• The backoff state is in-process: IngestWorker.backoffs, keyed by chconn.Target (URL, user, database), plus the table for table-scoped failures. With several worker processes, each one backs off on its own. That is harmless, because each probes at most once per window, but a shared backoff could live behind the coordination primitive (B).
• flushTable hands back exactly the unsettled rows through retryLater. Any batching/claiming scheme that owns acks can keep that contract.
• tableBatcher.add asks the backoff before buffering. A per-tenant consumer (feat(mq): give every tenant a queue of its own #612) could pause that tenant's Consume here instead of nak'ing arrivals.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EJr5tY4WQUy2sc4MbW67vL
https://claude.ai/code/session_01LYbFK7Na4RTNZxzL3h9Hs9

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 2b41763a-c287-464c-8a84-c7d35ca75994

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

📚 Docs preview is live → https://65a8004d-wavehouse-docs.wave-rf.workers.dev

• Commit — d1ef130: docs(ingest): split overview names what goes back; flow in order
• Author — @EricAndrechek, Claude Opus 5.5 (1M context)
• Committed — 2026-09-26 01:27 (UTC-04:00)
• Deployed — 2026-09-26 01:42 EDT

[github-code-quality]

Code Coverage Overview

Languages: Go

Go

The overall line coverage in commit d1ef130 in the feat/ch-error-classe... branch remains at 94%, unchanged from commit 77cf4a7 in the main branch.

Show a line coverage summary of the most impacted files.

File	main 77cf4a7	feat/ch-error-classe... d1ef130	+/-
internal/mq/mq.go	100%	98%	-2%
internal/ingest/worker.go	98%	97%	-1%
internal/chconn/errclass.go	0%	98%	+98%
internal/ingest/backoff.go	0%	98%	+98%

---

_{Updated September 26, 2026 05:42 UTC}