bug(observability): ingest traces end at the queue publish, and nothing tests trace continuity #774

Area: observability · ingest · query — regression · found during a backlog review

Expected: a trace for an ingest request continues past the queue to the worker's flush and the ClickHouse INSERT, and a query or pipe trace includes its ClickHouse call — and a test fails if that stops being true.

Actual:

• The only custom spans on main are IngestHandler.Handle (internal/api/ingest.go:174) and SchemaRegistry.Refresh (internal/discovery/discovery.go:254), plus the otelhttp server span.
• The worker-side and database spans that Add OpenTelemetry spans to primary API and Worker hot paths #18, Bridge Bento Processing into Distributed Traces #20 and Instrument ClickHouse-Go for Database Level Tracing #21 delivered were removed by later refactors and not replaced: clickhouse_delete in refactor(ingest)!: insert-only pipeline; mutations via /v1/query #164 (e07d5ede, 2026-05-20), clickhouse_insert and bento_queue_wait in refactor: full api --> ingest --> clickhouse --> dlq refactor #182 (7140df83, 2026-05-27), and SSE.PushEvent in fix(stream): emit SSE heartbeats so idle streams survive proxy timeouts #346 (725fdee9, 2026-06-26). feat(observability): latency histograms, error-rate counters, saturation gauges, query-path traces #94's body still lists them as existing.
• The trace context does cross the queue: publishers inject W3C headers (internal/mq/embedded.go:676, external.go:605), subscribers extract them (embedded.go:716, external.go:718), and the broker conformance suite checks it (SubscribeCarriesTheTraceContext, internal/mq/mqtest/mqtest.go). But the worker never starts a span from that context; the only use is copying the span context into the post-insert cache invalidation (internal/ingest/worker.go:972).
• ClickHouse calls (native inserts, structured queries, pipes, ops queries) have no client spans, and the query and pipe handlers have none of their own.
• tests/integration/otel_test.go covers sampling, per-signal export, the Prometheus scrape, TLS and headers; no test asserts a parent/child relationship or that an ingest trace continues past the publish.

Impact: with the api and ingest roles in separate processes, the publish→consume hop is exactly where a trace is most useful, and it is where every ingest trace stops. A slow or failed insert can't be tied back to the request that produced it.

Note: one worker batch carries rows from many requests, so a flush span can't have a single parent; span links are the usual shape.

Related: #94 (trace gaps 1 and 6), #18, #20, #21.

Found during a backlog review; validated by code-read and git log -S against origin/main at afd76695 on 2026-10-08.