Repository navigation
bug(sdk): liveQuery dedup boundary is wrong — duplicate and dropped events in the core streaming feature #396
Description
Activity
🔗 Related PRs
#174 - refactor(table names): handle unsafe table names [closed]
#182 - refactor: full api --> ingest --> clickhouse --> dlq refactor [closed]
#313 - feat(docs): view-transition polish, branded search + 404, chrome pass [closed]
#364 - feat(dedupe): observe or reject ingest rows missing the id_field [merged]
🧪 Issue enrichment is currently in open beta.
You can configure auto-planning by selecting labels in the issue_enrichment configuration.
To disable automatic issue enrichment, add the following to your
.coderabbit.yaml:issue_enrichment: auto_enrich: enabled: false
💬 Have feedback or questions? Drop into our discord!
Consolidating #449 into this issue — same bug, same lines (
clients/ts/src/stream/live-query.ts:63-81). #449 was filed independently from the PR #434 pre-push docs review; CodeRabbit flagged the overlap on 2026-08-11. This issue is the survivor because it is the superset: #449's two failure modes (positional-last boundary, lexical string compare) are both here, plus the missing-column case, the sub-second-precision silent drop, and the fetch-error buffer drop.Porting the three things #449 had that this issue does not:
1. There is a correct reference implementation — but it only fixes two of the three modes.
clients/go/live_query.go:89-100on thego-sdkbranch (PR #434 — not onmain, so this is a reference to read, not code to point users at yet) takes the max over parsedtime.RFC3339Nanovalues across all rows, independent of ordering:var lastTS time.Time for _, row := range rows { if s, ok := row["received_timestamp"].(string); ok { if ts, perr := time.Parse(time.RFC3339Nano, s); perr == nil && ts.After(lastTS) { lastTS = ts } } }
Worth being precise about the limit: that loop fixes the wrong-row and lexical-compare modes, but on a table with no
received_timestampcolumn the type assertion never succeeds,lastTSstays zero, and the Go client silently skips dedup exactly like the TS one. Porting the Go approach to TS closes two of three modes and leaves the worst one open. Whatever lands should settle the missing-column question explicitly (require it and fail loudly / let the caller nominate the ordering column / scope the feature to ingest-API tables) rather than inheriting the silence.2. Three live references point at #449 and will need updating with the fix.
docs/src/content/docs/sdk/streaming.md:115— "only under an ascending order (#449)"docs/src/content/docs/sdk/streaming.md:188— the "How it works" step 3 paragraph, which deliberately documents the current, incorrect behavior and ends "Tracked in [sdk(ts): live-query dedup uses the last fetched row, not the newest — duplicate delivery on desc-ordered fetches #449]"clients/ts/src/stream/live-query.ts:67— a source comment citing sdk(ts): live-query dedup uses the last fetched row, not the newest — duplicate delivery on desc-ordered fetches #449 and pointing at those docs as the single place the three modes are enumerated
Both docs lines are published on the public docs site. The prose at
:188was written to describe the bug and should revert to the plain "newest historical timestamp" description once this is fixed — that instruction lives only in #449 and is preserved here so it isn't lost behind a closed issue. The references themselves still resolve (a closed issue keeps its URL), but they should be repointed here or removed as part of the fix PR.3. A consumer already routed around the feature.
From Eric on #449: a first-party browser app building a live-events view chose
.stream()over.liveQuery()specifically because.liveQuery()couldn't be relied on for arbitrary user-defined tables. The feature was avoided rather than worked around — which is the strongest signal in either issue about what this costs, and does not otherwise appear on this one.Related: #449 (closed as duplicate of this), #434 (Go SDK, carries the reference implementation), #372, #98.
Consolidated by the pm-triage routine on 2026-08-25; verified by code-read against
b2eee9dandorigin/go-sdk.Priority raised P2 → P1, and #449 consolidated in (see above).
Rationale:
liveQuery()is a headline documented feature, this is live in the shipped public v0.1.0, and the consolidation added a datapoint that wasn't on either issue alone — a first-party consumer avoided the feature entirely rather than working around it. The docs currently describe the broken behaviour as the behaviour.This is under a re-anchored rubric (the old P0/P1 definitions were pinned to the launch, which shipped 08-19); the new anchor and all five changes are recorded in #501. Flagging rather than assuming — say so if you'd rate it differently.
pm-triage routine, 2026-08-25, on Eric's instruction.
Summary
LiveQuerydedupes buffered live events against the historical snapshot usingrows[rows.length - 1].received_timestampas the boundary. That boundary is wrong in the documented usage, and the column it relies on isn't even in the table.Detail
clients/ts/src/stream/live-query.ts:63-81:received_timestamp DESC, so the last row is the oldest → boundary is minimal → every buffered event passes and rows already in the snapshot are re-delivered vianext().received_timestampis never inserted into the table (internal/ingest/worker.goinserts only the inner payload), and.select(...)projections drop it anyway →lastTimestampisundefined→ no dedup at all, despite the "deduplicated automatically" docstring (query-builder.ts:180-188)....:05Z; lexicographically"...:05Z" > "...:05.123Z", so buffered events in the same second as the boundary are wrongly discarded — silent loss.:58-61) buffered live events are dropped, never delivered.Fix direction
Dedup on a stable server-provided key/sequence rather than the envelope timestamp string; use
Math.maxover the fetched rows' order column (not positional last), and compare with parsed instants, not lexically.Found in a repo-wide audit; verified by code + docs. Distinct from #372 (data-column zone drift) and #98 (replay-vs-live labeling).