Skip to content

SSE/live-query yields no events for tables populated by a ClickHouse MATERIALIZED VIEW (only ingest-API writes republish to NATS) #362

Description

@EricAndrechek

Summary

GET /v1/stream?table=T only emits events for rows written via POST /v1/ingest (which publishes to NATS ingest.<table>; the SSE hub subscribes ingest.>). If T is the target table of a ClickHouse MATERIALIZED VIEW — rows land in T via the MV, not the ingest API — no NATS event is published and the stream is silently empty (no error).

Why it surprised us

A natural WaveHouse pattern is: ingest raw rows into a base table, expose a derived/projection table via an MV (e.g. a column-masked positions_public), and stream the projection to public clients. That projection never streams — the base table does, but not the MV target.

(Distinct from the already-tracked row-filter drift in applyStreamPolicy, which is about which columns/rows an event carries. This is about MV-written rows producing no event at all.)

Repro

  1. CREATE TABLE base (...), CREATE TABLE derived (...), CREATE MATERIALIZED VIEW mv TO derived AS SELECT ... FROM base.
  2. GET /v1/stream?table=derived (subscribe).
  3. POST /v1/ingest?table=base a row that the MV projects into derived.
  4. → the row appears in derived via /v1/query, but the derived stream receives nothing.

Impact

Live dashboards built on derived/projection tables silently receive zero updates — easy to misdiagnose as a client/auth problem. We only found it by testing each surface.

Workaround

Stream the base (ingest-API-written) table directly and rely on the JWT column policy to mask columns per role.

Suggested

  • At minimum, document prominently that SSE/live-query only covers ingest-API-written tables (not MV/derived targets).
  • Optionally, a mechanism to surface MV/derived-table changes to the SSE hub (opt-in re-publish of an MV target to NATS, or CDC).

Found building the SCC 2026 tracker pilot against a self-hosted WaveHouse.

Activity

  1. added
    area/apiHTTP handlers, routing, middleware
    area/ingestIngest pipeline (Bento, batching, DLQ)
    on Jul 1, 2026
  2. coderabbitai commented on Jul 1, 2026

    @coderabbitai
    🔗 Related PRs

    #124 - chore(api)!: drop hub wildcard fan-out; SSE/WS use ?table= (closes #100) [merged]
    #174 - refactor(table names): handle unsafe table names [closed]
    #182 - refactor: full api --> ingest --> clickhouse --> dlq refactor [closed]
    #313 - feat(docs): view-transition polish, branded search + 404, chrome pass [closed]


    🧪 Issue enrichment is currently in open beta.

    You can configure auto-planning by selecting labels in the issue_enrichment configuration.

    To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

    issue_enrichment:
      auto_enrich:
        enabled: false

    💬 Have feedback or questions? Drop into our discord!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/apiHTTP handlers, routing, middlewarearea/ingestIngest pipeline (Bento, batching, DLQ)area/streamingSSE / live-query delivery path (/v1/stream)enhancementNew feature or request

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions