Repository navigation
feat!: policy v2, positional wire shape, and declared ingest format - #540
EricAndrechek wants to merge 10 commits into
Conversation
Additive metadata the upcoming native type layer needs, captured on the same refresh as the columns so they can never describe different servers: - Column gains DefaultExpression and Position, both scanned from the widened system.columns select. TableSchema.Columns was already ordered by position, so declaration order needed no new structure. - TableSchema gains DDL from system.tables.create_table_query. It is json:"-": the schema endpoint marshals TableSchema straight to the client and an external-engine table (S3, MySQL, Kafka) carries its credentials in that statement. A table listed in system.tables with no system.columns rows is skipped, never published column-less. - SchemaRegistry gains ServerVersion(), from a SELECT version() probe next to the existing SELECT timezone(). Both new queries fail the refresh on error, matching timezone() and system.columns: callers keep the prior cache and retry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
BREAKING CHANGE: policies.json moves from tables.<table>.select.<role> to tables.<table>.<role>.select. Field names and semantics are unchanged; only the nesting moves. There is no automatic conversion — convert the file by hand and run `wavehouse validate` before restarting. A role now appears once per table, with two optional blocks stating what it may do. The blocks are separate types rather than one struct whose halves were inert per operation: SelectPermissions carries the column lists, the row filter, the aggregation rules and the four limits; InsertPermissions carries the column lists and the check clauses. A field on the wrong side is a validation error now instead of being accepted and ignored. ResolvedPermissions splits the same way (.Select / .Insert), and IsColumnAllowed takes the side to consult — which is what keeps the read allowlist from ever answering a write question. Validate still checks BOTH sides for every role; the insert-side rejection of _neq/_gt/_lt on check clauses is unchanged. resolvedPredicate/resolvePredicates are exported, and the resolved insert side carries CheckPredicates alongside CheckClauses for a consumer that does not exist yet. settings validation gains a layout detector: a pre-v2 document is reported as one finding naming the table and operation with a pointer to the docs migration note, rather than a strict-decode "unknown field" error — or, for an empty operation block, silently decoding as a role named "select". A grant that sets neither operation now warns. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
BREAKING CHANGE: POST /v1/ingest rejects a request that declares no Content-Type, or one it does not read, with 415 — listing the accepted types. Clients that relied on body sniffing must send a header. The header used to be a hint: the first non-whitespace byte chose the format, and an application/x-ndjson body starting with '[' was silently re-read as a JSON array. Now the declaration decides. An NDJSON body is read as NDJSON whatever its first byte, so a line that is not a JSON object fails as a per-record error through the existing recordReject path instead of re-framing the whole request. The body still picks arity within the JSON family — '[' is an array, anything else a single object — since those are one format at different lengths. The choice is modeled as an IngestFormat where the sniffing lived, keeping the CSV slot where the old comment marked it. The TS SDK already sent a header on both paths; it now states it at each call site rather than leaning on the request default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
…encoder Groundwork only — behavior is identical after this commit. Buffered body: Handle reads the whole (MaxBytesReader-capped) body into a pooled *bytes.Buffer and runs the record readers over those bytes instead of the live connection. The 413 now surfaces at that read rather than mid-iteration; same status, same message. newRecordReader takes an already-resolved format and the body bytes, so the 415 is decided from the header before any byte is read. The sniff window is kept as-is so a body of leading whitespace longer than the window still reads as empty. UseNumber stays on every decoder. Seams, each an interface with a default implementation delegating to today's code unchanged: - RecordValidator (schema validation + timestamp canonicalization). The two call sites stay where they are, with the check-clause block between them — merging them would move checks onto canonicalized values. CanonicalizeTimestamps returns nothing, matching discovery's function: it is deliberately fail-open, and an error return would invite changing that. - InsertChecker (the _eq and _in comparisons), wrapping checkValueMatches and valueInSet verbatim. Auto-injection stays in processRecord. - stream.RowEvaluator (row visibility), reached by both the live fan-out and replay through the existing rowAdmitted step. All three are nil-safe: an un-wired handler or Hub uses the default rather than panicking past the check. New internal/ingest/compact.go: EncodeCompactRow renders a record as one JSONCompactEachRow line — one value per schema column in declaration order, a missing column as null, json.Number digits preserved. Serialization only; nothing calls it yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
BREAKING CHANGE: the NATS EventMessage envelope replaces `data` with
`format`/`columns`/`row`, and SSE data frames carry `row` (a positional
array) preceded by an `event: schema` frame naming the columns. In-flight
NATS messages from an older version are NOT readable by the new worker —
it acks and drops an envelope whose format is absent or unknown — so drain
the ingest queue before deploying.
A batch of rows for one table now carries the column names once instead of
repeating every key on every row, and a reader can tell a schema change
mid-stream from a reordering.
Worker: groups a batch by column list, so a schema change splits the
INSERT rather than corrupting it, and writes
`INSERT INTO {table} (cols) FORMAT JSONCompactEachRow` with the identifiers
quoted by the same helper the query builder uses. It adds
input_format_null_as_default=1, which turns a column the record omitted —
now an explicit null in its slot — back into that column's default.
TRANSITIONAL DIVERGENCE: a Nullable column that also has a DEFAULT
expression now takes its default where it previously stored NULL.
SSE: each connection is told its projected column list before its first
row — at subscribe time from the registry, and again on drift — and never
sees a data frame it cannot zip. The schema frame carries no `id:` line: an
empty one would clear the client's Last-Event-ID. Replay follows the same
contract with its own drift state, because it writes straight to the socket
while live events queue behind it.
Row cells are copied as their original bytes at every hop, so a 64-bit id
past 2^53 keeps every digit. An envelope whose columns and row cannot be
paired is dropped by the worker and by the fan-out — there is no way to say
which value belongs to which column, so no role gets it.
The TS SDK consumes the schema event and zips rows back into objects, so
.stream() and .liveQuery() still yield row objects: the public API is
unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
- SDK streaming guide: StreamEvent.data is still a row object, but say what moved underneath — the positional wire, the omitted-column null, and that a raw EventSource consumer must zip rows itself. - CHANGELOG: split the wire change into its two breaking halves (NATS envelope v2 with the drain-before-deploy warning, and the SSE schema-event contract), and add the two entries the earlier commits left out — the widened schema discovery, and the buffered ingest body plus the type-layer seams. The per-page doc updates for each change landed with the commit that made it; this is the sweep for what those missed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
Two defects the e2e run found. deliver() sent the schema frame and the data frame as separate non-blocking Sends, so a queue that was full for the first and drained by the handler before the second would deliver a row while its announcement was dropped — the client then zips that row against a stale column list, which is silent mislabeling where a dropped row is a visible gap. The announcement is now recorded only once it is actually queued, and a dropped announcement drops its row with it, so the next event announces again and a transient full queue stays recoverable. The e2e settings helper's referencedRoles() still walked the pre-v2 nesting (table.select as a role map), so the derived roles.json declared nothing and every reload was rejected. Under the role-first layout a table's own keys are its roles. tsc could not catch this: indexing a Record<string, RolePermissions> with the literal "select" type-checks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
The case sends a malformed body to assert a 400. It set no Content-Type, which `fetch` fills in as text/plain — now a 415 refused before the body is read, so the test was asserting the old contract. Declare application/json so it still exercises what it means to. Adds the companion case the change deserves end to end: an undeclared and an unreadable Content-Type are both 415, and the message names every type ingest accepts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
The published row carries only schema columns, by position, so a value
auto-injected for a checked column that does not exist in the table was
dropped on the way out: the record inserted WITHOUT the value the policy
required, and answered 200 {"ok":true}. The misconfiguration was invisible.
Under the previous wire shape the same policy surfaced loudly — the
unknown column reached ClickHouse and the row landed in the DLQ.
The check clause is now refused at the top of the loop, naming the column
and the table. The supplied-value case never reaches it (schema validation
rejects an unknown column one step earlier, with its own message), so in
practice this fires on the omitted-and-auto-injected path; the guard sits
at the top anyway so it does not depend on that ordering holding.
Status is 403, consistent with the check-clause rejections beside it,
though the fault is server-side configuration rather than anything the
caller can fix — the message is what carries the diagnosis.
Adds TableSchema.HasColumn for the lookup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
No behavior change. The replay and live paths track schema drift separately, and the earlier note claimed the only cost was a redundant announcement. That is not the whole truth: the two states are never reconciled, so a client that gap-fill-replays across a column-set change can end up with a list older than the one the live path already recorded at subscribe time, and live rows after the replay carry no fresh schema frame until the next drift or a reconnect. Rows are not mislabeled — a length mismatch is dropped, not guessed at — so the cost is availability. Recorded as a known limitation in the code, the CHANGELOG, the SSE reference and the SDK streaming guide, deferred to the schema-versioning work. The subscriber's schema comment implied concurrent delivery was anticipated. deliver()'s check→send→record is not atomic and is only safe because Broadcast runs on one goroutine — the single inline jetstream Consume callback the hub bridge registers. Documented, with what a future concurrency change would have to add. The comment also still claimed replay records against that field, which it no longer does. Stale comments: Frame.Kind omitted schema; groupByColumns overclaimed cross-group arrival order (groups follow first appearance); and three ingest tests still described the pre-rewrite sniffer and the streaming body-cap path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe PR introduces role-first policy v2, declared ingest media types, compact positional event envelopes, schema-aware SSE frames, pluggable validation seams, and expanded schema discovery metadata. SDK types, deployment settings, documentation, and tests are updated accordingly. ChangesSchema and policy contracts
Ingest and event envelopes
Schema-aware streaming
Documentation and changelog
Estimated code review effort: 5 (Critical) | ~120 minutes Merge Risk: 🟠 High · up to This PR introduces an incompatible ingest queue format and positional streaming changes. An incorrect deployment order or rollback can acknowledge queued records without inserting or preserving them, causing permanent data loss; merge should wait for a safer compatibility path or an enforced versioned rollout plan. Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 71.97% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 239 functions across 50 files. (12 skipped: 10 unsupported, 2 over the file limit.)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Code Coverage OverviewLanguages: Go GoThe overall line coverage in commit 39bc0f9 in the Show a line coverage summary of the most impacted files.
|
|
📚 Docs preview is live → https://e9d32fbc-wavehouse-docs.wave-rf.workers.dev
|
There was a problem hiding this comment.
Actionable comments posted: 8
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
clients/ts/src/table.ts (1)
143-144: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winCorrect the server parsing statement.
IngestHandler.Handlebuffers the capped request body before it creates the record reader. The server does not stream parsing. State that the server buffers the capped body before parsing it.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Team
Run ID: 8b811267-08c7-4c05-8873-8bf8da811ef4
📒 Files selected for processing (62)
CHANGELOG.mdclients/ts/src/index.tsclients/ts/src/stream/sse.test.tsclients/ts/src/stream/sse.tsclients/ts/src/table.tsclients/ts/src/types.tsdeployments/compose/settings/policies.jsondocs/src/content/docs/access-control.mdxdocs/src/content/docs/api.mddocs/src/content/docs/architecture.mddocs/src/content/docs/ingest-pipeline.mddocs/src/content/docs/sdk/admin.mddocs/src/content/docs/sdk/queries.mddocs/src/content/docs/sdk/streaming.mddocs/src/content/docs/settings-directory.mdxinternal/api/boot_chain_test.gointernal/api/bufpool.gointernal/api/bufpool_test.gointernal/api/errors_test.gointernal/api/ingest.gointernal/api/ingest_seams.gointernal/api/ingest_seams_test.gointernal/api/ingest_test.gointernal/api/policy_test.gointernal/api/record_reader.gointernal/api/stream.gointernal/api/structured_query.gointernal/api/structured_query_test.gointernal/auth/auth_test.gointernal/discovery/discovery.gointernal/discovery/discovery_test.gointernal/discovery/timestamp_test.gointernal/ingest/compact.gointernal/ingest/compact_test.gointernal/ingest/types.gointernal/ingest/types_test.gointernal/ingest/worker.gointernal/ingest/worker_test.gointernal/policy/policy.gointernal/policy/policy_test.gointernal/policy/rowfilter.gointernal/policy/rowfilter_test.gointernal/query/builder.gointernal/query/builder_test.gointernal/settings/validate.gointernal/settings/validate_test.gointernal/stream/filter_test.gointernal/stream/hub.gointernal/stream/hub_test.gointernal/stream/metrics.gointernal/stream/roweval_test.gointernal/stream/subscriber.gointernal/testutil/testutil.gotests/e2e/sdk/admin.test.tstests/e2e/sdk/ingest.test.tstests/e2e/sdk/query.test.tstests/e2e/sdk/settings.tstests/e2e/sdk/setup.tstests/e2e/sdk/streaming.test.tstests/integration/dlq_test.gotests/integration/query_limits_test.gotests/integration/rowfilter_narrowing_test.go
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.
📜 Review details
⏰ Context from checks skipped due to timeout. (3)
- GitHub Check: Docs preview
- GitHub Check: Coverage
- GitHub Check: E2E tests
🧰 Additional context used
📓 Path-based instructions (7)
**Opt a page into the Cloud CTA with `cloudCta` frontmatter**, not by importing the component.
📄 CodeRabbit inference engine (AGENTS.md)
Files:
docs/src/content/docs/access-control.mdxdocs/src/content/docs/settings-directory.mdx
A new test file must (1) add its suite name to `SUITES` in `tables.ts` and (2) get its names via `const T = suiteTables("")`, then reference `T.clicks` etc. — never a bare `clicks`.
📄 CodeRabbit inference engine (AGENTS.md)
Files:
tests/e2e/sdk/admin.test.tstests/e2e/sdk/query.test.tstests/e2e/sdk/ingest.test.tstests/e2e/sdk/streaming.test.ts
Every code change updates its docs + `CHANGELOG.md` in the same PR
📄 CodeRabbit inference engine (AGENTS.md)
Files:
clients/ts/src/table.tstests/e2e/sdk/admin.test.tsinternal/stream/metrics.gointernal/discovery/timestamp_test.gotests/e2e/sdk/settings.tsinternal/auth/auth_test.godocs/src/content/docs/sdk/streaming.mddocs/src/content/docs/sdk/admin.mdinternal/api/structured_query.gointernal/stream/filter_test.gotests/integration/rowfilter_narrowing_test.gotests/e2e/sdk/setup.tsinternal/api/ingest_seams_test.goclients/ts/src/index.tsdocs/src/content/docs/sdk/queries.mdinternal/api/errors_test.gointernal/ingest/types.gotests/e2e/sdk/query.test.tstests/integration/query_limits_test.gointernal/ingest/compact_test.gointernal/api/stream.gointernal/ingest/compact.gointernal/settings/validate.gointernal/settings/validate_test.gointernal/api/record_reader.gointernal/api/bufpool_test.goclients/ts/src/stream/sse.tsdocs/src/content/docs/access-control.mdxinternal/api/policy_test.gointernal/ingest/worker.gotests/e2e/sdk/ingest.test.tsinternal/discovery/discovery.gointernal/query/builder_test.gointernal/query/builder.godocs/src/content/docs/ingest-pipeline.mdinternal/api/boot_chain_test.gointernal/ingest/types_test.gointernal/testutil/testutil.gointernal/api/structured_query_test.gotests/integration/dlq_test.gointernal/api/ingest.goclients/ts/src/stream/sse.test.tsinternal/policy/rowfilter.godocs/src/content/docs/settings-directory.mdxinternal/ingest/worker_test.godocs/src/content/docs/architecture.mdtests/e2e/sdk/streaming.test.tsinternal/api/bufpool.godocs/src/content/docs/api.mdinternal/discovery/discovery_test.gointernal/policy/policy.gointernal/stream/hub.gointernal/stream/subscriber.goclients/ts/src/types.tsinternal/api/ingest_seams.gointernal/api/ingest_test.gointernal/stream/hub_test.gointernal/stream/roweval_test.gointernal/policy/rowfilter_test.goCHANGELOG.mdinternal/policy/policy_test.go
Every new function should have corresponding test cases.
📄 CodeRabbit inference engine (AGENTS.md)
Files:
internal/discovery/timestamp_test.gointernal/auth/auth_test.gointernal/stream/filter_test.gotests/integration/rowfilter_narrowing_test.gointernal/api/ingest_seams_test.gointernal/api/errors_test.gotests/integration/query_limits_test.gointernal/ingest/compact_test.gointernal/settings/validate_test.gointernal/api/bufpool_test.gointernal/api/policy_test.gointernal/query/builder_test.gointernal/api/boot_chain_test.gointernal/ingest/types_test.gointernal/api/structured_query_test.gotests/integration/dlq_test.gointernal/ingest/worker_test.gointernal/discovery/discovery_test.gointernal/api/ingest_test.gointernal/stream/hub_test.gointernal/stream/roweval_test.gointernal/policy/rowfilter_test.gointernal/policy/policy_test.go
Exactly one runtime dependency — `eventsource-parser` (SSE framing, itself dependency-free); adding a second needs the same scrutiny the first got.
📄 CodeRabbit inference engine (AGENTS.md)
Files:
clients/ts/src/table.tstests/e2e/sdk/admin.test.tstests/e2e/sdk/settings.tstests/e2e/sdk/setup.tsclients/ts/src/index.tstests/e2e/sdk/query.test.tsclients/ts/src/stream/sse.tstests/e2e/sdk/ingest.test.tsclients/ts/src/stream/sse.test.tstests/e2e/sdk/streaming.test.tsclients/ts/src/types.ts
**In MDX, leave a blank line between a JSX tag and a code fence.**
📄 CodeRabbit inference engine (AGENTS.md)
Files:
docs/src/content/docs/access-control.mdxdocs/src/content/docs/settings-directory.mdx
**Never hard-wrap prose. One paragraph is one line.** No wrapping at 72/80 columns, no "semantic linefeeds" splitting a paragraph at sentence boundaries.
📄 CodeRabbit inference engine (AGENTS.md)
Files:
docs/src/content/docs/sdk/streaming.mddocs/src/content/docs/sdk/admin.mddocs/src/content/docs/sdk/queries.mddocs/src/content/docs/ingest-pipeline.mddocs/src/content/docs/architecture.mddocs/src/content/docs/api.mdCHANGELOG.md
🧠 Learnings (7)
📚 Learning: 2026-06-26T12:23:22.696Z
Learnt from: EricAndrechek
Repo: Wave-RF/WaveHouse PR: 346
File: internal/stream/subscriber_test.go:9-28
Timestamp: 2026-06-26T12:23:22.696Z
Learning: In this Go repository, prefer table-driven tests (e.g., `[]struct{...}` with `t.Run(...)`) only for tests that cover multiple scenarios/inputs and can be cleanly enumerated. Do not artificially rewrite a clear single-scenario sequential behavioral-flow test into a table-driven form just to fit the pattern; if there’s only one meaningful scenario, keep the test as a straightforward linear flow (as in `TestSubscriber_SendDeliversThenDropsWhenFull`).
Applied to files:
internal/stream/filter_test.gointernal/settings/validate_test.gointernal/discovery/discovery_test.gointernal/api/ingest_test.gointernal/stream/roweval_test.go
📚 Learning: 2026-05-23T01:23:59.268Z
Learnt from: EricAndrechek
Repo: Wave-RF/WaveHouse PR: 174
File: internal/api/ingest_test.go:111-111
Timestamp: 2026-05-23T01:23:59.268Z
Learning: In WaveHouse Go tests in internal/api/**/*_test.go, use internal/testutil.AssertJSONErrorResponse(t, w) for HTTP error-path JSON assertions. Do not use (or reintroduce) package-local assertJSONErrorResponse helpers. AssertJSONErrorResponse verifies the response Content-Type is application/json, includes the X-Content-Type-Options: nosniff header, and that the JSON body contains an "error" field.
Applied to files:
internal/api/ingest_seams_test.gointernal/api/ingest_test.go
📚 Learning: 2026-08-19T15:44:27.183Z
Learnt from: taitelee
Repo: Wave-RF/WaveHouse PR: 500
File: internal/settings/settings.go:31-31
Timestamp: 2026-08-19T15:44:27.183Z
Learning: In the WaveHouse Go codebase, do not flag package-level lookup tables or precomputed stateless values when they are immutable and read-only, including settings.Files, validate.pipeParamTypes, mutationVerbs, nonMutationVerbs, identEscaper, intBounds, and package-level regular expressions. The no-global-state guideline applies to injected application dependencies and mutable singletons, not immutable lookup data.
Applied to files:
internal/settings/validate.gointernal/api/record_reader.go
📚 Learning: 2026-08-13T12:17:52.620Z
Learnt from: EricAndrechek
Repo: Wave-RF/WaveHouse PR: 470
File: docs/src/content/docs/reverse-proxy.mdx:137-144
Timestamp: 2026-08-13T12:17:52.620Z
Learning: For Wave-RF/WaveHouse documentation, verify claims about implementation control flow against the authoritative implementation source (for example, internal/auth/auth.go) rather than relying solely on docs/** content. Documentation may lag behind or paraphrase behavior, so control-flow claims should be confirmed in source code.
Applied to files:
docs/src/content/docs/access-control.mdx
📚 Learning: 2026-08-11T21:55:53.726Z
Learnt from: jfwoods
Repo: Wave-RF/WaveHouse PR: 434
File: clients/go/stream_test.go:159-176
Timestamp: 2026-08-11T21:55:53.726Z
Learning: For Go files in this repository, do not report direct type assertions solely because the `forcetypeassert` rule is commented out in `.golangci.yml`. Only flag a type assertion when there is an independent correctness, safety, or maintainability issue.
Applied to files:
internal/discovery/discovery_test.go
📚 Learning: 2026-05-20T01:02:00.784Z
Learnt from: EricAndrechek
Repo: Wave-RF/WaveHouse PR: 164
File: internal/api/router_test.go:289-350
Timestamp: 2026-05-20T01:02:00.784Z
Learning: In WaveHouse’s internal API tests (files matching internal/api/**/*_test.go), follow the existing separation-of-concerns convention for testing the RequireRole middleware: inject `ContextKeyRole` directly into the request `context.Context` instead of using `testutil.MakeJWT`/JWT-driven flows. Do not refactor role-gate tests to use JWT tokens—JWT parsing and token handling are covered separately in `middleware_test.go` (the dedicated JWT parsing tests), and mixing those concerns would expand the failure surface and reduce isolation.
Applied to files:
internal/api/ingest_test.go
📚 Learning: 2026-06-10T15:01:09.027Z
Learnt from: EricAndrechek
Repo: Wave-RF/WaveHouse PR: 312
File: docs/src/content/docs/development.md:0-0
Timestamp: 2026-06-10T15:01:09.027Z
Learning: In this repo’s Markdown review (all .md files), do not flag capitalization/style issues for literal paths starting with ".github/" (or any substring that is a path beginning with ".github/"). Treat ".github" as the correct lowercase dotfile directory name, even when it appears inside prose or code spans; automated checks such as LanguageTool’s "(GITHUB)" rule commonly produce false positives for this literal filesystem path.
Applied to files:
CHANGELOG.md
🪛 LanguageTool
docs/src/content/docs/architecture.md
[typographical] ~128-~128: Consider using an em dash in dialogues and enumerations.
Context: - worker.go — StartIngestWorker lau...
(DASH_RULE)
docs/src/content/docs/api.md
[style] ~600-~600: Consider using “who” when you are referring to a person instead of an object.
Context: ... the schema-versioning work. A consumer that drops a row whose length disagrees with...
(THAT_WHO)
CHANGELOG.md
[typographical] ~25-~25: Consider using an em dash in dialogues and enumerations.
Context: - **SSE: the column list is announced as ...
(DASH_RULE)
[style] ~25-~25: Since ownership is already implied, this phrasing may be redundant.
Context: ...lay follows the identical contract with its own drift state, because it writes straight...
(PRP_OWN)
[style] ~25-~25: This sentence is over 40 words long. Consider splitting it up, as shorter sentences make the text easier to read.
Context: ...ad of it with nothing to zip against. Known limitation, deferred to the schema-versioning work: those two states are not reconciled when they disagree, so if a table's column set changes while a client is connected and that client gap-fill-replays across the change, live rows arriving after the replay may carry no fresh schema frame until the next drift or a reconnect. Rows are not mislabeled — the SDK drops...
(TOO_LONG_SENTENCE)
🔇 Additional comments (37)
clients/ts/src/types.ts (1)
344-357: LGTM!Also applies to: 381-386
internal/api/policy_test.go (1)
35-35: LGTM!Also applies to: 60-60, 97-97, 138-138
internal/api/structured_query_test.go (1)
160-160: LGTM!Also applies to: 186-186, 214-217, 302-305, 317-317, 340-340, 366-366, 381-381, 397-397, 412-412, 458-458, 477-477, 493-493, 516-519
internal/auth/auth_test.go (1)
128-133: LGTM!Also applies to: 162-166
docs/src/content/docs/access-control.mdx (1)
105-117: LGTM!Also applies to: 126-170, 178-179, 201-202, 250-251, 281-287, 316-317, 414-415, 427-428, 454-454
docs/src/content/docs/sdk/admin.md (1)
41-42: LGTM!docs/src/content/docs/settings-directory.mdx (1)
46-46: LGTM!Also applies to: 60-69
tests/e2e/sdk/admin.test.ts (1)
77-86: LGTM!clients/ts/src/index.ts (1)
36-36: LGTM!Also applies to: 53-53
internal/settings/validate_test.go (1)
28-28: LGTM!Also applies to: 221-221, 248-250, 328-334, 365-373, 388-442, 469-469
internal/api/errors_test.go (1)
167-167: LGTM!tests/e2e/sdk/settings.ts (1)
56-59: LGTM!tests/e2e/sdk/setup.ts (1)
75-87: LGTM!tests/integration/rowfilter_narrowing_test.go (1)
203-203: LGTM!internal/api/structured_query.go (1)
185-186: LGTM!Also applies to: 200-204
tests/e2e/sdk/query.test.ts (1)
193-193: LGTM!Also applies to: 224-224, 267-269, 307-310, 438-440
deployments/compose/settings/policies.json (1)
5-25: LGTM!internal/policy/policy.go (2)
197-215: LGTM!
533-536: 🩺 Stability & AvailabilityNo action is needed for
IsColumnAllowedcall-site selection. Query and stream paths resolve"select"and passfalse; ingest resolves"insert"and passestrue.internal/policy/rowfilter.go (1)
15-15: LGTM!Also applies to: 121-121, 131-131
internal/policy/policy_test.go (1)
203-243: LGTM!internal/policy/rowfilter_test.go (1)
25-25: LGTM!Also applies to: 350-351
internal/query/builder.go (1)
108-110: LGTM!Also applies to: 148-149, 192-192
internal/query/builder_test.go (1)
184-184: LGTM!Also applies to: 287-287, 681-681, 710-710, 743-763, 788-788, 802-802, 819-819
internal/settings/validate.go (2)
601-620: LGTM!
299-313: 🩺 Stability & AvailabilityNo change required.
internal/settings/validate.goalready importsbytes, which resolvesbytes.TrimSpace.tests/integration/query_limits_test.go (1)
56-63: LGTM!Also applies to: 73-73, 83-83, 98-98
tests/e2e/sdk/ingest.test.ts (1)
182-182: LGTM!Also applies to: 236-236, 281-291, 293-314, 327-331, 394-396
internal/api/stream.go (1)
90-105: LGTM!Also applies to: 122-133
internal/stream/hub.go (1)
7-7: LGTM!Also applies to: 33-65, 168-168, 178-229, 235-238, 302-363, 365-367, 406-450, 463-491, 493-570, 576-578, 583-598, 602-623, 628-633
internal/stream/hub_test.go (1)
7-10: LGTM!Also applies to: 52-77, 119-169, 181-185, 200-201, 216-219, 229-236, 247-269, 280-281, 295-299, 321-324, 351-364, 379-379, 396-405, 419-420, 456-457, 480-480, 493-494, 520-520, 532-533, 545-546, 558-558, 650-657, 676-676, 696-714, 730-750, 833-833, 848-853, 874-878, 889-890, 925-932, 985-985, 997-997, 1074-1290
internal/stream/metrics.go (1)
17-17: LGTM!internal/stream/roweval_test.go (1)
1-111: LGTM!internal/stream/subscriber.go (1)
3-3: LGTM!Also applies to: 18-18, 52-88
docs/src/content/docs/sdk/streaming.md (1)
100-101: LGTM!Also applies to: 120-121
internal/stream/filter_test.go (1)
10-98: LGTM!tests/e2e/sdk/streaming.test.ts (1)
34-63: LGTM!
|
Superseded by a seven-PR stack. Same work, same final tree, split so each piece is reviewable on its own. Merge top to bottom. Each PR is based on the one above it, so GitHub shows only that layer's diff.
Why the splitThis PR reached 35 commits across six unrelated subsystems. The reviewers kept finding real defects in it — a blocker that broke ingest for any table with a What changed while splitting
Review stateBoth pre-push reviewers returned Follow-ups filed along the way
Note on CI
Closing in favour of the stack. Nothing here is lost — #555's tree is the finished state. |
## Summary `POST /v1/ingest` used to sniff the body and treat the header as a hint: the first non-whitespace byte chose between a single object and an array, and an NDJSON body whose first line happened to start with `[` was re-framed as a JSON array — silently, as a whole-request reinterpretation rather than a per-record error. The header is now required and authoritative. A request declaring nothing, or a media type not in the accepted list, is `415` before the body is parsed, with the supported types named in the body. The declared type chooses the format *family* — `application/json` versus the four NDJSON spellings — and within the JSON family the first non-whitespace byte still picks array versus single object. The bytes never choose the family, so an NDJSON body is read as NDJSON whatever its first byte and a bad line fails as a per-record error. The header is parsed by `mime.ParseMediaType`, so the grammar is RFC 9110's §8.3 `media-type` rather than one of ours. Only the media type selects the format; parameters are ignored, so no malformed parameter costs the request — `; charset`, `;;`, a value left mid-quote, even a name repeated with different values all read as `application/json`. One exception, below: a malformed parameter on a line that also contains a comma. **Duplicate declarations are the part most worth reviewing.** `Content-Type` is a *singleton* field (§8.3), and §5.3 forbids repeating a field line unless the field allows comma-list recombination — `media-type` does not. So both duplicate spellings are malformed input, and §8.3 says so directly, warning that recipients who resolve the resulting pseudo-list "by using the last syntactically valid member" cause "interoperability and security issues". Ingest therefore takes no member: - **Repeated header lines** (what curl and many proxies send) are all resolved and must agree on the format *and* on whether ingest reads it at all. `application/x-ndjson` alongside `application/ndjson; charset=utf-8` reads as NDJSON, because once they agree, which one gets honored stops mattering. Disagreement is `415` rather than resolution to the first — honoring the first would let an NDJSON body be read as one JSON object, ingesting record one and discarding the rest behind a `200`. - **A comma-joined value** (what a proxy merging duplicates sends) is refused. Where it fails outright there is no media type to take. Where it does parse a media type — `application/json; charset=utf-8, application/x-ndjson` yields `application/json` — it is still refused, because the comma may be a second declaration joined on and the error cannot distinguish that from a comma inside data. The security-critical detail is in `ingestFormatOne`. `ParseMediaType` returns `ErrInvalidMediaParameter` both for a merely-malformed parameter *and* when a second declaration was comma-joined on after a parameter — `application/json; charset=utf-8, application/x-ndjson` yields mediatype `application/json`. Tolerating that error unconditionally silently resolves a joined disagreement to its first member, which is exactly the truncation above. The two are indistinguishable from the error alone, so a comma on a line that did not parse cleanly is refused. That guard fails closed, and it is where I would look first. **Four additional shapes 0.1.0 accepted now `415`** (beyond repeated header lines that disagree, which it also accepted; see the CHANGELOG): a present-but-empty header; a value with a leading or trailing comma; a comma-joined value that does not parse as a single media type; and a malformed parameter on a line that also carries a comma. The last is an over-rejection and is tracked in #563. Note a comma is not disqualifying on its own: inside a quoted parameter value it is legal data, so `application/json; a=", application/x-ndjson; b="` is one media type and is accepted, even though an intermediary may have built it by illegally joining two lines. The server cannot tell. The TS SDK sends `application/json` for a single object and `application/x-ndjson` for arrays and `insertNDJSON`, on every path, so no SDK caller is affected. A hand-rolled client that relied on sniffing must now declare the type. ## Stacked PR This is **part 3 of 7** (parts 0, 1 and 2 merged) in a stack that replaces #540. Each PR is based on the one above it, so review this PR's own diff against its base — GitHub shows only this layer's changes. | # | Branch | Base | | | - | ------ | ---- | - | | 0 | `stack/0-classify-paths` | `main` | ✅ merged as #549 | | 1 | `stack/1-discovery` | `main` | ✅ merged as #550 | | 2 | `stack/2-policy` | `main` | ✅ merged as #551 | | 3 | `stack/3-content-type` | `main` | **→ this PR** | | 4 | `stack/4-seams` | `stack/3-content-type` | | | 5 | `stack/5-positional-wire` | `stack/4-seams` | | | 6 | `stack/6-computed-columns` | `stack/5-positional-wire` | | Merge in order, top to bottom. Rebasing or squashing out of order will make the later PRs' diffs unreadable. ## Response-size bound The 415 echoes what the caller declared, and it is decided **before the body is read** — so a header-only request, needing no credentials under the shipped compose policy, could size the response. Measured before the fix: 1.03 MB of headers produced a 4.65 MB body (and far worse over HTTP/2, where HPACK indexes a repeated header line to about a byte on the wire). Both dimensions are caller-controlled and both are bounded now: at most **four distinct** declarations, each capped at 128 bytes, then `"…and N more"`. The declaration that actually disagreed is pinned into the echo, so four agreeing spellings cannot crowd it out. Pinned by tests that fail if either bound is removed. ## Test plan - [x] `make ci` green locally on this branch's exact tree (verify, unit, integration against live ClickHouse, e2e, all coverage gates). Now that #551 has merged and this PR's base is `main`, GitHub CI runs on `pull_request` automatically — earlier runs on this branch were manual dispatches, so check the SHA before relying on an old one. - [x] The branch descends from its base and carries only this layer's change (plus any follow-up commits answering review) - [x] `go.mod` / `go.sum` untouched; no new dependencies ## Review Both pre-push reviewers ran against this branch — no logged skip. They gated every push, over several rounds; the last rounds reviewed the `mime.ParseMediaType` rewrite as a fresh change rather than as a delta, since the earlier approvals covered code the rewrite deleted. Reviewer findings that shaped the result, rather than only polishing it: - The first cut of the `ErrInvalidMediaParameter` guard tolerated the error unconditionally, which reintroduced the silent-truncation hazard the agreement rule exists to prevent. - A fourth (now fifth) tightening was neither listed nor pinned, and the known-limit test could not have caught a regression in it. - "A duplicate parameter name is a 415" was false: Go only errors when the values differ. - The docs' RFC delegation was falsifiable by their own example with its closing quote dropped. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Summary
Groundwork for the upcoming native type layer (chtypes). Nothing here imports or references it — this lands everything that does not depend on it, so the follow-up is a wiring change rather than a rewrite.
Four breaking changes, each with its own CHANGELOG entry:
policies.jsonis role-first.tables.<table>.select.<role>becomestables.<table>.<role>.select. Field names and semantics are unchanged; only the nesting moves.selectandinsertare now distinct types, so a field on the wrong side is a validation error instead of being accepted and ignored. There is no automatic conversion — convert the file by hand and runwavehouse validatebefore restarting. A pre-v2 document is reported as one clear finding pointing at the migration note, rather than a confusing strict-decode error.Content-Type, and honors it. The header used to be a hint the body could override. A request declaring nothing, or something ingest doesn't read, is now415before the body is parsed. An NDJSON body is read as NDJSON whatever its first byte, so a bad line fails as a per-record error instead of silently re-framing the request. The body still picks arity within the JSON family.databecomesformat/columns/row(oneJSONCompactEachRowline). In-flight messages from an older version are not readable by the new worker — drain the ingest queue before deploying. The worker groups a batch by column list and writesINSERT INTO … (cols) FORMAT JSONCompactEachRow.event: schemaframe before its first row and again on drift; data frames carryrowinstead ofdata. The TS SDK consumes it and still yields row objects, so.stream()/.liveQuery()/StreamEvent.dataare unchanged; a rawEventSourceconsumer must now zip rows itself.Also: schema discovery captures each table's DDL, column ordinals and default expressions plus the server version; ingest reads its body up front and puts schema validation, insert checks and row visibility behind interfaces with today's behavior as the default.
Transitional divergence worth knowing
input_format_null_as_default=1turns an omitted field (now an explicitnullin its column's slot) back into the column default. ANullablecolumn that also has aDEFAULTexpression now takes its default where it previously storedNULL. Every other column type behaves as before.Deferred, tracked for the policy workstream
POST /v1/ops/policy/validatedecodes leniently — no strict fields, no duplicate-key scan, no legacy-layout detector — so a pre-v2policies.jsonanswers{"valid": true}while file adoption refuses it. Verified: the legacy document decodes to a role literally namedselectwith no grants, whichpolicy.Validateaccepts. So the dry-run greenlights a file that would refuse to boot, and the cross-file "grant sets neither select nor insert" warning doesn't reach that path either. This matters because an operator following the migration note would most naturally dry-run their converted file first.Smaller: the legacy-layout detector false-positives on a genuine v2 policy whose role is literally named
selectorinsert, with the same misleading migration error. Documented in the detector's comment; rename such a role.Deferred to the schema-versioning task
Replay and the live fan-out track schema drift in separate state and never reconcile it. If a table's column set changes while a client is connected and that client gap-fill-replays across the change, live rows after the replay may carry no fresh
event: schemaframe until the next drift or a reconnect. The SDK drops a row whose length disagrees with its last announced list rather than zipping it under the wrong names — lost rows, never wrong ones; reconnecting resynchronizes. Noted inhub.go,api.mdandsdk/streaming.md.Test plan
make cipasses locally — verify, unit, integration (live ClickHouse), e2e, all coverage gatesgo test ./...on its own (replayed in a scratch worktree)TestRowFilterNumeric_DifferentialAgainstClickHouse(6 subtests) andTestTimestampCanonicalization_DifferentialAgainstClickHouse(6 subtests)Validate()still checks both sides of every role's grant; two tests pin that neither side excuses the othergo.mod/go.sumuntouched; no new dependenciesCoverage: unit 90.7% (≥80), integration 31.2% (≥20), e2e 61.3% (≥60), Go total 90.8% (≥80), ts-total 80.12% (≥50).
Two defects were found by the e2e suite during development and fixed in-branch: a full subscriber queue could drop a schema frame while landing the row it described (silent mislabeling — now the row is withheld with its announcement), and a check clause naming a column the table lacks silently dropped its auto-injected value under the new positional encoder (now rejected, naming the column).
Related Issues
None — groundwork for the chtypes work.
Important
The two pre-push reviewer agents were skipped at the author's request, recorded via
scripts/skip-pre-push-review.sh. The markers on this push are logged skips, not passes — this branch has had no independent code or docs review. Given it touches the policy engine and the SSE row-visibility path, it wants a careful human read.🤖 Generated with Claude Code
https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ