Skip to content

feat!: policy v2, positional wire shape, and declared ingest format - #540

Closed
EricAndrechek wants to merge 10 commits into
mainfrom
pre-chtypes-prep
Closed

EricAndrechek wants to merge 10 commits into
mainfrom
pre-chtypes-prep

Conversation

@EricAndrechek

Copy link
Copy Markdown
Member

Summary

Groundwork for the upcoming native type layer (chtypes). Nothing here imports or references it — this lands everything that does not depend on it, so the follow-up is a wiring change rather than a rewrite.

Four breaking changes, each with its own CHANGELOG entry:

  • Policy format v2 — policies.json is role-first. tables.<table>.select.<role> becomes tables.<table>.<role>.select. Field names and semantics are unchanged; only the nesting moves. select and insert are now distinct types, so a field on the wrong side is a validation error instead of being accepted and ignored. There is no automatic conversion — convert the file by hand and run wavehouse validate before restarting. A pre-v2 document is reported as one clear finding pointing at the migration note, rather than a confusing strict-decode error.
  • Ingest requires a declared Content-Type, and honors it. The header used to be a hint the body could override. A request declaring nothing, or something ingest doesn't read, is now 415 before the body is parsed. An NDJSON body is read as NDJSON whatever its first byte, so a bad line fails as a per-record error instead of silently re-framing the request. The body still picks arity within the JSON family.
  • NATS envelope v2 — the row travels positionally. data becomes format / columns / row (one JSONCompactEachRow line). In-flight messages from an older version are not readable by the new worker — drain the ingest queue before deploying. The worker groups a batch by column list and writes INSERT INTO … (cols) FORMAT JSONCompactEachRow.
  • SSE — the column list is announced separately. Each connection gets an event: schema frame before its first row and again on drift; data frames carry row instead of data. The TS SDK consumes it and still yields row objects, so .stream() / .liveQuery() / StreamEvent.data are unchanged; a raw EventSource consumer must now zip rows itself.

Also: schema discovery captures each table's DDL, column ordinals and default expressions plus the server version; ingest reads its body up front and puts schema validation, insert checks and row visibility behind interfaces with today's behavior as the default.

Transitional divergence worth knowing

input_format_null_as_default=1 turns an omitted field (now an explicit null in its column's slot) back into the column default. A Nullable column that also has a DEFAULT expression now takes its default where it previously stored NULL. Every other column type behaves as before.

Deferred, tracked for the policy workstream

POST /v1/ops/policy/validate decodes leniently — no strict fields, no duplicate-key scan, no legacy-layout detector — so a pre-v2 policies.json answers {"valid": true} while file adoption refuses it. Verified: the legacy document decodes to a role literally named select with no grants, which policy.Validate accepts. So the dry-run greenlights a file that would refuse to boot, and the cross-file "grant sets neither select nor insert" warning doesn't reach that path either. This matters because an operator following the migration note would most naturally dry-run their converted file first.

Smaller: the legacy-layout detector false-positives on a genuine v2 policy whose role is literally named select or insert, with the same misleading migration error. Documented in the detector's comment; rename such a role.

Deferred to the schema-versioning task

Replay and the live fan-out track schema drift in separate state and never reconcile it. If a table's column set changes while a client is connected and that client gap-fill-replays across the change, live rows after the replay may carry no fresh event: schema frame until the next drift or a reconnect. The SDK drops a row whose length disagrees with its last announced list rather than zipping it under the wrong names — lost rows, never wrong ones; reconnecting resynchronizes. Noted in hub.go, api.md and sdk/streaming.md.

Test plan

  • make ci passes locally — verify, unit, integration (live ClickHouse), e2e, all coverage gates
  • Every one of the 10 commits builds, vets and passes go test ./... on its own (replayed in a scratch worktree)
  • Security oracles ran and passed, not skipped: TestRowFilterNumeric_DifferentialAgainstClickHouse (6 subtests) and TestTimestampCanonicalization_DifferentialAgainstClickHouse (6 subtests)
  • Validate() still checks both sides of every role's grant; two tests pin that neither side excuses the other
  • go.mod / go.sum untouched; no new dependencies
  • CHANGELOG entry per breaking change; docs updated for policy shape, ingest format and the SSE contract

Coverage: unit 90.7% (≥80), integration 31.2% (≥20), e2e 61.3% (≥60), Go total 90.8% (≥80), ts-total 80.12% (≥50).

Two defects were found by the e2e suite during development and fixed in-branch: a full subscriber queue could drop a schema frame while landing the row it described (silent mislabeling — now the row is withheld with its announcement), and a check clause naming a column the table lacks silently dropped its auto-injected value under the new positional encoder (now rejected, naming the column).

Related Issues

None — groundwork for the chtypes work.

Important

The two pre-push reviewer agents were skipped at the author's request, recorded via scripts/skip-pre-push-review.sh. The markers on this push are logged skips, not passes — this branch has had no independent code or docs review. Given it touches the policy engine and the SSE row-visibility path, it wants a careful human read.

🤖 Generated with Claude Code

https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ

EricAndrechek and others added 10 commits September 1, 2026 08:58
Additive metadata the upcoming native type layer needs, captured on the
same refresh as the columns so they can never describe different servers:

- Column gains DefaultExpression and Position, both scanned from the
  widened system.columns select. TableSchema.Columns was already ordered
  by position, so declaration order needed no new structure.
- TableSchema gains DDL from system.tables.create_table_query. It is
  json:"-": the schema endpoint marshals TableSchema straight to the
  client and an external-engine table (S3, MySQL, Kafka) carries its
  credentials in that statement. A table listed in system.tables with no
  system.columns rows is skipped, never published column-less.
- SchemaRegistry gains ServerVersion(), from a SELECT version() probe
  next to the existing SELECT timezone().

Both new queries fail the refresh on error, matching timezone() and
system.columns: callers keep the prior cache and retry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
BREAKING CHANGE: policies.json moves from tables.<table>.select.<role> to
tables.<table>.<role>.select. Field names and semantics are unchanged; only
the nesting moves. There is no automatic conversion — convert the file by
hand and run `wavehouse validate` before restarting.

A role now appears once per table, with two optional blocks stating what it
may do. The blocks are separate types rather than one struct whose halves
were inert per operation: SelectPermissions carries the column lists, the
row filter, the aggregation rules and the four limits; InsertPermissions
carries the column lists and the check clauses. A field on the wrong side
is a validation error now instead of being accepted and ignored.

ResolvedPermissions splits the same way (.Select / .Insert), and
IsColumnAllowed takes the side to consult — which is what keeps the read
allowlist from ever answering a write question. Validate still checks BOTH
sides for every role; the insert-side rejection of _neq/_gt/_lt on check
clauses is unchanged. resolvedPredicate/resolvePredicates are exported, and
the resolved insert side carries CheckPredicates alongside CheckClauses for
a consumer that does not exist yet.

settings validation gains a layout detector: a pre-v2 document is reported
as one finding naming the table and operation with a pointer to the docs
migration note, rather than a strict-decode "unknown field" error — or, for
an empty operation block, silently decoding as a role named "select". A
grant that sets neither operation now warns.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
BREAKING CHANGE: POST /v1/ingest rejects a request that declares no
Content-Type, or one it does not read, with 415 — listing the accepted
types. Clients that relied on body sniffing must send a header.

The header used to be a hint: the first non-whitespace byte chose the
format, and an application/x-ndjson body starting with '[' was silently
re-read as a JSON array. Now the declaration decides. An NDJSON body is
read as NDJSON whatever its first byte, so a line that is not a JSON
object fails as a per-record error through the existing recordReject
path instead of re-framing the whole request.

The body still picks arity within the JSON family — '[' is an array,
anything else a single object — since those are one format at different
lengths. The choice is modeled as an IngestFormat where the sniffing
lived, keeping the CSV slot where the old comment marked it.

The TS SDK already sent a header on both paths; it now states it at each
call site rather than leaning on the request default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
…encoder

Groundwork only — behavior is identical after this commit.

Buffered body: Handle reads the whole (MaxBytesReader-capped) body into a
pooled *bytes.Buffer and runs the record readers over those bytes instead
of the live connection. The 413 now surfaces at that read rather than
mid-iteration; same status, same message. newRecordReader takes an
already-resolved format and the body bytes, so the 415 is decided from the
header before any byte is read. The sniff window is kept as-is so a body of
leading whitespace longer than the window still reads as empty. UseNumber
stays on every decoder.

Seams, each an interface with a default implementation delegating to
today's code unchanged:

- RecordValidator (schema validation + timestamp canonicalization). The two
  call sites stay where they are, with the check-clause block between them
  — merging them would move checks onto canonicalized values.
  CanonicalizeTimestamps returns nothing, matching discovery's function: it
  is deliberately fail-open, and an error return would invite changing that.
- InsertChecker (the _eq and _in comparisons), wrapping checkValueMatches
  and valueInSet verbatim. Auto-injection stays in processRecord.
- stream.RowEvaluator (row visibility), reached by both the live fan-out and
  replay through the existing rowAdmitted step.

All three are nil-safe: an un-wired handler or Hub uses the default rather
than panicking past the check.

New internal/ingest/compact.go: EncodeCompactRow renders a record as one
JSONCompactEachRow line — one value per schema column in declaration order,
a missing column as null, json.Number digits preserved. Serialization only;
nothing calls it yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
BREAKING CHANGE: the NATS EventMessage envelope replaces `data` with
`format`/`columns`/`row`, and SSE data frames carry `row` (a positional
array) preceded by an `event: schema` frame naming the columns. In-flight
NATS messages from an older version are NOT readable by the new worker —
it acks and drops an envelope whose format is absent or unknown — so drain
the ingest queue before deploying.

A batch of rows for one table now carries the column names once instead of
repeating every key on every row, and a reader can tell a schema change
mid-stream from a reordering.

Worker: groups a batch by column list, so a schema change splits the
INSERT rather than corrupting it, and writes
`INSERT INTO {table} (cols) FORMAT JSONCompactEachRow` with the identifiers
quoted by the same helper the query builder uses. It adds
input_format_null_as_default=1, which turns a column the record omitted —
now an explicit null in its slot — back into that column's default.
TRANSITIONAL DIVERGENCE: a Nullable column that also has a DEFAULT
expression now takes its default where it previously stored NULL.

SSE: each connection is told its projected column list before its first
row — at subscribe time from the registry, and again on drift — and never
sees a data frame it cannot zip. The schema frame carries no `id:` line: an
empty one would clear the client's Last-Event-ID. Replay follows the same
contract with its own drift state, because it writes straight to the socket
while live events queue behind it.

Row cells are copied as their original bytes at every hop, so a 64-bit id
past 2^53 keeps every digit. An envelope whose columns and row cannot be
paired is dropped by the worker and by the fan-out — there is no way to say
which value belongs to which column, so no role gets it.

The TS SDK consumes the schema event and zips rows back into objects, so
.stream() and .liveQuery() still yield row objects: the public API is
unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
- SDK streaming guide: StreamEvent.data is still a row object, but say
  what moved underneath — the positional wire, the omitted-column null,
  and that a raw EventSource consumer must zip rows itself.
- CHANGELOG: split the wire change into its two breaking halves (NATS
  envelope v2 with the drain-before-deploy warning, and the SSE
  schema-event contract), and add the two entries the earlier commits
  left out — the widened schema discovery, and the buffered ingest body
  plus the type-layer seams.

The per-page doc updates for each change landed with the commit that made
it; this is the sweep for what those missed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
Two defects the e2e run found.

deliver() sent the schema frame and the data frame as separate
non-blocking Sends, so a queue that was full for the first and drained by
the handler before the second would deliver a row while its announcement
was dropped — the client then zips that row against a stale column list,
which is silent mislabeling where a dropped row is a visible gap. The
announcement is now recorded only once it is actually queued, and a
dropped announcement drops its row with it, so the next event announces
again and a transient full queue stays recoverable.

The e2e settings helper's referencedRoles() still walked the pre-v2
nesting (table.select as a role map), so the derived roles.json declared
nothing and every reload was rejected. Under the role-first layout a
table's own keys are its roles. tsc could not catch this: indexing a
Record<string, RolePermissions> with the literal "select" type-checks.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
The case sends a malformed body to assert a 400. It set no Content-Type,
which `fetch` fills in as text/plain — now a 415 refused before the body
is read, so the test was asserting the old contract. Declare
application/json so it still exercises what it means to.

Adds the companion case the change deserves end to end: an undeclared and
an unreadable Content-Type are both 415, and the message names every type
ingest accepts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
The published row carries only schema columns, by position, so a value
auto-injected for a checked column that does not exist in the table was
dropped on the way out: the record inserted WITHOUT the value the policy
required, and answered 200 {"ok":true}. The misconfiguration was invisible.
Under the previous wire shape the same policy surfaced loudly — the
unknown column reached ClickHouse and the row landed in the DLQ.

The check clause is now refused at the top of the loop, naming the column
and the table. The supplied-value case never reaches it (schema validation
rejects an unknown column one step earlier, with its own message), so in
practice this fires on the omitted-and-auto-injected path; the guard sits
at the top anyway so it does not depend on that ordering holding.

Status is 403, consistent with the check-clause rejections beside it,
though the fault is server-side configuration rather than anything the
caller can fix — the message is what carries the diagnosis.

Adds TableSchema.HasColumn for the lookup.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
No behavior change.

The replay and live paths track schema drift separately, and the earlier
note claimed the only cost was a redundant announcement. That is not the
whole truth: the two states are never reconciled, so a client that
gap-fill-replays across a column-set change can end up with a list older
than the one the live path already recorded at subscribe time, and live
rows after the replay carry no fresh schema frame until the next drift or
a reconnect. Rows are not mislabeled — a length mismatch is dropped, not
guessed at — so the cost is availability. Recorded as a known limitation
in the code, the CHANGELOG, the SSE reference and the SDK streaming guide,
deferred to the schema-versioning work.

The subscriber's schema comment implied concurrent delivery was
anticipated. deliver()'s check→send→record is not atomic and is only safe
because Broadcast runs on one goroutine — the single inline jetstream
Consume callback the hub bridge registers. Documented, with what a future
concurrency change would have to add. The comment also still claimed
replay records against that field, which it no longer does.

Stale comments: Frame.Kind omitted schema; groupByColumns overclaimed
cross-group arrival order (groups follow first appearance); and three
ingest tests still described the pre-rewrite sniffer and the streaming
body-cap path.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ
@github-actions github-actions Bot added documentation Improvements or additions to documentation go Pull requests that update go code area/observability Metrics, logs, traces, health, profiling area/api HTTP handlers, routing, middleware area/ingest Ingest pipeline (Bento, batching, DLQ) area/query Structured query AST, SQL builder area/policy Access control policies (Hasura-style) area/sdk TypeScript SDK (clients/ts/) area/docs Documentation, site/, README area/infra CI, build, deploy, Docker, release labels Sep 1, 2026
@coderabbitai

coderabbitai Bot commented Sep 1, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Summary by CodeRabbit

  • Breaking Changes

    • Ingest requests must include a supported Content-Type; otherwise, they return 415.
    • Streaming now announces schemas separately and sends positional rows.
    • Access policies now use role-first nesting with separate read and write permissions.
  • New Features

    • Schema discovery includes column order, defaults, table DDL, and server version metadata.
    • TypeScript exports now include select and insert permission types.
  • Documentation

    • Updated API, streaming, ingest, access-control, and configuration guidance with migration details.

Walkthrough

The PR introduces role-first policy v2, declared ingest media types, compact positional event envelopes, schema-aware SSE frames, pluggable validation seams, and expanded schema discovery metadata. SDK types, deployment settings, documentation, and tests are updated accordingly.

Changes

Schema and policy contracts

Layer / File(s) Summary
Schema discovery metadata
internal/discovery/..., internal/testutil/testutil.go
Schema refresh records server version, table DDL, column defaults, and declaration positions. DDL is excluded from serialized schema output.
Role-first policy model
internal/policy/..., internal/settings/validate.go, clients/ts/src/types.ts, clients/ts/src/index.ts, deployments/compose/settings/policies.json
Policies now use table → role → select/insert. Read and write permissions use separate types and resolved state. Legacy operation-first layouts are reported during validation.
Policy integrations and references
internal/query/..., internal/api/structured_query.go, tests/e2e/sdk/*, tests/integration/*, docs/src/content/docs/access-control.mdx, docs/src/content/docs/settings-directory.mdx, docs/src/content/docs/api.md
Query limits, column checks, filters, fixtures, examples, and validation tests use the new policy shape.

Ingest and event envelopes

Layer / File(s) Summary
Declared ingest format and request buffering
internal/api/record_reader.go, internal/api/ingest.go, internal/api/bufpool.go, clients/ts/src/table.ts
Ingest requires a supported Content-Type and returns 415 before body parsing for missing or unsupported types. Request bodies use capped pooled buffers. The TypeScript single-object insert path declares application/json.
Compact event encoding and worker insertion
internal/ingest/*, internal/api/ingest.go, internal/ingest/worker.go, tests/integration/dlq_test.go
Events carry format, columns, and positional row data. The worker groups messages by column signature and inserts each group with JSONCompactEachRow.
Validation seams and ingest coverage
internal/api/ingest_seams.go, internal/api/ingest_test.go, internal/api/ingest_seams_test.go, internal/ingest/*_test.go
Record validation, timestamp canonicalization, and insert checks use nil-safe interfaces with default implementations. Tests cover format selection, compact rows, policy checks, and malformed inputs.

Schema-aware streaming

Layer / File(s) Summary
SSE schema and positional rows
internal/stream/hub.go, internal/stream/subscriber.go, internal/stream/metrics.go, internal/api/stream.go
Connections announce projected columns with schema frames. Data frames carry positional rows. Subscribers and replay projectors track schema signatures independently.
Row admission and replay flow
internal/stream/hub.go, internal/stream/roweval_test.go, internal/stream/hub_test.go
Live and replay delivery use a configurable row evaluator. Schema frames precede data frames and are re-announced when projections change.
TypeScript decoding and streaming documentation
clients/ts/src/stream/sse.ts, clients/ts/src/stream/sse.test.ts, docs/src/content/docs/sdk/streaming.md, docs/src/content/docs/api.md
The SDK stores the latest schema, validates positional rows, reconstructs row objects, and drops rows without a valid matching schema.

Documentation and changelog

Layer / File(s) Summary
Wire-format and migration documentation
CHANGELOG.md, docs/src/content/docs/architecture.md, docs/src/content/docs/ingest-pipeline.md, docs/src/content/docs/sdk/queries.md
Documentation describes the new compact event envelope, declared ingest formats, schema frames, worker batching, and policy migration behavior.

Estimated code review effort: 5 (Critical) | ~120 minutes

Merge Risk: 🟠 High · up to 39bc0

This PR introduces an incompatible ingest queue format and positional streaming changes. An incorrect deployment order or rollback can acknowledge queued records without inserting or preserving them, causing permanent data loss; merge should wait for a safer compatibility path or an enforced versioned rollout plan.

Suggested reviewers: taitelee

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 71.97% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 239 functions across 50 files. (12 skippe… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main breaking changes: policy v2, positional wire formats, and declared ingest formats.
Description check ✅ Passed The description is directly related to the changeset and explains the breaking changes, implementation scope, deferred issues, and test coverage.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 71.97% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 239 functions across 50 files. (12 skipped: 10 unsupported, 2 over the file limit.)

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch pre-chtypes-prep
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch pre-chtypes-prep

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: Go

Go

The overall line coverage in commit 39bc0f9 in the pre-chtypes-prep branch remains at 90%, unchanged from commit 2812cb8 in the main branch.

Show a line coverage summary of the most impacted files.
File main 2812cb8 pre-chtypes-prep 39bc0f9 +/-
internal/api/record_reader.go 92% 87% -5%
internal/api/ingest.go 97% 95% -2%
internal/ingest/worker.go 95% 93% -2%
internal/settings/validate.go 97% 95% -2%
internal/policy/policy.go 98% 96% -2%
internal/stream/hub.go 97% 96% -1%
internal/api/stream.go 55% 58% +3%
internal/api/bufpool.go 0% 83% +83%
internal/api/ingest_seams.go 0% 100% +100%
internal/ingest/compact.go 0% 100% +100%

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

📚 Docs preview is live → https://e9d32fbc-wavehouse-docs.wave-rf.workers.dev

  • Commit — 39bc0f9: docs: record the replay/schema deferral and correct stale comments
  • Author — @EricAndrechek, Claude Opus 5 (1M context)
  • Committed — 2026-09-01 10:37 (UTC-04:00)
  • Deployed — 2026-09-01 10:50 EDT

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
clients/ts/src/table.ts (1)

143-144: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Correct the server parsing statement.

IngestHandler.Handle buffers the capped request body before it creates the record reader. The server does not stream parsing. State that the server buffers the capped body before parsing it.


ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Team

Run ID: 8b811267-08c7-4c05-8873-8bf8da811ef4

📥 Commits

Reviewing files that changed from the base of the PR and between 2812cb8 and 39bc0f9.

📒 Files selected for processing (62)
  • CHANGELOG.md
  • clients/ts/src/index.ts
  • clients/ts/src/stream/sse.test.ts
  • clients/ts/src/stream/sse.ts
  • clients/ts/src/table.ts
  • clients/ts/src/types.ts
  • deployments/compose/settings/policies.json
  • docs/src/content/docs/access-control.mdx
  • docs/src/content/docs/api.md
  • docs/src/content/docs/architecture.md
  • docs/src/content/docs/ingest-pipeline.md
  • docs/src/content/docs/sdk/admin.md
  • docs/src/content/docs/sdk/queries.md
  • docs/src/content/docs/sdk/streaming.md
  • docs/src/content/docs/settings-directory.mdx
  • internal/api/boot_chain_test.go
  • internal/api/bufpool.go
  • internal/api/bufpool_test.go
  • internal/api/errors_test.go
  • internal/api/ingest.go
  • internal/api/ingest_seams.go
  • internal/api/ingest_seams_test.go
  • internal/api/ingest_test.go
  • internal/api/policy_test.go
  • internal/api/record_reader.go
  • internal/api/stream.go
  • internal/api/structured_query.go
  • internal/api/structured_query_test.go
  • internal/auth/auth_test.go
  • internal/discovery/discovery.go
  • internal/discovery/discovery_test.go
  • internal/discovery/timestamp_test.go
  • internal/ingest/compact.go
  • internal/ingest/compact_test.go
  • internal/ingest/types.go
  • internal/ingest/types_test.go
  • internal/ingest/worker.go
  • internal/ingest/worker_test.go
  • internal/policy/policy.go
  • internal/policy/policy_test.go
  • internal/policy/rowfilter.go
  • internal/policy/rowfilter_test.go
  • internal/query/builder.go
  • internal/query/builder_test.go
  • internal/settings/validate.go
  • internal/settings/validate_test.go
  • internal/stream/filter_test.go
  • internal/stream/hub.go
  • internal/stream/hub_test.go
  • internal/stream/metrics.go
  • internal/stream/roweval_test.go
  • internal/stream/subscriber.go
  • internal/testutil/testutil.go
  • tests/e2e/sdk/admin.test.ts
  • tests/e2e/sdk/ingest.test.ts
  • tests/e2e/sdk/query.test.ts
  • tests/e2e/sdk/settings.ts
  • tests/e2e/sdk/setup.ts
  • tests/e2e/sdk/streaming.test.ts
  • tests/integration/dlq_test.go
  • tests/integration/query_limits_test.go
  • tests/integration/rowfilter_narrowing_test.go

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Review details
⏰ Context from checks skipped due to timeout. (3)
  • GitHub Check: Docs preview
  • GitHub Check: Coverage
  • GitHub Check: E2E tests
🧰 Additional context used
📓 Path-based instructions (7)
**Opt a page into the Cloud CTA with `cloudCta` frontmatter**, not by importing the component.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • docs/src/content/docs/access-control.mdx
  • docs/src/content/docs/settings-directory.mdx
A new test file must (1) add its suite name to `SUITES` in `tables.ts` and (2) get its names via `const T = suiteTables("")`, then reference `T.clicks` etc. — never a bare `clicks`.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • tests/e2e/sdk/admin.test.ts
  • tests/e2e/sdk/query.test.ts
  • tests/e2e/sdk/ingest.test.ts
  • tests/e2e/sdk/streaming.test.ts
Every code change updates its docs + `CHANGELOG.md` in the same PR

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • clients/ts/src/table.ts
  • tests/e2e/sdk/admin.test.ts
  • internal/stream/metrics.go
  • internal/discovery/timestamp_test.go
  • tests/e2e/sdk/settings.ts
  • internal/auth/auth_test.go
  • docs/src/content/docs/sdk/streaming.md
  • docs/src/content/docs/sdk/admin.md
  • internal/api/structured_query.go
  • internal/stream/filter_test.go
  • tests/integration/rowfilter_narrowing_test.go
  • tests/e2e/sdk/setup.ts
  • internal/api/ingest_seams_test.go
  • clients/ts/src/index.ts
  • docs/src/content/docs/sdk/queries.md
  • internal/api/errors_test.go
  • internal/ingest/types.go
  • tests/e2e/sdk/query.test.ts
  • tests/integration/query_limits_test.go
  • internal/ingest/compact_test.go
  • internal/api/stream.go
  • internal/ingest/compact.go
  • internal/settings/validate.go
  • internal/settings/validate_test.go
  • internal/api/record_reader.go
  • internal/api/bufpool_test.go
  • clients/ts/src/stream/sse.ts
  • docs/src/content/docs/access-control.mdx
  • internal/api/policy_test.go
  • internal/ingest/worker.go
  • tests/e2e/sdk/ingest.test.ts
  • internal/discovery/discovery.go
  • internal/query/builder_test.go
  • internal/query/builder.go
  • docs/src/content/docs/ingest-pipeline.md
  • internal/api/boot_chain_test.go
  • internal/ingest/types_test.go
  • internal/testutil/testutil.go
  • internal/api/structured_query_test.go
  • tests/integration/dlq_test.go
  • internal/api/ingest.go
  • clients/ts/src/stream/sse.test.ts
  • internal/policy/rowfilter.go
  • docs/src/content/docs/settings-directory.mdx
  • internal/ingest/worker_test.go
  • docs/src/content/docs/architecture.md
  • tests/e2e/sdk/streaming.test.ts
  • internal/api/bufpool.go
  • docs/src/content/docs/api.md
  • internal/discovery/discovery_test.go
  • internal/policy/policy.go
  • internal/stream/hub.go
  • internal/stream/subscriber.go
  • clients/ts/src/types.ts
  • internal/api/ingest_seams.go
  • internal/api/ingest_test.go
  • internal/stream/hub_test.go
  • internal/stream/roweval_test.go
  • internal/policy/rowfilter_test.go
  • CHANGELOG.md
  • internal/policy/policy_test.go
Every new function should have corresponding test cases.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • internal/discovery/timestamp_test.go
  • internal/auth/auth_test.go
  • internal/stream/filter_test.go
  • tests/integration/rowfilter_narrowing_test.go
  • internal/api/ingest_seams_test.go
  • internal/api/errors_test.go
  • tests/integration/query_limits_test.go
  • internal/ingest/compact_test.go
  • internal/settings/validate_test.go
  • internal/api/bufpool_test.go
  • internal/api/policy_test.go
  • internal/query/builder_test.go
  • internal/api/boot_chain_test.go
  • internal/ingest/types_test.go
  • internal/api/structured_query_test.go
  • tests/integration/dlq_test.go
  • internal/ingest/worker_test.go
  • internal/discovery/discovery_test.go
  • internal/api/ingest_test.go
  • internal/stream/hub_test.go
  • internal/stream/roweval_test.go
  • internal/policy/rowfilter_test.go
  • internal/policy/policy_test.go
Exactly one runtime dependency — `eventsource-parser` (SSE framing, itself dependency-free); adding a second needs the same scrutiny the first got.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • clients/ts/src/table.ts
  • tests/e2e/sdk/admin.test.ts
  • tests/e2e/sdk/settings.ts
  • tests/e2e/sdk/setup.ts
  • clients/ts/src/index.ts
  • tests/e2e/sdk/query.test.ts
  • clients/ts/src/stream/sse.ts
  • tests/e2e/sdk/ingest.test.ts
  • clients/ts/src/stream/sse.test.ts
  • tests/e2e/sdk/streaming.test.ts
  • clients/ts/src/types.ts
**In MDX, leave a blank line between a JSX tag and a code fence.**

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • docs/src/content/docs/access-control.mdx
  • docs/src/content/docs/settings-directory.mdx
**Never hard-wrap prose. One paragraph is one line.** No wrapping at 72/80 columns, no "semantic linefeeds" splitting a paragraph at sentence boundaries.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • docs/src/content/docs/sdk/streaming.md
  • docs/src/content/docs/sdk/admin.md
  • docs/src/content/docs/sdk/queries.md
  • docs/src/content/docs/ingest-pipeline.md
  • docs/src/content/docs/architecture.md
  • docs/src/content/docs/api.md
  • CHANGELOG.md
🧠 Learnings (7)
📚 Learning: 2026-06-26T12:23:22.696Z
Learnt from: EricAndrechek
Repo: Wave-RF/WaveHouse PR: 346
File: internal/stream/subscriber_test.go:9-28
Timestamp: 2026-06-26T12:23:22.696Z
Learning: In this Go repository, prefer table-driven tests (e.g., `[]struct{...}` with `t.Run(...)`) only for tests that cover multiple scenarios/inputs and can be cleanly enumerated. Do not artificially rewrite a clear single-scenario sequential behavioral-flow test into a table-driven form just to fit the pattern; if there’s only one meaningful scenario, keep the test as a straightforward linear flow (as in `TestSubscriber_SendDeliversThenDropsWhenFull`).

Applied to files:

  • internal/stream/filter_test.go
  • internal/settings/validate_test.go
  • internal/discovery/discovery_test.go
  • internal/api/ingest_test.go
  • internal/stream/roweval_test.go
📚 Learning: 2026-05-23T01:23:59.268Z
Learnt from: EricAndrechek
Repo: Wave-RF/WaveHouse PR: 174
File: internal/api/ingest_test.go:111-111
Timestamp: 2026-05-23T01:23:59.268Z
Learning: In WaveHouse Go tests in internal/api/**/*_test.go, use internal/testutil.AssertJSONErrorResponse(t, w) for HTTP error-path JSON assertions. Do not use (or reintroduce) package-local assertJSONErrorResponse helpers. AssertJSONErrorResponse verifies the response Content-Type is application/json, includes the X-Content-Type-Options: nosniff header, and that the JSON body contains an "error" field.

Applied to files:

  • internal/api/ingest_seams_test.go
  • internal/api/ingest_test.go
📚 Learning: 2026-08-19T15:44:27.183Z
Learnt from: taitelee
Repo: Wave-RF/WaveHouse PR: 500
File: internal/settings/settings.go:31-31
Timestamp: 2026-08-19T15:44:27.183Z
Learning: In the WaveHouse Go codebase, do not flag package-level lookup tables or precomputed stateless values when they are immutable and read-only, including settings.Files, validate.pipeParamTypes, mutationVerbs, nonMutationVerbs, identEscaper, intBounds, and package-level regular expressions. The no-global-state guideline applies to injected application dependencies and mutable singletons, not immutable lookup data.

Applied to files:

  • internal/settings/validate.go
  • internal/api/record_reader.go
📚 Learning: 2026-08-13T12:17:52.620Z
Learnt from: EricAndrechek
Repo: Wave-RF/WaveHouse PR: 470
File: docs/src/content/docs/reverse-proxy.mdx:137-144
Timestamp: 2026-08-13T12:17:52.620Z
Learning: For Wave-RF/WaveHouse documentation, verify claims about implementation control flow against the authoritative implementation source (for example, internal/auth/auth.go) rather than relying solely on docs/** content. Documentation may lag behind or paraphrase behavior, so control-flow claims should be confirmed in source code.

Applied to files:

  • docs/src/content/docs/access-control.mdx
📚 Learning: 2026-08-11T21:55:53.726Z
Learnt from: jfwoods
Repo: Wave-RF/WaveHouse PR: 434
File: clients/go/stream_test.go:159-176
Timestamp: 2026-08-11T21:55:53.726Z
Learning: For Go files in this repository, do not report direct type assertions solely because the `forcetypeassert` rule is commented out in `.golangci.yml`. Only flag a type assertion when there is an independent correctness, safety, or maintainability issue.

Applied to files:

  • internal/discovery/discovery_test.go
📚 Learning: 2026-05-20T01:02:00.784Z
Learnt from: EricAndrechek
Repo: Wave-RF/WaveHouse PR: 164
File: internal/api/router_test.go:289-350
Timestamp: 2026-05-20T01:02:00.784Z
Learning: In WaveHouse’s internal API tests (files matching internal/api/**/*_test.go), follow the existing separation-of-concerns convention for testing the RequireRole middleware: inject `ContextKeyRole` directly into the request `context.Context` instead of using `testutil.MakeJWT`/JWT-driven flows. Do not refactor role-gate tests to use JWT tokens—JWT parsing and token handling are covered separately in `middleware_test.go` (the dedicated JWT parsing tests), and mixing those concerns would expand the failure surface and reduce isolation.

Applied to files:

  • internal/api/ingest_test.go
📚 Learning: 2026-06-10T15:01:09.027Z
Learnt from: EricAndrechek
Repo: Wave-RF/WaveHouse PR: 312
File: docs/src/content/docs/development.md:0-0
Timestamp: 2026-06-10T15:01:09.027Z
Learning: In this repo’s Markdown review (all .md files), do not flag capitalization/style issues for literal paths starting with ".github/" (or any substring that is a path beginning with ".github/"). Treat ".github" as the correct lowercase dotfile directory name, even when it appears inside prose or code spans; automated checks such as LanguageTool’s "(GITHUB)" rule commonly produce false positives for this literal filesystem path.

Applied to files:

  • CHANGELOG.md
🪛 LanguageTool
docs/src/content/docs/architecture.md

[typographical] ~128-~128: Consider using an em dash in dialogues and enumerations.
Context: - worker.go — StartIngestWorker lau...

(DASH_RULE)

docs/src/content/docs/api.md

[style] ~600-~600: Consider using “who” when you are referring to a person instead of an object.
Context: ... the schema-versioning work. A consumer that drops a row whose length disagrees with...

(THAT_WHO)

CHANGELOG.md

[typographical] ~25-~25: Consider using an em dash in dialogues and enumerations.
Context: - **SSE: the column list is announced as ...

(DASH_RULE)


[style] ~25-~25: Since ownership is already implied, this phrasing may be redundant.
Context: ...lay follows the identical contract with its own drift state, because it writes straight...

(PRP_OWN)


[style] ~25-~25: This sentence is over 40 words long. Consider splitting it up, as shorter sentences make the text easier to read.
Context: ...ad of it with nothing to zip against. Known limitation, deferred to the schema-versioning work: those two states are not reconciled when they disagree, so if a table's column set changes while a client is connected and that client gap-fill-replays across the change, live rows arriving after the replay may carry no fresh schema frame until the next drift or a reconnect. Rows are not mislabeled — the SDK drops...

(TOO_LONG_SENTENCE)

🔇 Additional comments (37)
clients/ts/src/types.ts (1)

344-357: LGTM!

Also applies to: 381-386

internal/api/policy_test.go (1)

35-35: LGTM!

Also applies to: 60-60, 97-97, 138-138

internal/api/structured_query_test.go (1)

160-160: LGTM!

Also applies to: 186-186, 214-217, 302-305, 317-317, 340-340, 366-366, 381-381, 397-397, 412-412, 458-458, 477-477, 493-493, 516-519

internal/auth/auth_test.go (1)

128-133: LGTM!

Also applies to: 162-166

docs/src/content/docs/access-control.mdx (1)

105-117: LGTM!

Also applies to: 126-170, 178-179, 201-202, 250-251, 281-287, 316-317, 414-415, 427-428, 454-454

docs/src/content/docs/sdk/admin.md (1)

41-42: LGTM!

docs/src/content/docs/settings-directory.mdx (1)

46-46: LGTM!

Also applies to: 60-69

tests/e2e/sdk/admin.test.ts (1)

77-86: LGTM!

clients/ts/src/index.ts (1)

36-36: LGTM!

Also applies to: 53-53

internal/settings/validate_test.go (1)

28-28: LGTM!

Also applies to: 221-221, 248-250, 328-334, 365-373, 388-442, 469-469

internal/api/errors_test.go (1)

167-167: LGTM!

tests/e2e/sdk/settings.ts (1)

56-59: LGTM!

tests/e2e/sdk/setup.ts (1)

75-87: LGTM!

tests/integration/rowfilter_narrowing_test.go (1)

203-203: LGTM!

internal/api/structured_query.go (1)

185-186: LGTM!

Also applies to: 200-204

tests/e2e/sdk/query.test.ts (1)

193-193: LGTM!

Also applies to: 224-224, 267-269, 307-310, 438-440

deployments/compose/settings/policies.json (1)

5-25: LGTM!

internal/policy/policy.go (2)

197-215: LGTM!


533-536: 🩺 Stability & Availability

No action is needed for IsColumnAllowed call-site selection. Query and stream paths resolve "select" and pass false; ingest resolves "insert" and passes true.

internal/policy/rowfilter.go (1)

15-15: LGTM!

Also applies to: 121-121, 131-131

internal/policy/policy_test.go (1)

203-243: LGTM!

internal/policy/rowfilter_test.go (1)

25-25: LGTM!

Also applies to: 350-351

internal/query/builder.go (1)

108-110: LGTM!

Also applies to: 148-149, 192-192

internal/query/builder_test.go (1)

184-184: LGTM!

Also applies to: 287-287, 681-681, 710-710, 743-763, 788-788, 802-802, 819-819

internal/settings/validate.go (2)

601-620: LGTM!


299-313: 🩺 Stability & Availability

No change required. internal/settings/validate.go already imports bytes, which resolves bytes.TrimSpace.

tests/integration/query_limits_test.go (1)

56-63: LGTM!

Also applies to: 73-73, 83-83, 98-98

tests/e2e/sdk/ingest.test.ts (1)

182-182: LGTM!

Also applies to: 236-236, 281-291, 293-314, 327-331, 394-396

internal/api/stream.go (1)

90-105: LGTM!

Also applies to: 122-133

internal/stream/hub.go (1)

7-7: LGTM!

Also applies to: 33-65, 168-168, 178-229, 235-238, 302-363, 365-367, 406-450, 463-491, 493-570, 576-578, 583-598, 602-623, 628-633

internal/stream/hub_test.go (1)

7-10: LGTM!

Also applies to: 52-77, 119-169, 181-185, 200-201, 216-219, 229-236, 247-269, 280-281, 295-299, 321-324, 351-364, 379-379, 396-405, 419-420, 456-457, 480-480, 493-494, 520-520, 532-533, 545-546, 558-558, 650-657, 676-676, 696-714, 730-750, 833-833, 848-853, 874-878, 889-890, 925-932, 985-985, 997-997, 1074-1290

internal/stream/metrics.go (1)

17-17: LGTM!

internal/stream/roweval_test.go (1)

1-111: LGTM!

internal/stream/subscriber.go (1)

3-3: LGTM!

Also applies to: 18-18, 52-88

docs/src/content/docs/sdk/streaming.md (1)

100-101: LGTM!

Also applies to: 120-121

internal/stream/filter_test.go (1)

10-98: LGTM!

tests/e2e/sdk/streaming.test.ts (1)

34-63: LGTM!

Comment thread clients/ts/src/stream/sse.ts
Comment thread docs/src/content/docs/architecture.md
Comment thread internal/api/ingest_seams_test.go
Comment thread internal/ingest/worker.go
Comment thread internal/policy/policy_test.go
Comment thread internal/policy/policy.go
Comment thread internal/settings/validate.go
Comment thread tests/integration/dlq_test.go
@EricAndrechek

Copy link
Copy Markdown
Member Author

Superseded by a seven-PR stack. Same work, same final tree, split so each piece is reviewable on its own.

Merge top to bottom. Each PR is based on the one above it, so GitHub shows only that layer's diff.

# PR What it is
0 #549 classify-paths.sh reads a grep error as "no match" — unrelated CI fix, at the base so the rest inherit a reliable gate
1 #550 Schema discovery widened: DDL, position, default_expression, server version
2 #551 BREAKING — policy format v2: role-first policies.json, split select/insert permission types
3 #552 BREAKING — ingest requires a declared Content-Type, and honors it
4 #553 Whole-body buffering, the type-layer seams, and the positional encoder
5 #554 BREAKING — positional rows on NATS and SSE, columns announced separately
6 #555 BREAKING — computed columns excluded from the insert column list

Why the split

This PR reached 35 commits across six unrelated subsystems. The reviewers kept finding real defects in it — a blocker that broke ingest for any table with a MATERIALIZED column, a policy check that silently didn't apply, a row that could be delivered without its schema announcement — and each round was harder to reason about than the last because the diff spanned everything at once.

What changed while splitting

main moved three times underneath the work. Two of those mattered:

Review state

Both pre-push reviewers returned ship_it with zero findings at any severity on the stack tip, whose delta against main is the union of all seven branches. make ci is green on each branch's exact tree.

Follow-ups filed along the way

Note on CI

ci.yml triggers on pull_request: branches: [main], so only the PR currently based on main gets a CI run. As each merges, GitHub retargets the next to main and its CI runs then. That is why the stack must merge in order.

Closing in favour of the stack. Nothing here is lost — #555's tree is the finished state.

@github-project-automation github-project-automation Bot moved this from In review to Done in WaveHouse Task Board Sep 2, 2026
github-merge-queue Bot pushed a commit that referenced this pull request Sep 4, 2026
## Summary

`POST /v1/ingest` used to sniff the body and treat the header as a hint:
the
first non-whitespace byte chose between a single object and an array,
and an
NDJSON body whose first line happened to start with `[` was re-framed as
a JSON
array — silently, as a whole-request reinterpretation rather than a
per-record
error.

The header is now required and authoritative. A request declaring
nothing, or a
media type not in the accepted list, is `415` before the body is parsed,
with the
supported types named in the body. The declared type chooses the format
*family* — `application/json` versus the four NDJSON spellings — and
within the
JSON family the first non-whitespace byte still picks array versus
single
object. The bytes never choose the family, so an NDJSON body is read as
NDJSON
whatever its first byte and a bad line fails as a per-record error.

The header is parsed by `mime.ParseMediaType`, so the grammar is RFC
9110's
§8.3 `media-type` rather than one of ours. Only the media type selects
the
format; parameters are ignored, so no malformed parameter costs the
request —
`; charset`, `;;`, a value left mid-quote, even a name repeated with
different
values all read as `application/json`. One exception, below: a malformed
parameter on a line that also contains a comma.

**Duplicate declarations are the part most worth reviewing.**
`Content-Type` is
a *singleton* field (§8.3), and §5.3 forbids repeating a field line
unless the
field allows comma-list recombination — `media-type` does not. So both
duplicate
spellings are malformed input, and §8.3 says so directly, warning that
recipients who resolve the resulting pseudo-list "by using the last
syntactically valid member" cause "interoperability and security
issues".

Ingest therefore takes no member:

- **Repeated header lines** (what curl and many proxies send) are all
resolved
  and must agree on the format *and* on whether ingest reads it at all.
`application/x-ndjson` alongside `application/ndjson; charset=utf-8`
reads as
NDJSON, because once they agree, which one gets honored stops mattering.
Disagreement is `415` rather than resolution to the first — honoring the
first
would let an NDJSON body be read as one JSON object, ingesting record
one and
  discarding the rest behind a `200`.
- **A comma-joined value** (what a proxy merging duplicates sends) is
refused.
Where it fails outright there is no media type to take. Where it does
parse a
media type — `application/json; charset=utf-8, application/x-ndjson`
yields
`application/json` — it is still refused, because the comma may be a
second
declaration joined on and the error cannot distinguish that from a comma
  inside data.

The security-critical detail is in `ingestFormatOne`. `ParseMediaType`
returns
`ErrInvalidMediaParameter` both for a merely-malformed parameter *and*
when a
second declaration was comma-joined on after a parameter —
`application/json;
charset=utf-8, application/x-ndjson` yields mediatype
`application/json`.
Tolerating that error unconditionally silently resolves a joined
disagreement to
its first member, which is exactly the truncation above. The two are
indistinguishable from the error alone, so a comma on a line that did
not parse
cleanly is refused. That guard fails closed, and it is where I would
look first.

**Four additional shapes 0.1.0 accepted now `415`** (beyond repeated
header lines
that disagree, which it also accepted; see the CHANGELOG): a
present-but-empty
header; a value with a leading or trailing comma; a comma-joined value
that does
not parse as a single media type; and a malformed parameter on a line
that also
carries a comma. The last is an over-rejection and is tracked in #563.

Note a comma is not disqualifying on its own: inside a quoted parameter
value it
is legal data, so `application/json; a=", application/x-ndjson; b="` is
one media
type and is accepted, even though an intermediary may have built it by
illegally
joining two lines. The server cannot tell.

The TS SDK sends `application/json` for a single object and
`application/x-ndjson` for arrays and `insertNDJSON`, on every path, so
no SDK
caller is affected. A hand-rolled client that relied on sniffing must
now
declare the type.

## Stacked PR

This is **part 3 of 7** (parts 0, 1 and 2 merged) in a stack that
replaces #540. Each PR is based on the one above it, so review this PR's
own diff against its base — GitHub shows only this layer's changes.

| # | Branch | Base | |
| - | ------ | ---- | - |
| 0 | `stack/0-classify-paths` | `main` | ✅ merged as #549 |
| 1 | `stack/1-discovery` | `main` | ✅ merged as #550 |
| 2 | `stack/2-policy` | `main` | ✅ merged as #551 |
| 3 | `stack/3-content-type` | `main` | **→ this PR** |
| 4 | `stack/4-seams` | `stack/3-content-type` |  |
| 5 | `stack/5-positional-wire` | `stack/4-seams` |  |
| 6 | `stack/6-computed-columns` | `stack/5-positional-wire` |  |

Merge in order, top to bottom. Rebasing or squashing out of order will
make the later PRs' diffs unreadable.

## Response-size bound

The 415 echoes what the caller declared, and it is decided **before the
body is read** — so a header-only request, needing no credentials under
the shipped compose policy, could size the response. Measured before the
fix: 1.03 MB of headers produced a 4.65 MB body (and far worse over
HTTP/2, where HPACK indexes a repeated header line to about a byte on
the wire).

Both dimensions are caller-controlled and both are bounded now: at most
**four distinct** declarations, each capped at 128 bytes, then `"…and N
more"`. The declaration that actually disagreed is pinned into the echo,
so four agreeing spellings cannot crowd it out. Pinned by tests that
fail if either bound is removed.

## Test plan

- [x] `make ci` green locally on this branch's exact tree (verify, unit,
integration against live ClickHouse, e2e, all coverage gates). Now that
#551 has merged and this PR's base is `main`, GitHub CI runs on
`pull_request` automatically — earlier runs on this branch were manual
dispatches, so check the SHA before relying on an old one.
- [x] The branch descends from its base and carries only this layer's
change (plus any follow-up commits answering review)
- [x] `go.mod` / `go.sum` untouched; no new dependencies

## Review

Both pre-push reviewers ran against this branch — no logged skip. They
gated every push, over several rounds; the last rounds reviewed the
`mime.ParseMediaType` rewrite as a fresh change rather than as a delta,
since the earlier approvals covered code the rewrite deleted.

Reviewer findings that shaped the result, rather than only polishing it:

- The first cut of the `ErrInvalidMediaParameter` guard tolerated the
error unconditionally, which reintroduced the silent-truncation hazard
the agreement rule exists to prevent.
- A fourth (now fifth) tightening was neither listed nor pinned, and the
known-limit test could not have caught a regression in it.
- "A duplicate parameter name is a 415" was false: Go only errors when
the values differ.
- The docs' RFC delegation was falsifiable by their own example with its
closing quote dropped.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_018Epn88jTEw4ZkXrvTKzZXQ

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/api HTTP handlers, routing, middleware area/docs Documentation, site/, README area/infra CI, build, deploy, Docker, release area/ingest Ingest pipeline (Bento, batching, DLQ) area/observability Metrics, logs, traces, health, profiling area/policy Access control policies (Hasura-style) area/query Structured query AST, SQL builder area/sdk TypeScript SDK (clients/ts/) documentation Improvements or additions to documentation go Pull requests that update go code

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

1 participant