Skip to content

Pipeline is insert-only; all mutations move to raw admin SQL #158

Description

@EricAndrechek

Summary

Lock the ingest pipeline to inserts only. Every mutation other than INSERT — DELETE, UPDATE, TRUNCATE, DROP, ALTER, REPLACE, etc. — must be issued through POST /v1/query, which already gates on the admin/service role (or a policy role with RawSQL: true).

Why

Our policy engine authorizes mutations by evaluating row/column rules and JWT-claim check clauses against the payload of an operation (the columns being written). That model works cleanly for inserts; it does not work for predicate-driven mutations:

  1. We have no way to prove a WHERE predicate is satisfiable only for rows the caller's role is allowed to touch — no row-level filter injection on DML.
  2. Blast radius is unbounded — a single WHERE 1=1 (or a missing WHERE on DROP/TRUNCATE) wipes a table.
  3. Cache-tag invalidation is straightforward for insert and for table-scoped mutations issued through raw SQL (we already extract mutation targets there); a partially-authorized predicate path would muddy that contract.

Rather than ship a partial enforcement story, we narrow the pipeline to the one shape policy can authorize today (insert) and route everything else through the existing admin-only raw SQL gate. The policy contract stays honest — "pipelined endpoints are policy-enforced; raw SQL is admin-only" — and the WHERE-authorization design is deferred without blocking alpha.

Scope — what to remove / change

  • internal/ingest/bento.go: drop the inline action: "delete" branch (DELETE FROM <table> WHERE id = ?) and its associated DLQ delete-envelope shape. Collapse the action check to "anything other than insert/empty → reject".
  • internal/ingest/bento_test.go: remove the delete-action coverage (drain, DLQ routing, unsafe-table, double-ack) — unreachable once the branch is gone.
  • Docs (docs/src/content/docs/architecture.md, api.md): drop the delete-envelope DLQ description and the Wave-DLQ-Type: delete-envelope header note. State explicitly that the ingest pipeline is insert-only and that all other mutations (DELETE/UPDATE/TRUNCATE/DROP/ALTER/REPLACE) require POST /v1/query under an admin-equivalent role.
  • /v1/query mutation-response shape: confirm DDL/DML statements that return no rows from ClickHouse marshal to HTTP 200 with [], not 500 (root cause of bug: TRUNCATE via /v1/query returns HTTP 500 #118). Likely needs the handler to route mutations through Exec rather than Query, or to tolerate a nil/empty ColumnTypes() result.
  • CHANGELOG: breaking-change entry — action: "delete" envelopes on ingest.<table> subjects are no longer honored; the ingest pipeline accepts inserts only. Use POST /v1/query with an admin role for any other mutation.
  • Confirm no other pipelined mutation path exists: /v1/tables/{table}/query AST (internal/query/ast.go), named pipes (internal/pipes), structured query handler. Today these look read-only; verify before closing.

Out of scope (deferred)

Closes

  • Closes bug: TRUNCATE via /v1/query returns HTTP 500 #118 — TRUNCATE via /v1/query returning HTTP 500 is the same class of bug (a mutation statement through the raw-SQL handler not returning a clean success shape). Since /v1/query becomes the only sanctioned surface for non-insert mutations, fixing that response path is part of this work.

Activity

  1. added
    area/apiHTTP handlers, routing, middleware
    area/policyAccess control policies (Hasura-style)
    breaking-changeBreaking change to public API, CLI, or config
    on May 18, 2026
  2. moved this from Backlog to Ready in WaveHouse Task Boardon May 18, 2026
  3. self-assigned this
    on May 18, 2026
  4. changed the title [-]Remove DELETE and UPDATE pipelined API options, only admin SQL[/-] [+]Pipeline is insert-only; all mutations move to raw admin SQL[/+] on May 18, 2026
  5. moved this from Ready to In progress in WaveHouse Task Boardon May 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/apiHTTP handlers, routing, middlewarearea/ingestIngest pipeline (Bento, batching, DLQ)area/policyAccess control policies (Hasura-style)breaking-changeBreaking change to public API, CLI, or configenhancementNew feature or request

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions