Skip to content

Commit 8d76c2d

Browse files
fix(driver-sql, driver-turso): every filter-compile refusal goes through the withheld provenance seam, so a read scope refused there names no field or literal (#20039) (#20093)
Fixes #20039 Clause-②: no ## What changed A read scope is the RLS, sharing or tenant predicate that `plugin-security` (ordinary reads) and `service-analytics` (`ObjectQLStrategy.withReadScope`) AND into the caller's `where`. Both merges mark the scope `'policy'` and the caller's own predicate `'author'` (#8220, `packages/spec/src/data/filter-subtree-provenance.ts`). The contract for `'policy'`: "A refusal raised from inside it keeps the #7929 redaction: identity (`INVALID_FILTER` / 400) and capability statement on the wire, operands in the server log." The fail direction: "Unmarked or ambiguous ⇒ withheld." PR #20037 brought four `SqlDriver` doors under that seam. This PR closes the class: **every refusal on `SqlDriver`'s filter-compile path now goes through `withheldFilterError` with the node it was raised from**, and one pin enumerates the builders so that a new one cannot disclose by omission. **The nine `SqlDriver` builders converted** (each of them called `unsupportedFilterError` directly at base): | builder | the node the seam resolves | |:--|:--| | `icontainsComparandError` | the comparand if it is an object, else the field's operator map | | `likePatternComparandError` | the same | | `danglingLikeEscapeError` | the field's operator map | | `unrenderableTextComparandError` | the comparand (an object or array here) | | `unbindableListMemberError` | the member if it is an object, else the list | | `undefinedComparandError` | the nearest object holding the `undefined`: the node carrying the field, the operator map, or the list | | `assertFilterNode` | the element if it is an object, else the `$and` / `$or` list that holds it, or the node carrying `$not` | | `unknownLogicalOperatorError` | the node CARRYING the key, never its value | | `filterArrayReachedDriverError` | the array itself (it is the `where` root). Not on the card: the enumeration found it, and it printed the whole array | The walk's refusals need the node that encloses the refused position. The spec's shared walk hands its hooks only a value, a key and a path, so `sqlFilterVerdictHooks` recovers the node from the path index it already kept for the combinator door (#20020). It adds one map of `$and` / `$or` lists, keyed by the combinator key's position. The spec hook signature is unchanged. Per door, what stays on the wire is the class statement: which kind of refusal fired and the capability sentence. The field, the operator variant (`$like` vs `$ilike`, which of the three list operators, which of the five text operators), the comparand, the list index and the filter path go to the server log. `'author'`-marked subtrees get the text the door answered before, byte for byte. The source templates are the same literals, now passed as the diagnostic. Code and status are unchanged on every path (`INVALID_FILTER` / 400), and the set of refused filters is unchanged. **Turso remote transport.** `RemoteTransport.buildWhereSQL` refuses these classes on its own (measured, see below). Its twins now go through `withheldRefusal`, with the same node rules (`refusalNode` is `refusalSubtree`'s twin): `icontainsComparand`, `likePatternComparand`, `danglingLikeEscape`, `undefinedComparand`, `uncompilableSubFilter` (a non-node `$and` / `$or` element or `$not` operand), `undeclaredCombinator` (both of its tails share one withheld sentence) and `uncompilableWhere` (a non-object top-level `where`). The transport's remaining two refusals that named the field go through too, as a bounded in-place fix within this claim's file surface: `emptyFieldFilter`, whose `driver-sql` twin has withheld since #8197, and `unsupportedOperator`'s `$between` arm. `TursoDriver` never reaches the `$between` arm (it lowers `$between` first). After this, `buildWhereSQL` has no refusal outside the seam. For six classes the withheld sentence is `driver-sql`'s behind the `[RemoteTransport]` prefix, pinned through `TursoDriver` local vs remote. ## Measured: H1–H3 **H1.** The probe (scratch, not committed) ran a real `ObjectQL` over a real `SqlDriver` (better-sqlite3), with a first-registered middleware copying `plugin-security`'s merge and marking. It marks the injected scope `'policy'` and the caller's verbatim `where` `'author'`, then composes `{ $and: [where, scope] }`, or the scope alone when there is no caller `where`. The scope used distinctive names (`policy_stage`, `PSECRET…`). A cell counts as a leak when one of them appears in the response message. | class | base (`b76aad5f6f`) | this branch | |:--|:--|:--| | empty `$icontains` | field | none, field in the log | | non-string `$icontains` | field, comparand | none, both in the log | | non-string `$like` | field, comparand | none, both in the log | | `$like` trailing escape | field, pattern | none, both in the log | | object `$contains` comparand | field, comparand | none, both in the log | | object `$in` member | field, comparand | none, both in the log | | `undefined` comparand (direct, `$eq`, `$in` member) | field | none, field in the log | | non-node element of `$or` / `$not` operand | the literal | none, literal in the log | | undeclared node combinator | the key | none, key in the log | | scope handed over as an array | whole array (field and literal) | none, both in the log | The policy-scope cells are 13 shapes × {with, without} a caller `where`, 26 cells in all. All 26 leaked at base and none leak at this branch. The author arm used the same 13 shapes in the caller's own `where`: 13 of 13 full messages are byte-identical at base and at this branch. (Six of those 13 are refused by the engine's own comparand or `where`-shape door before any middleware runs, and are identical on both sides for that reason.) Base was measured by the whole-file ablation below, which reverted `sql-driver.ts` and rebuilt `dist/`. **H2: the enumeration.** A TypeScript-parser walk over `sql-driver.ts` lists the functions calling `unsupportedFilterError` directly, and the functions calling `withheldFilterError`. - **Base:** 12 direct callers (`grep -c 'unsupportedFilterError('` = 13, including the definition), and 12 seam callers. - **This branch:** 3 direct callers (grep 4): the seam itself, the `'author'` re-issue in `resolveWithheldFilterRefusal`, and `unresolvableFilterColumnError`. That last one is raised from a dialect error after the statement ran, and resolves its own provenance by name (`unresolvableColumnProvenance`, #20020). 21 seam callers (`withheldFilterError(` grep 22). - **Positive control:** the doors PR #20037 converted (`retiredFilterOperatorError`, `unsupportedFilterOperatorError`, `assertFilterNodeList`, `nonBooleanNullComparandError`, `nonBooleanExistsComparandError`) are in the base seam set, and its column door is in the direct set. - **The eighth door (H1's PM-grep guess):** `assertFilterNode` is confirmed. The enumeration adds a ninth, `filterArrayReachedDriverError`. `RemoteTransport`: base has 11 direct `invalidFilterError` callers and this branch 2, the seam and the re-issue. The seam callers go from 5 methods to 13. **H3.** Measured on `RemoteTransport` with a `'policy'`-marked `where`. The transport refuses these on its own, naming `'object.field'` and the comparand: `$icontains`, non-string `$like`, trailing escape, `undefined` comparand, non-node `$or` element and `$not` operand, undeclared combinator (including a misplaced field operator), non-object `where` (the whole array), empty operator map, un-lowered `$between`. An object `$contains` comparand and an object `$in` member were already withheld there: both are `uncompilableComparand`, which #8197 converted. Through `TursoDriver` in REMOTE mode, `toRemoteFilter` rebuilds every node, so no mark reaches the transport. Every mark-reading refusal there withholds for every caller, author included. That is pinned as the declared fail-closed cost. **H4: consumer pins.** Two sweeps, over test files in `packages/`, `examples/` and `apps/`: - **Wording grep:** the nine author texts' distinctive fragments and the remote twins'. 24 files, of which 9 are outside the three driver packages: `driver-memory` / `driver-mongodb` (their own wording) and a `packages/spec` test (its own). Control: the re-pinned `driver-sql` / `driver-turso` files are among the 24. - **Assertion-shaped grep** over the test files outside the three drivers that reference a SQL-family driver and `INVALID_FILTER`: 55 files, 17 of which construct a real SQL-family driver. Every hit is `service-analytics`' own normalizer or read-scope compiler wording (`comparand at … is undefined`, `normalizeAnalyticsFilterTree`), `plugin-auth`'s `$regex` face-3 pin (the retired door, unchanged here), or a negative assertion. **No consumer pin outside the three driver packages reads these doors' wording.** None was changed, and the suites PR #20037's round 2 named are green (below). ## Compile surfaces (per `references/compile-surfaces.md`, located again at this head) | # | face | this PR | |:--|:--|:--| | 1 | `driver-sql` `applyFilterCondition` (`sql-driver.ts:15907`); `driver-sqlite-wasm` and Turso local by inheritance | CHANGED: nine builders, and the class is closed | | 2 | Turso `RemoteTransport` `buildWhereSQL` (`remote-transport.ts:2667`) | CHANGED: seven twin classes plus the empty-operator-map and `$between` arms; class closed | | 3 | service-analytics `compileScopedFilterToSql` (`read-scope-sql.ts:582`) | not touched: outside the claim; its refusals are the withheld `READ_SCOPE_COMPILE_FAILED` / 500 | | 4 | service-analytics `lowerAnalyticsWhere` (`filter-normalizer.ts:1966`) | not touched (outside the claim) | | 5 | `formula` `matchesFilterCondition` (`matches-filter.ts:212`) | not touched, not measured | | half | objectql `applyHaving` / `matchesHaving` (`having-filter.ts:279` / `:292`) | not touched, not measured | | — | `driver-memory` / `driver-mongodb` | not touched; they do not read the mark, and disclosure there is not measured | ## Tests **New pins:** - `packages/drivers/driver-sql/src/sql-driver-compile-refusal-seam.test.ts`, 91 cases: - the enumeration (positive control, direct-caller set, seam-caller set equal to the table); - 21 builders × {policy, author, unmarked, merged `$and` in both arm orders}, where the array-root builder has no merged arm. Each row is bound to its builder by the error's own stack; - five node-lookup cases. - `packages/drivers/driver-turso/src/remote-transport-compile-refusal-seam.test.ts`, 96 cases: - the same enumeration over `remote-transport.ts`; - 13 methods plus 7 extra arms under every mark; - two node cases; - `TursoDriver` local vs remote: the withheld sentences of six classes are one sentence, and REMOTE withholds an author-marked `where`. Per door, the pins assert: policy-marked gives `INVALID_FILTER` / 400, the error's own keys exactly `code,status`, no secret in the message, and every secret in the log or sink; author-marked gives the full text, which the policy case logged; unmarked is byte-identical to policy. **Existing pins that read a converted door's full text through an unmarked `where`.** Each now marks it `'author'`, as a merge boundary marks a caller's own predicate, on a shallow copy where the case is a shared constant: - `driver-sql`: `filter-refusal-envelope` (array cases), `not-null-safe`, `out-of-contract-filter-input`, `silent-empty-predicate` (its #7929 comment was updated to say the family is now narrowed too), `undefined-comparand-refusal`. - `driver-turso`: `bare-date-routing`, `boolean-identity`, `node-operator-refusal`, `not-operator`, `top-level-where`, `undefined-comparand-refusal`. - `driver-sqlite-wasm`: `out-of-contract-filter-input`, which also gains an unmarked withheld leg. Two remote shapes cannot get the author text back, and their pins say so: - a PRIMITIVE top-level `where` cannot carry a mark, so it is withheld for everyone and its pin asserts the class statement; - a refusal raised inside `$not` resolves against the NULL-safe REWRITE of the operand, which the resolver cannot find under the root. It is withheld even for an author, fail-closed by construction as `driver-sql` documents for its own `$not` rewrite. Those pins read the naming half from the diagnostic sink. `turso-local-remote-text-parity.test.ts`: a withheld remote answer must still name none of the case's mentions, except the ones the refusal's CLASS statement carries. The class statement is read off the local face's unmarked answer; for the `$icontains` rows it is `$icontains` itself. **Suites.** Driver suites at `8b4f1bb3ed` (after merging `main`, which brought #20054's `turso-driver.ts` change), with `driver-sql` and the driver closures rebuilt: - `driver-sql`: 185 files passed, 11 skipped; 2936 tests passed, 170 skipped. - `driver-turso`: 68 files, 1606 tests passed. - `driver-sqlite-wasm`: 32 files, 606 tests passed. - `typecheck` exits 0 for all three. `tsc --listFiles` counts 1 each for the two new pin files and the changed sqlite-wasm pin. Consumer suites at `ab61f7a832`: the same driver code, before the `main` merge. The merged commits touch no `driver-sql` path, and in `driver-turso` only `turso-driver.ts` (#20054's constructor seam). `driver-sql` and `driver-turso` `dist/` were rebuilt first. - `plugin-auth`: 114 files, 2440 tests passed. - `service-storage`: 40 files, 627 tests passed. - `cloud-connection`: 30 files, 396 tests passed. - `plugin-dev`: 8 files, 80 tests passed. - `connector-mcp`: 3 files, 23 tests passed. - `hono`: 5 files, 122 tests passed. - `cli` unit project (`--project unit`): 224 files, 3158 tests passed. The integration tier is declared to CI; this diff touches no cli file. - `plugin-security`: 133 files, 2648 tests passed. - `plugin-sharing`: 37 files, 913 tests passed. - `service-analytics`: 120 files, 2689 tests passed. **Ablations.** All ran on committed state. Each restore is proved by the blob equal to `HEAD` and an empty `git diff HEAD`. 1. **`sql-driver.ts` at the base blob** (`463b99cf41`; marker 1 → 0), `driver-sql` rebuilt, and `ablation-dist-preflight --absent` passed. The new seam pin gave 41 failed, 50 passed: - red: both enumeration assertions, the 35 policy/author/unmarked/merged cases of the nine builders, and four node cases; - green: the positive control, the 12 already-converted builders × 4, and the author-arm node case, which base satisfies by disclosing to everyone. The five re-pinned `driver-sql` files stayed 110/110 green at base, so the author text they read is unchanged. The same run produced the base column of the H1 table. Restored to blob `a89054ee3a`, rebuilt, and the preflight found the marker in `dist/`. 2. **`remote-transport.ts` at the base blob** (`55148ccc94`): the remote seam pin gave 67 failed, 29 passed. It imports `src`, so no build leg was needed. 3. **The `classifyKey` enclosing node replaced with `undefined`** (`ablation-replace.mjs`, anchor 1 → 0): 3 failed, exactly the unknown-combinator author and merged legs and the direct-`undefined` node case. 4. **The `assertNode` enclosing replaced with `undefined`:** 4 failed, the non-node element's author and merged legs and the two primitive-inheritance node cases. 5. **One builder bypassing the seam** (`icontainsComparandError` calling `unsupportedFilterError`): 6 failed, both enumeration assertions and its four rows. This is the red a new bypassing builder gets. ## Gates - `dispatch-gates --commands --repo objectstack-ai/objectstack` at `8b4f1bb3ed` (merge base `249172975b`, 18 paths) derived 62 families. They are the PM's 55 plus the seven that apply once the changeset exists. - All 62 exited 0. `check:dual-build-cjs-loads` first exited 3 (PREREQUISITE NOT MET), then 0 after a full `turbo run build --filter=./packages/* --filter=./packages/*/*` (72 of 72 tasks cached). The other dist-reading families were re-run on that build and exited 0. - `--ran`: "62 derived, 62 run, 0 NOT-MEASURED, 0 UNRUN (a DERIVED zero)". - The three roster families under `packages/` that the derivation flags as unreadable exited 0: `check:authz-resolver`, `check:error-code-casing` and `check:filter-alias-parity`. - `check-issue-citations --base 2491729`: exit 0, 56 citations resolve. - `check:driver-conformance`: "50 covered cell(s), 0 in the DEBT ledger, 0 exempt" at base (`b76aad5f6f`, read from an archived tree) and at this head. The ledger did not move. - ESLint (`--no-inline-config --format json`) on the 17 changed `.ts` files: 17 files, 0 errors, 0 warnings. `eslint.config.mjs` sets no `parserOptions.project` (the printed config for `sql-driver.ts` is `{"ecmaVersion":"latest","sourceType":"module"}`), so this diff cannot move a verdict on an untouched file. The repo-wide run is CI's. - A control-byte scan of the changed files found no hits. ## Acceptance notes - **What an unmarked caller loses:** its own diagnostic from these nine refusals. It gets the withheld wording with the same code and status. This covers no security plugin, system context, anonymous calls, a `where` holding a `{placeholder}` token, and Turso remote mode. A member's plain `where` under `plugin-security` is marked `'author'` and keeps the full text. The changeset says this. - **The two node-shape refusals inside `$not` on the remote transport** (`uncompilableSubFilter`, `undeclaredCombinator`) resolve against the rewritten operand, so they withhold even for an author-marked `where`. The comparand doors there still answer an author in full. *(Narrowed by the seat from contract review 5826632400.)* PR #20037's `nonListCombinator` has the same limitation inside `$not`. It is fail-closed, and invisible in production, where `TursoDriver` never hands the transport a mark. - **Two findings are reported to the seat and not fixed here.** Both are outside this claim's file surface or class: - `TursoDriver`'s remote `$between` lowering (`toRemoteFieldSpec`) refuses a non-two-element range with a bare `Error` carrying no `code` / `status`. Local mode answers `INVALID_FILTER` / 400. - A second finding, in `RemoteTransport`'s handling of `where` keys, is routed to the maintainer. *(The seat removed its detail from this body before landing, pending the maintainer's decision.)* --- _Generated by [Claude Code](https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent d624002 commit 8d76c2d

18 files changed

Lines changed: 1706 additions & 147 deletions
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
---
2+
'@objectstack/driver-sql': patch
3+
'@objectstack/driver-sqlite-wasm': patch
4+
'@objectstack/driver-turso': patch
5+
---
6+
7+
fix(driver-sql, driver-turso): every filter-compile refusal stops naming a read scope's field or literal unless the refused predicate is marked as the caller's own (#20039)
8+
9+
Clause-②: no
10+
11+
A read scope is the RLS, sharing or tenant predicate that `plugin-security` (ordinary reads) and `service-analytics` (the ObjectQL analytics face) AND into the caller's `where`. Both merges mark the scope `'policy'` and the caller's own predicate `'author'` (`markFilterSubtreeProvenance`, `@objectstack/spec/data`). Nine more `SqlDriver` filter-compile refusals did not check the mark, so when one of them refused a scope, its `INVALID_FILTER` / 400 message named the scope's field, and for most of them its literal too. Measured through an ObjectQL `find` under a merge shaped like `plugin-security`'s, with the scope in the `'policy'` arm:
12+
13+
- an empty or non-string `$icontains` comparand;
14+
- a non-string `$like` / `$ilike` comparand;
15+
- a `$like` / `$ilike` pattern ending in a lone backslash;
16+
- an object or array comparand on `$contains`, `$notContains`, `$startsWith`, `$endsWith` or `$icontains`;
17+
- an `$in` / `$nin` / `$between` member that cannot be bound;
18+
- an `undefined` comparand, in any position;
19+
- an element of `$and` / `$or`, or the operand of `$not`, that is not a filter condition object;
20+
- a `$`-prefixed key in a node position that is not `$and`, `$or` or `$not`;
21+
- a `where` that reaches the driver as an array (the message printed the whole array).
22+
23+
Each of them now reads the mark on the node it was raised from, as the other compile refusals already did:
24+
25+
- **`'policy'`, unmarked or ambiguous:** same `INVALID_FILTER` / 400. The message says which kind of refusal fired, but names no field, operator variant, comparand, list position or filter path. Those go to the server log.
26+
- **`'author'`:** the full message, the same text the refusal answered before.
27+
28+
With these nine, every refusal on `SqlDriver`'s filter-compile path goes through the same seam.
29+
30+
`SqliteWasmDriver` (`@objectstack/driver-sqlite-wasm`) and `TursoDriver` in local mode extend `SqlDriver`, so they inherit this change from it: the same refusals answer the same way there.
31+
32+
**What an unmarked caller loses:** its own diagnostic from these refusals. That is every caller whose predicate reaches the driver unmarked, for example with no security plugin in the stack, in a system-context or anonymous call, or with a `where` that holds a `{placeholder}` token (the engine rewrites it before the merge). That caller gets the withheld wording with the same code and status, and the full text is in the server log. A member's plain `where` under `plugin-security` is marked `'author'` and keeps the full text.
33+
34+
The Turso REMOTE transport (`RemoteTransport`) compiles filters itself. Its copies of these refusals now read the mark the same way: the `$icontains`, `$like` / `$ilike`, lone-backslash, `undefined`, non-node element or operand, undeclared-key and non-object `where` refusals. So do its two other compile refusals that still named the field: an operator map with no operator in it, and a `$between` that reached the transport without being lowered. The operands go to its diagnostic sink. For six of these classes, the withheld sentence is the local one behind the `[RemoteTransport]` prefix. An object text comparand and an unbindable list member already answered there through its comparand refusal, which withholds. `TursoDriver`'s remote mode rebuilds every filter node before the transport sees it, so no mark reaches the transport there, and these refusals keep the withheld wording for every caller in that mode.
35+
36+
Not changed: which filters are refused, and the code and status of every refusal.

0 commit comments

Comments
 (0)