Skip to content

Commit 862f12c

Browse files
fix(driver-turso)!: the remote filter compiler refuses the JSON-column family and answers $contains by membership (#21178) (#21208)
Fixes #21178 Clause-②: yes (narrowing) ## What changes `RemoteTransport.buildWhereSQL` (`@objectstack/driver-turso`, the filter compiler every remote-mode `TursoDriver` read and filtered write uses) now applies the JSON-column half of the filter contract exactly as the local face (`SqlDriver`, which local and replica mode inherit) does, from the same shared home in `@objectstack/core`: - **Refusal.** On a field the driver stores as a JSON TEXT column, every operator in `JSON_COLUMN_INCOMPATIBLE_OPERATORS` is refused with `INVALID_FILTER` / 400 before any statement runs: in an operator map (at the top of the per-operator loop, ahead of every arm), in the bare `{ field: value }` spelling, and in the bare `{ field: null }` spelling. The message and the withheld diagnostic are `jsonColumnOperatorRefusalText`'s output, byte for byte the local face's, through this transport's existing withheld-refusal seam (#8220 provenance, diagnostic sink). - **Membership.** `$contains` / `$notContains` on such a column answer membership through `jsonMembershipPredicate('sqlite', ...)` (libSQL is SQLite); the negated form sits inside `nullSafeNegative`, so a row with no value satisfies `$notContains`. A scalar string column keeps the substring test. - ⛔ No copy of the set, the sentence or the construct in `driver-turso`; no remote-only dialect. ### The widening (for the contract review) The transport keeps no schema, so it learns "is this a JSON-stored column" the way it learns every other declared fact, through an injected resolver. This adds ONE optional public method on the exported `RemoteTransport` class, and one type: - `RemoteTransport.setJsonColumnResolver(resolver: JsonColumnResolver): void` - `type JsonColumnResolver`: a function taking `(object: string, field: string)` and returning `boolean`, exported from `remote-transport.ts` and NOT re-exported from `index.ts`, like its siblings `NonTextColumnResolver` and `DeclaredValueShapeResolver`; it reaches the published `.d.ts` only as the method's parameter type. `TursoDriver`'s constructor wires it to the inherited `SqlDriver.isJsonColumn`, beside `setDeclaredValueShapeResolver` (constructor wiring only in `turso-driver.ts`; the `upsert` regions are untouched). In remote mode `registerRemoteFieldMetadata` calls `registerExternalObject`, which fills the same `jsonFields` registry the local face's gate and membership reading ask, so both faces read one population. A `RemoteTransport` driven standalone without the resolver treats no column as JSON-stored and compiles as before. This is the fourth sibling of `setFilterColumnSql`, `setNonTextColumnResolver` and `setDeclaredValueShapeResolver`; the last of those shipped as "New optional API" in commit `fb386074`'s changeset. The seat's answer to the fork is on the card (claim amendment 5935032674, option A, open to the maintainer's veto). ## Why (measured at base `0b12b9ea`, on the libsql SQLite stub harness) Over a `multiple: true` lookup holding `["u1","u2"]` r1, `["u2"]` r2, `["u3","u1"]` r3, `["u10"]` r4 and null r5, the remote face answered, while the local face answered the right-hand column: | filter | remote, before | local (the contract) | |:--|:--|:--| | `$contains: 'u1'` | r1, r3, r4 (u10 by substring) | r1, r3 | | `$notContains: 'u1'` | r2, r5 | r2, r4, r5 | | `$nin: ['u1']`, `$ne: 'u1'` | all five rows (fail-open) | `INVALID_FILTER` / 400 | | `$eq`, `$in`, bare equality | no row | `INVALID_FILTER` / 400 | | `$lt` / `$lte` `'u1'` | r1-r4 (lexicographic over the serialization) | `INVALID_FILTER` / 400 | | `$startsWith: '['`, `$endsWith: ']'` | r1-r4 | `INVALID_FILTER` / 400 | | `$icontains: 'u1'` | r1, r3, r4 | `INVALID_FILTER` / 400 | | `{ owners: null }` | r5 | `INVALID_FILTER` / 400 | | `json` field, `$contains: 'u1'` | r1, r4 (text inside the object) | no row (array-only membership) | ## Pins - `turso-local-remote-json-column-parity.test.ts` (new): one fixture on BOTH faces; every case asserts remote equals local AND the canonical answer. It iterates the refused set from `JSON_COLUMN_INCOMPATIBLE_OPERATORS` as it stands (27 members at base), asserting `code` + `status` + equality with `jsonColumnOperatorRefusalText(...).message` (never the literal words); the bare infix spellings in an operator map stay refused on both faces. Also: bare equality, `null` equality (`{f: null}`, `$eq: null`, `$ne: null`), the gate under `$and` / `$or` / `$not`, the population (a `tags` field and a `json` field), `count()`; membership `u1` vs `["u10"]`, `u10`, `$notContains` complement with the NULL row, `tags`, the `json` object, bind alignment beside sibling predicates, `count()`; the scalar text-field control (`$contains`, `$nin`, `$eq`, bare and null equality, `$startsWith` unchanged); `$null` / `$exists` still answered. - `remote-transport-compile-refusal-seam.test.ts`: the enumeration requires every seam refusal method to have a row, so `jsonColumnOperator` gets one per position (operator map, bare value, bare null) on a half-2 transport told exactly one column is JSON-stored. Policy / author / unmarked / merged-arm provenance all hold. ## Ablation (one-time proofs, via `scripts/ablation-replace.mjs`, restore proven by blob hash and empty `git diff HEAD`) The suite imports `./turso-driver.js` from source (vitest), so no `dist/` leg applies. - **Unwire the resolver** in `turso-driver.ts` (`this.remoteTransport.setJsonColumnResolver(` replaced by a no-op call; anchor 1 to 0, blob `a9affc72` to `b90200ef`): the parity suite goes **24 failed / 19 passed** of 43, as predicted. Red: all 14 operator-map refusals, bare equality, depth, population, both `count()` pins, `u1` membership, `$notContains`, the `json` object, bind alignment, null equality. Still green, as predicted: the 13 bare-infix rows (both faces refuse as an object comparand whatever the gate), the set check, `u10` and `tags` membership (substring coincides), the scalar controls, presence. - **Disable the membership arm** in `remote-transport.ts` (`pushJsonMembership` answers false): **5 failed / 38 passed**, exactly the five membership pins whose substring answer differs; every refusal pin stays green. - Restored: `git diff HEAD` empty, blob equals HEAD in both legs. ## Filter-semantics compile surfaces, one conclusion per face 1. `driver-sql` `applyFilterCondition` (`sql-driver.ts`): **already conformant** — it is the contract; the parity suite's local column is its answer, and `driver-sqlite-wasm` and local/replica `driver-turso` inherit it. Not edited. 2. `driver-turso` `RemoteTransport.buildWhereSQL` (`remote-transport.ts`): **changed** (this PR). 3. `service-analytics` `compileScopedFilterToSql` (`read-scope-sql.ts`): **out of scope** — another face; it already reaches the shared membership construct through `contains-membership-sql.ts` (imports `jsonMembershipPredicate`). 4. `service-analytics` `lowerAnalyticsWhere` (`strategies/filter-normalizer.ts`): **out of scope** — another face, same shared construct as 3. 5. `formula` `matchesFilterCondition` (`matches-filter.ts`): **out of scope** — another face, and seat 2's in-flight #20822 group 3b owns it. 6. `objectql` `applyHaving` / `matchesHaving` (`having-filter.ts`): **out of scope** — another face (it already imports the shared set and sentence), and seat 2's #20822 group 3b owns it. 7. `driver-memory` query path (refuses the family since `45ce12a4`, `filter-refusal.ts` imports the shared set) and `driver-mongodb` `translateFieldOperators` (`mongodb-filter.ts`): **out of scope** — other faces, not touched. ## Tests and gates (all on HEAD `c67a136e` unless noted) - `pnpm --filter @objectstack/driver-turso exec vitest run --maxWorkers=2`: 84 files passed, 2286 tests passed, 33 skipped (exit 0, via `os-verify-lock`). - `pnpm --filter @objectstack/driver-turso typecheck`: exit 0; `tsc --listFiles` includes both edited test files. - `pnpm check:driver-conformance`: before (base `0b12b9ea`) 50 covered / 0 DEBT / 0 exempt; after (`c67a136e`) 50 covered / 0 DEBT / 0 exempt. The ledger did not move. - `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --commands` derived 63 commands at `c67a136e`; all 63 were run and reconciled with `--ran` (each line recording its exit code): 62 run, 1 NOT MEASURED. `pnpm check:dual-build-cjs-loads` exited 3 (PREREQUISITE NOT MET: it reads every package's `dist/`, and a repo-wide build is CI's); narrowed in its place, `require` of `driver-turso`'s CJS build and `import` of its ESM build both load and expose `setJsonColumnResolver`. `check-plugin-teardown-shape --self-test` and `check:lean-entry-closure` first exited 3 on prerequisites (a fixture commit outside the shallow clone; objectql's `dist/`) and exited 0 after fetching that commit and building objectql's closure. - `check-adr-0087-registration --base origin/main`: the changeset reads as declared-breaking with one disposition, `not-required (no-migration-prescription)`; `check-changeset-no-major`: no major bump. - Lint, narrowed to the 4 touched TS files: `eslint --no-inline-config --format json` read 4 files, 0 errors, 0 warnings. All four are in `eslint .`'s population (`--print-config` resolves each). The config never enables type-aware linting (the resolved `parserOptions` are `ecmaVersion` and `sourceType` only, and `eslint.config.mjs` says so in its header), so this diff cannot move a verdict on any untouched file. The full `pnpm lint` is CI's. - Downstream consumers (`cli`, `qa/dogfood`, `runtime`, `service-datasource`) are declared to CI: none references `RemoteTransport`'s API, and the one remote-mode consumer test (`date-bucket-parity-turso`) aggregates without a JSON-field filter. ## Changeset `.changeset/21178-remote-json-column-gate.md`: `@objectstack/driver-turso` `minor`, BREAKING banner, `Clause-②: yes (narrowing)`, ADR-0087 disposition `not-required (no-migration-prescription)`, the new optional API named, and the migration text (write `$contains` for "holds this member", an `$or` of `$contains` for any-of, `$not` around either for the exclusion, `$null` / `$exists` / `$empty` for presence). ## Siblings - #21067 (in flight) owns the shared sentence's words in `json-column-operator-refusal.ts`; this PR edits neither that file nor that branch, and its pins compare against the builder's output, so the reword cannot flip them. Whichever of the two lands second merges `main` and re-runs its pins. - #21185 is ruled to edit the `upsert` regions of `remote-transport.ts` and `turso-driver.ts`; this PR is region-disjoint. Whichever lands second merges `main`. ## Acceptance notes - The local face's gate reads the operator, not the comparand, so on a JSON column it refuses the equality spellings of a null comparand (`{ f: null }`, `$eq: null`, `$ne: null`) although `IS NULL` is a well-formed presence question there; this PR matches it on the remote face (that is the parity invariant), and `$null` / `$exists` / `$empty` remain the presence spellings on both. Noted, not filed: no declared contract says otherwise (`$eq` is a declared member of the set). Carrier: none. - Refusal order differs from the local face only for a doubly-wrong filter: the remote gate runs ahead of each arm's own comparand gate (the remote comparand gate lives inside each arm), so `{ owners: { $in: [{ ... }] } }` reads the JSON-column sentence remotely and the comparand sentence locally. Both are `INVALID_FILTER` / 400. - The bare infix spellings (`=`, `in`, …) in an operator map are refused on both faces as an object comparand, with different wording per face; pre-existing, not JSON-specific. --- _Generated by [Claude Code](https://claude.ai/code/session_017xfMoEjKUuSh2xYB8sCozp)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent ce8a6d2 commit 862f12c

5 files changed

Lines changed: 553 additions & 0 deletions

File tree

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
"@objectstack/driver-turso": minor
3+
---
4+
5+
fix(driver-turso)!: in remote mode, a filter on a declared JSON-stored field is refused with `INVALID_FILTER` / 400 for every operator the local face refuses there, and `$contains` / `$notContains` answer membership instead of a substring of the stored text (#21178)
6+
7+
Clause-②: yes (narrowing)
8+
9+
<!-- adr-0087: not-required (no-migration-prescription) a refusal of a QUERY shape at the remote transport's filter compiler: the operators refused on a JSON-stored column are exactly the ones driver-sql's where refuses there (and so this driver's local transport), read from the one JSON_COLUMN_INCOMPATIBLE_OPERATORS set @objectstack/core holds, and $contains / $notContains move from a substring test to the membership test the local transport already answers, through the same jsonMembershipPredicate. No authorable key, spelling or stored metadata shape moves: FilterConditionSchema and every object, view and dataset definition parse and save as before, and nothing reads or rewrites a stored row. There is nothing for objectstack migrate meta to rewrite, since what changes is which query this transport answers, not what any metadata says; the refusal itself names the spelling to use. The other categories are closed on facts: the bumped package publishes (not unpublished); no ADR-0087 id covers a filter operator on a JSON-stored column and this diff adds none (not registered / already-registered); and the change is runtime behaviour, with no published export or type narrowed or removed — the one interface change is an ADDED optional method, a widening (not runtime-interface-only / type-surface-only). -->
10+
11+
**BREAKING** (`@objectstack/driver-turso`, remote mode): this narrows what `TursoDriver` answers when its `url` is a remote libSQL endpoint (such as `libsql://` or `https://`), the transport every hosted tenant database runs on, for every door that compiles a `where`: `find`, `findOne`, `count`, `updateMany`, `deleteMany`, `aggregate` and distinct values. It ships as `minor` under the launch-window convention for accept-set narrowings. Local and embedded-replica mode inherit `driver-sql`'s compiler and already answered this way; nothing moves there.
12+
13+
**What is refused.** On a field the object declares JSON-stored (a structured-JSON type such as `json` or `address`, an inherently multi-value option type such as `tags`, `multiselect` or `checkboxes`, or a `select`, `radio`, `lookup`, `user`, `file` or `image` field declared `multiple: true`), a `where` that aims any operator in `@objectstack/core`'s `JSON_COLUMN_INCOMPATIBLE_OPERATORS` at the field is refused with `INVALID_FILTER` / 400, at any depth under `$and` / `$or` / `$not`, before any statement runs: `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$between`, `$in`, `$nin`, `$startsWith`, `$endsWith`, `$icontains`, `$like`, `$ilike`, and implicit equality (`{ "owners": "u1" }`), whatever the comparand, `null` included.
14+
15+
**What `$contains` / `$notContains` answer now.** Membership: `{ "owners": { "$contains": "u1" } }` matches the rows whose stored list holds `u1` as an element, so it no longer matches a row holding only `u10`; `$notContains` is its exact complement, a row with no value included; and a structured-JSON object answers no member at all, instead of matching text inside its serialization. On a scalar text field both remain the substring test they were.
16+
17+
**What an author sees now.** The body the local transport answers for the same filter, byte for byte: the filter WAS NOT APPLIED, the comparison can never equal one member of a stored list, and the spelling to use, `{ "FIELD": { "$contains": "a" } }` for membership, or an `$or` of `$contains` for any-of. The field and the operator are withheld from the message, and the full diagnostic, naming both, is written to the driver's logger at `warn`.
18+
19+
**Why.** The remote transport compiles its own SQL and read neither the shared refused set nor the membership construct, so over a `multiple: true` lookup holding `["u1","u2"]`, `["u2"]`, `["u3","u1"]` and `["u10"]` it answered: `$nin: ["u1"]` and `$ne: "u1"` every row, the rows holding `u1` included; `$eq`, `$in` and implicit equality no row; `$lt` / `$lte` a lexicographic verdict over the serialization; `$startsWith: "["` and `$endsWith: "]"` every row with a value; `$contains: "u1"` the row holding only `u10` too. One driver gave two answers to one filter depending only on the connection string, and the exclusion operators failed open.
20+
21+
**Who is affected.** A caller, saved filter, list view, report or read scope that reaches a remote-mode `TursoDriver` with one of those operators on a JSON-stored field and read the rows it got as the answer. Write `$contains` for "holds this member", an `$or` of `$contains` for "holds any of these", `$not` around either for the exclusion, and `$null` / `$exists` / `$empty` for presence.
22+
23+
**New optional API.** `RemoteTransport.setJsonColumnResolver(resolver)` in `@objectstack/driver-turso`, which `TursoDriver` wires to its own `isJsonColumn`, beside `setDeclaredValueShapeResolver`. A `RemoteTransport` driven standalone without it treats no column as JSON-stored and compiles as before.
24+
25+
**Unchanged.** `$contains` and `$notContains` on a scalar field, `$exists`, `$null` and `$empty`; every operator on a field that is not declared JSON-stored; and a table this driver holds no declaration for, where nothing is judged.

‎packages/drivers/driver-turso/src/remote-transport-compile-refusal-seam.test.ts‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ import { readFileSync } from 'node:fs';
3838
import ts from 'typescript';
3939
import type { DriverQuery } from '@objectstack/spec/contracts';
4040
import { lowerFilterCondition, markFilterSubtreeProvenance } from '@objectstack/spec/data';
41+
import { jsonColumnOperatorRefusalText } from '@objectstack/core';
4142
import { RemoteTransport } from './remote-transport.js';
4243
import { TursoDriver } from './turso-driver.js';
4344
import { asLibsqlClient, makeLibsqlSqliteStub, type LibsqlSqliteStub } from './libsql-sqlite-stub.testkit.js';
@@ -51,6 +52,11 @@ const POLICY_COL = 'secret_policy_col';
5152
const SECRET = 'PSECRET_LITERAL';
5253
const SECRET_NUM = 7770123;
5354
const UNDECLARED_KEY = '$psecret_combinator';
55+
/**
56+
* [#21178] The one column the half-2 transport is told is stored as JSON — the
57+
* JSON-column gate's refusal needs the driver's rule injected to be reachable.
58+
*/
59+
const POLICY_JSON_COL = 'secret_policy_json_col';
5460

5561
type Door = {
5662
/** The `RemoteTransport` method this row drives — asserted by the error's stack. */
@@ -171,6 +177,16 @@ const DOORS: readonly Door[] = [
171177
secrets: [POLICY_COL, SECRET],
172178
klass: 'a value this transport cannot bind',
173179
},
180+
// ── #21178: the JSON-column gate, born in the seam ─────────────────────────
181+
{
182+
// The class is read from the shared builder, never spelled here: its words
183+
// belong to `@objectstack/core`, and the operator it names in prose
184+
// (`$nin`) is the class, not a secret — the FIELD is what is withheld.
185+
builder: 'jsonColumnOperator',
186+
where: () => ({ [POLICY_JSON_COL]: { $nin: [SECRET] } }),
187+
secrets: [POLICY_JSON_COL],
188+
klass: jsonColumnOperatorRefusalText(POLICY_JSON_COL, '$nin', false).message,
189+
},
174190
];
175191

176192
/**
@@ -238,6 +254,19 @@ const ARMS: readonly Door[] = [
238254
klass: 'A comparand in this filter is undefined',
239255
label: "{ secret_policy_col: { $in: ['a', undefined] } }",
240256
},
257+
// [#21178] The gate's two bare-equality positions: a value and `null`.
258+
{
259+
builder: 'jsonColumnOperator',
260+
where: () => ({ [POLICY_JSON_COL]: SECRET }),
261+
secrets: [POLICY_JSON_COL],
262+
klass: jsonColumnOperatorRefusalText(POLICY_JSON_COL, '=', true).message,
263+
},
264+
{
265+
builder: 'jsonColumnOperator',
266+
where: () => ({ [POLICY_JSON_COL]: null }),
267+
secrets: [POLICY_JSON_COL],
268+
klass: jsonColumnOperatorRefusalText(POLICY_JSON_COL, '=', true).message,
269+
},
241270
];
242271

243272
// ── Half 1: the enumeration ───────────────────────────────────────────────────
@@ -341,6 +370,9 @@ function transport() {
341370
const t = new RemoteTransport();
342371
t.setClient(client as any);
343372
t.setDiagnosticSink((m) => sink.push(m));
373+
// [#21178] Only `POLICY_JSON_COL` is a JSON column, so every other row
374+
// compiles exactly as it does on a transport handed no rule at all.
375+
t.setJsonColumnResolver((_object, field) => field === POLICY_JSON_COL);
344376
return { t, sink, client };
345377
}
346378

0 commit comments

Comments
 (0)