Skip to content

Commit f09d412

Browse files
fix(driver-sql, driver-turso): on SQLite, $like / $ilike read the whole stored value past a U+0000 (#20024) (#20140)
Fixes #20024 Clause-②: no Item 2 (ii) of #20024, direction C (landing record 5823785768). On the SQLite faces, a `$like` / `$ilike` pattern without U+0000 over a stored value holding U+0000 now gets the answer `likePatternToRegexSource` gives, which is also what `@objectstack/formula` and `driver-memory` return. Item 1 landed as PR #20038 and item 2 (i) as PR #20124 (#20041). This is the card's last item, so the PR closes it. `patch` for `@objectstack/driver-sql`, `@objectstack/driver-sqlite-wasm` (it inherits the fix) and `@objectstack/driver-turso`. Nothing that was accepted is refused now: the refusal doors, `hasNulInLikePattern` and `hasDanglingLikeEscape` are untouched. The grid, plan and cost numbers below were taken at `b424b90f7c` or earlier. Head `89e02a2521` differs from `b424b90f7c` by one changeset line only, and its emitter code is byte-identical to `7434e6646e`; every commit since then changed changeset prose only. ## The defect, measured at base `fe677aeeed` `glob()` reads the stored value as a C string, only up to its first U+0000. The grid has 59 stored values (27 holding U+0000, leading, interior, trailing, doubled, next to multi-byte and U+0001 characters, plus NULL). It runs 130 queries against them: 81 `$like` and 27 `$ilike` bare patterns, and 22 `$not` / `$or` / `$and` compositions. That is 7670 cells per face, each compared with `formula`. The four faces answered identically: | stored value | cells per face | base: differ from `formula` | head: differ from `formula` | |---|---|---|---| | holds U+0000 (26 values in the Basic Multilingual Plane) | 3380 | 359 | **0** | | no U+0000, in the Basic Multilingual Plane (28 values) | 3640 | 0 | 0 | | NULL | 130 | 0 | 0 | | outside the Basic Multilingual Plane, no U+0000 (3 values) | 390 | 20 | 20 (older than this PR, see H3) | | outside the Basic Multilingual Plane, with U+0000 (1 value) | 130 | 6 (6 against a code-point oracle) | 2 (0 against a code-point oracle) | The four faces are `SqlDriver` on better-sqlite3 (SQLite 3.53.4), `SqliteWasmDriver` on sql.js (3.49.1), `TursoDriver` local, and `TursoDriver` remote over a real `@libsql/client` `file::memory:` engine (3.45.1). At base, 74 of the 108 bare patterns got at least one row holding U+0000 wrong. Some examples: - `$like: 'a'` returned `'a'` + U+0000 + `'b'`; - `$like: ''` returned U+0000 + `'z'`; - `$like: '_'` missed a lone U+0000; - `$ilike: 'A_B'` missed `'a'` + U+0000 + `'b'`. **Base vs head, every stored value without U+0000 and NULL: 0 of 16640 cells moved, on any face.** ## What changes `likePatternPredicate`'s SQLite arm in `sql-driver.ts` now calls `sqliteLikePatternMatch`. `RemoteTransport.pushLikePattern` in `remote-transport.ts` restates the same rule for the remote transport. The PR's own parity test holds the two to the same predicate text and bindings, the precedent `pushLike` already follows. Sharing one compiler would need a new export from `driver-sql`'s `index.ts`, which is outside this claim's file surface. How a pattern compiles: - **A literal prefix followed only by `%`** (`'ab%'`, `'%'`, `'%%'`) keeps the bare `col GLOB ?` with the same binding as before. A prefix free of U+0000 cannot be changed by the cut, the same argument `$startsWith` stands on. - **Every other pattern** compiles to `CASE WHEN instr(CAST(col AS BLOB), X'00') > 0 THEN (a recursive CTE that replaces each U+0000 with one stand-in character, then GLOB) ELSE col GLOB ? END`. - A value without U+0000 takes the `ELSE` arm: the same `GLOB` as before. - NULL also takes the `ELSE` arm and stays NULL, so the NULL-safe `$not` wrapper composes unchanged. - **A case-exact pattern with a literal prefix** leads with `col GLOB 'PREFIX*'`, where PREFIX is the text before the first unescaped wildcard. Every value the pattern matches satisfies that conjunct, cut or not, and it keeps the index range. `$ilike` has no index range to keep, since `lower(col)` is not an indexed column, so it gets no conjunct. **Why the rewrite is exact.** The stand-in is the smallest code point from U+0001 up that is not a literal character of the pattern, not an ASCII letter and not a surrogate. The pattern holds no U+0000, because `hasNulInLikePattern` refuses it at every door. So a U+0000 in the value can only be matched by `%` or `_`. The stand-in is one character, as U+0000 is, and no literal position of the pattern matches it, just as none matched U+0000. Because it is not a letter, the `$ilike` fold cannot turn it into one. So the rewritten value matches exactly when the original does. **H2 is measured, not assumed.** On all three engines, `replace(v, char(0), char(1))` returns its input unchanged, `GLOB` and `LIKE` cut at U+0000, `lower()` keeps the bytes past U+0000, and `instr()` over BLOB finds U+0000. A correlated recursive CTE inside a scalar subquery works on all three and splices every U+0000, with NULL staying NULL. The earlier sentinel measurement (223 of 2392 cells wrong) is consistent with this: `replace()` never reached a U+0000 there. ## H3: characters outside the Basic Multilingual Plane under `_` (older than this PR, not widened) The spec's regex has no `u` flag, so `_` in `formula` and `driver-memory` matches one UTF-16 unit, while `GLOB`'s `?` matches one code point. At base and at head alike, 20 of 390 cells over such values without U+0000 differ from `formula` on every SQLite face (17 queries). For example, `$like: '_'` returns a stored '😀' on SQLite and not in `formula`. This PR's U+0000 arm uses `GLOB` too, so it follows the code-point reading, consistent with the arm for values without U+0000. On the astral value holding U+0000, 0 of 130 head cells differ from a code-point oracle (the spec's regex compiled with the `u` flag). The divergence is reported to the seat as an out-of-scope finding and not fixed here. ## H4: plans and cost `EXPLAIN QUERY PLAN` of the statements the real compilers emit (the `SqlDriver` compile, and the remote statement captured from a recording libSQL client) over `plan_probe(id integer primary key, v text)` with an index on `v`. Identical on better-sqlite3, sql.js and libSQL: | filter | before | after | |---|---|---| | `$like 'ab%'` | SEARCH USING COVERING INDEX (v greater-than ? AND v less-than ?) | same, same statement | | `$like 'ab_'`, `'ab%cd'`, `'abc'`, `'a%b%'` | SEARCH USING COVERING INDEX (range) | the same SEARCH, plus a correlated scalar subquery | | `$like '%ab'`, `'_b'`, `''` | SCAN | SCAN, plus a correlated scalar subquery | | `$like '%'`, `$ilike 'ab%'` | SCAN | SCAN, same statement | | `$ilike 'ab_'` | SCAN | SCAN, plus a correlated scalar subquery | | `$eq`, `$startsWith` (controls) | SEARCH | unchanged | Cost is the median of 7 runs of `count(*)` over 10,000 rows, base `GLOB` against head. This is a shared box, so read the ratios: | rows holding U+0000 | `$like '%ab%c'` | `$like '_b%'` | `$like 'ab_%'` | `$like 'ab%'` | `$ilike '%AB%C'` | |---|---|---|---|---|---| | 0% | 1.4–1.8x | 1.7–2.4x | 1.1–2.2x | 1.0x | 1.3–1.4x | | 10% | 4.2–4.7x | 5.4–7.8x | 3.3–3.7x | 0.9–1.0x | 2.4–2.8x | | 100% (1–3 each) | 25–37x (23–34 ms) | 26–44x (25–31 ms) | 23–33x (5–7 ms) | 0.7–1.1x | 12–14x (25–35 ms) | The cost for values without U+0000 is the per-row `instr()` guard. A text-`instr` spelling of it is also correct on all three engines, but it measured slower than the BLOB one. Every head count equals `formula`'s count, and base counts differ wherever U+0000 rows exist. ## H5: the remote face, executed `TursoDriver` remote over a real `@libsql/client` engine runs the whole grid in the probe (0 of 3380 cells differ at head) and the whole pin suite in CI. Its predicate text and bindings equal the local compiler's, modulo identifier quoting. A real Turso server, and its wire encoding of a string holding U+0000, are NOT MEASURED. ## H6: two pending changesets, two DELIBERATE CORRECTIONS (please confirm) This PR makes one sentence false in each of two pending changesets, and it corrects each sentence in place. One line changes in each file, and every other line of both files is byte-identical. - `Check Changeset` goes RED on both by design. `scripts/check-empty-changeset.mjs`'s foreign-changeset rule is content-blind: any `M` row on a changeset present at the merge base is refused, whichever card owns it. - Both are the **DELIBERATE CORRECTION** class. Its remedy is to confirm the correction here. ⛔ Restoring either file would put a false sentence back. 1. `.changeset/20024-sqlite-glob-stored-nul.md` is this card's own item 1 entry. The last bullet changed (line 29 of 29): - Before: "- `$like` and `$ilike` still compile to `GLOB` on SQLite, so they still read a stored value only up to its first U+0000. A pattern holding U+0000 is no longer cut there: every driver that answers `$like` now refuses it (`INVALID_FILTER` / 400), by its own entry in this release." - After: "- `$like` and `$ilike` are outside this entry. Two other entries in this release cover them: on SQLite they now read the whole stored value as well, and every driver that answers `$like` refuses a pattern holding U+0000 (`INVALID_FILTER` / 400)." 2. `.changeset/20041-like-nul-pattern-refused.md` is #20041's entry. The first bullet under **Not changed here** changed (line 36 of 44). The seat's claim amendment 5830697185 on #20024 (ruling A) added this bullet to this PR's surface. - Before: "- A pattern without U+0000 matched against a STORED value that holds U+0000 still differs from `formula` on the SQLite faces, because `GLOB` also reads the stored value only up to its first U+0000. No refusal of the pattern can reach that half." - After: "- A pattern without U+0000 matched against a STORED value that holds U+0000 is not refused: it is well formed, and on the SQLite faces it reads the whole stored value, by its own entry in this release." - The new sentence is true at head `89e02a2521`: - The refusal doors read only the pattern (a non-string pattern, a dangling escape, or a pattern holding U+0000). - The three pin suites pass there: 65 of 65, 22 of 22 and 23 of 23. They include patterns without U+0000 over stored values holding U+0000 on every SQLite face, and every such case returns `formula`'s rows and none is refused. - "Its own entry" is `.changeset/20024-like-stored-nul.md`. The new entry is `.changeset/20024-like-stored-nul.md`. ## Tests (at `7434e6646e`; the code is identical at head) - New suites: - `sql-driver-20024-like-stored-nul.test.ts`: 65 of 65 pass. It has 29 literal cases, each pinned and checked against `formula`, a 67-pattern grid held to `formula` cell by cell, the exact compiled statements, and an `EXPLAIN QUERY PLAN` pin. - `sqlite-wasm-20024-like-stored-nul.test.ts`: 22 of 22 pass (19 literal cases, the grid, a plan pin, and the stored-bytes premise). - `turso-20024-like-stored-nul.test.ts`: 23 of 23 pass. It runs local, remote over the stub, and remote over a real libSQL engine, plus the grid and the emitter-parity case. `driver-turso` does not depend on `formula`, so its grid is held to the spec's `matchesLikePattern`. - Full suites: - `pnpm --filter @objectstack/driver-sql exec vitest run --maxWorkers=2`: 187 files passed and 11 skipped; 3089 tests passed and 170 skipped. - `@objectstack/driver-sqlite-wasm`: 34 of 34 files, 644 of 644 tests. - `@objectstack/driver-turso`: 70 of 70 files, 1686 of 1686 tests. - No existing pin moved: before the new suites were added, all three full suites were green against the new emitters. - `pnpm --filter … run typecheck` exits 0 for all three packages, and `tsc --listFiles` counts each new suite once. - Ablations, each with a prediction made from the base statement or a simulation, a mutation proved on disk through `scripts/ablation-replace.mjs`, a `driver-sql` rebuild, and `ablation-dist-preflight` (marker present, then `--absent` after restore). Each restore was proved as the blob hash equal to HEAD, with `git status --porcelain` empty. - (1) Dispatch reverted in both emitters, which gives the base statement. Predicted and observed: driver-sql 25 failed / 40 passed, sqlite-wasm 15 / 7, turso 16 / 7. - (2) Stand-in fixed at U+0001 in both emitters. Predicted and observed: 4 / 61, 2 / 20, 2 / 21. The failures are the U+0001 cases, the grid, and the stand-in statement pin. - (3) Remote emitter only. Predicted and observed: turso 17 failed / 6 passed, including the parity test. ## Gates (the full union at `b424b90f7c`; the changeset gates re-run at `89e02a2521`) At `89e02a2521`, the head that adds the second correction: - `node scripts/check-empty-changeset.mjs --base origin/main` exits 1. This is expected and names both files: "This PR changes a changeset it did not add: .changeset/20024-sqlite-glob-stored-nul.md present on the merge base and CHANGED by this PR ... .changeset/20041-like-nul-pattern-refused.md present on the merge base and CHANGED by this PR". Both are the H6 DELIBERATE CORRECTIONS. - `node scripts/check-changeset-no-major.mjs --base origin/main --event`, over this body, exits 0: "This diff introduces no `major` bump.", and "LEVEL AXIS: this PR declares clause-② `no`, so no package here is declared to have grown a published surface" (declaration line `Clause-②: no`, no direction arm). - `node scripts/check-adr-0087-registration.mjs --base origin/main` exits 0: "check-adr-0087-registration: this PR adds no declared-breaking changeset (3 non-breaking changeset(s) seen)." - `node scripts/check-issue-citations.mjs --base fe677ae` exits 0: 7 citations across 2 files resolve. At `b424b90f7c`: - `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derives 62 commands. All 62 were run and reconciled with `--ran`: 62 derived, 62 run, 0 NOT MEASURED, 0 UNRUN. - Every command exits 0 except `node scripts/check-empty-changeset.mjs --base origin/main`, which exits 1 for the H6 correction above. - `check:dual-build-cjs-loads`, `check:lean-entry-closure` and `check:type-check-debt` first exited 3 (PREREQUISITE NOT MET). They exit 0 after `turbo run build` over `./packages/*` and `./packages/*/*` (72 tasks). The dist-reading gates were re-run over that build and are green too. - `check:driver-conformance`: 50 covered cells, 0 DEBT, 0 exempt. - `node scripts/check-issue-citations.mjs --base fe677ae`: 7 citations resolve. - The rosters beside these paths are green: `check-changeset-fixed`, `check:authz-resolver`, `check:error-code-casing`, `check:filter-alias-parity`. - ESLint, narrowed: `eslint --no-inline-config --format json` over the 5 changed `.ts` files gives 5 files, 0 errors, 0 warnings, 0 ignored. - The population is every changed `.ts` file, each resolving a config under `--print-config`. - `eslint.config.mjs` never enables type-aware linting, so no untouched file's verdict can move. The full `pnpm lint` run is CI's. ## Compile surfaces 1. `driver-sql` `likePatternPredicate`, SQLite arm: TOUCHED. Measured on better-sqlite3 3.53.4. `SqliteWasmDriver` (sql.js 3.49.1) and `TursoDriver` local inherit it, and each is measured by the grid and by its own suite. The Postgres and MySQL arms are not touched and NOT MEASURED (no live server here). 2. turso `RemoteTransport.pushLikePattern`: TOUCHED. Measured on `makeLibsqlSqliteStub` and on a real `@libsql/client` engine (3.45.1). A real Turso server is NOT MEASURED. 3. and 4. service-analytics `compileScopedFilterToSql` / `lowerAnalyticsWhere`: not touched. `$like` / `$ilike` are in their unevaluated set (`preview-unevaluable-operator.test.ts`), so these faces do not compile them. Rows are NOT MEASURED. 5. `formula` `matchesFilterCondition`: not touched; it is the oracle. - Half face: objectql `having-filter`: not touched, and `$like` does not occur in `having-filter.ts`. - Thawed faces: - `driver-memory` `checkCondition`: not touched. Measured on the same grid: 0 of 7670 cells differ from `formula`. - `driver-mongodb`: not touched, NOT MEASURED (no mongod). ## Acceptance notes - H3's divergence outside the Basic Multilingual Plane is reported to the seat with its measured reach, and is not fixed here. - The #20041 changeset: its line 32 ("every `$like` / `$ilike` pattern without U+0000 answers exactly as before") is scoped to #20041's own change and is left as it is. The corrected line 36, in the same section, states what this PR changes for stored values. - Draft PR #20104 edits `RemoteTransport` elsewhere in `remote-transport.ts`. This PR's hunk is inside `pushLikePattern` only. Written by `session_01Bvd69VPa6puiNzzPUroDBx` (os-dev, `mode:subagent`) for the `domain:engine` seat. --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent a08e059 commit f09d412

8 files changed

Lines changed: 1004 additions & 12 deletions

File tree

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
---
2+
"@objectstack/driver-sql": patch
3+
"@objectstack/driver-sqlite-wasm": patch
4+
"@objectstack/driver-turso": patch
5+
---
6+
7+
fix(driver-sql, driver-turso): on SQLite, `$like` / `$ilike` read the whole stored value, instead of stopping at its first U+0000 (#20024)
8+
9+
Clause-②: no
10+
11+
On the SQLite faces `$like` and `$ilike` compiled to `GLOB`, and SQLite's `glob()` reads the stored value only up to its first U+0000. So a pattern without U+0000 answered a different question over a value holding one, and nothing raised. Measured on `SqlDriver` over better-sqlite3 (SQLite 3.53.4), on `SqliteWasmDriver` over sql.js (3.49.1), on `TursoDriver`'s local mode, and on its remote transport over a local libSQL engine (3.45.1). All four answered alike:
12+
13+
- `$like: 'a'` returned a value stored as `'a'` + U+0000 + `'b'`, and `$like: ''` returned U+0000 + `'z'`;
14+
- `$like: '%b'`, `$like: 'a_b'` and `$ilike: 'A_B'` did not return `'a'` + U+0000 + `'b'`;
15+
- `$like: '_'` did not return a value that is a lone U+0000.
16+
17+
Over 108 patterns and 22 `$not` / `$or` / `$and` compositions against 59 stored values, 359 of the 3380 cells over values holding U+0000 differed from `@objectstack/formula` on each face.
18+
19+
What changes: a stored value holding U+0000 now has each U+0000 replaced by one stand-in character before `GLOB` reads it. The stand-in is never a literal character of the pattern, never an ASCII letter, and never U+0000. A U+0000 in the value can only be matched by `%` or `_`, and so can the stand-in, so the answer is the one the whole value gives. Such a filter now returns the rows `driver-memory` and `@objectstack/formula` return for it, under `$not`, `$or` and `$and` as well: 0 of those 3380 cells differ on any of the four faces. `$ilike` still folds ASCII letters only.
20+
21+
- `@objectstack/driver-sql`: the SQLite arm of `SqlDriver`'s `$like` / `$ilike` compiler. `SqliteWasmDriver` and `TursoDriver`'s local mode inherit it.
22+
- `@objectstack/driver-sqlite-wasm`: none of its own code changes. It inherits the fix.
23+
- `@objectstack/driver-turso`: the remote transport's own emitter, changed the same way.
24+
25+
What does not change:
26+
27+
- A stored value without U+0000 gets the same answer as before: 0 of 16640 such cells moved on any face.
28+
- A pattern that is a literal prefix followed only by `%` (`'ab%'`, `'%'`) compiles to the same `GLOB` with the same bound pattern as before. Cutting the value at its first U+0000 cannot change that answer.
29+
- No index is lost. Under `EXPLAIN QUERY PLAN` over an indexed TEXT column on all three engines, each `$like` pattern measured that starts with a literal (`'ab%'`, `'ab_'`, `'ab%cd'`, `'abc'`, `'a%b%'`) keeps its covering-index search, and each one that starts with a wildcard still scans. A case-exact pattern with a literal prefix now leads with a `GLOB` on that prefix followed by `*`, which every matching value satisfies and which is what keeps that search.
30+
- `_` still matches one character, as `GLOB`'s `?` does. A character outside the Basic Multilingual Plane is one character to `_` on SQLite and two to `@objectstack/formula`, which counts UTF-16 units. That difference is older than this change, and this change does not alter it.
31+
- A pattern holding U+0000 is still refused (`INVALID_FILTER` / 400).
32+
- The Postgres and MySQL arms are untouched.
33+
34+
Cost, measured over 10,000 rows on the three engines: against a value without U+0000 the new compile adds one `instr()` per row, and those queries took 1.1 to 2.4 times as long as `GLOB` alone, at most 4.5 ms. A value holding U+0000 pays the rewrite: 10,000 rows each holding one to three U+0000 took up to 35 ms, against at most 3 ms for `GLOB`.

‎.changeset/20024-sqlite-glob-stored-nul.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,4 +26,4 @@ What does not change:
2626
- `$startsWith` with a comparand without U+0000 compiles to the same `GLOB` with the same bound pattern as before. The stored value's cut cannot change its answer.
2727
- No index is lost. The SQL `SqlDriver` compiles, run under `EXPLAIN QUERY PLAN` over an indexed TEXT column on all three engines, scanned the table for these four operators under `GLOB` and still does; `$startsWith` keeps its index search.
2828
- The Postgres and MySQL arms are untouched.
29-
- `$like` and `$ilike` still compile to `GLOB` on SQLite, so they still read a stored value only up to its first U+0000. A pattern holding U+0000 is no longer cut there: every driver that answers `$like` now refuses it (`INVALID_FILTER` / 400), by its own entry in this release.
29+
- `$like` and `$ilike` are outside this entry. Two other entries in this release cover them: on SQLite they now read the whole stored value as well, and every driver that answers `$like` refuses a pattern holding U+0000 (`INVALID_FILTER` / 400).

‎.changeset/20041-like-nul-pattern-refused.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ A pattern that ends in a lone unpaired backslash AND holds U+0000 keeps the dang
3333

3434
**Not changed here:**
3535

36-
- A pattern without U+0000 matched against a STORED value that holds U+0000 still differs from `formula` on the SQLite faces, because `GLOB` also reads the stored value only up to its first U+0000. No refusal of the pattern can reach that half.
36+
- A pattern without U+0000 matched against a STORED value that holds U+0000 is not refused: it is well formed, and on the SQLite faces it reads the whole stored value, by its own entry in this release.
3737
- `@objectstack/formula` still evaluates such a pattern. It refuses nothing, and answers `false` for a dangling escape rather than refusing it, so it is not one of these doors.
3838
- `driver-mongodb`, objectql `having` and `service-analytics` refused every `$like` / `$ilike` before this change, and still do.
3939

0 commit comments

Comments
 (0)