Repository navigation
Commit f09d412
Fixes #20024
Clause-②: no
Item 2 (ii) of #20024, direction C (landing record 5823785768). On the
SQLite faces, a `$like` / `$ilike` pattern without U+0000 over a stored
value holding U+0000 now gets the answer `likePatternToRegexSource`
gives, which is also what `@objectstack/formula` and `driver-memory`
return. Item 1 landed as PR #20038 and item 2 (i) as PR #20124 (#20041).
This is the card's last item, so the PR closes it.
`patch` for `@objectstack/driver-sql`, `@objectstack/driver-sqlite-wasm`
(it inherits the fix) and `@objectstack/driver-turso`. Nothing that was
accepted is refused now: the refusal doors, `hasNulInLikePattern` and
`hasDanglingLikeEscape` are untouched. The grid, plan and cost numbers
below were taken at `b424b90f7c` or earlier. Head `89e02a2521` differs
from `b424b90f7c` by one changeset line only, and its emitter code is
byte-identical to `7434e6646e`; every commit since then changed
changeset prose only.
## The defect, measured at base `fe677aeeed`
`glob()` reads the stored value as a C string, only up to its first
U+0000. The grid has 59 stored values (27 holding U+0000, leading,
interior, trailing, doubled, next to multi-byte and U+0001 characters,
plus NULL). It runs 130 queries against them: 81 `$like` and 27 `$ilike`
bare patterns, and 22 `$not` / `$or` / `$and` compositions. That is 7670
cells per face, each compared with `formula`. The four faces answered
identically:
| stored value | cells per face | base: differ from `formula` | head:
differ from `formula` |
|---|---|---|---|
| holds U+0000 (26 values in the Basic Multilingual Plane) | 3380 | 359
| **0** |
| no U+0000, in the Basic Multilingual Plane (28 values) | 3640 | 0 | 0
|
| NULL | 130 | 0 | 0 |
| outside the Basic Multilingual Plane, no U+0000 (3 values) | 390 | 20
| 20 (older than this PR, see H3) |
| outside the Basic Multilingual Plane, with U+0000 (1 value) | 130 | 6
(6 against a code-point oracle) | 2 (0 against a code-point oracle) |
The four faces are `SqlDriver` on better-sqlite3 (SQLite 3.53.4),
`SqliteWasmDriver` on sql.js (3.49.1), `TursoDriver` local, and
`TursoDriver` remote over a real `@libsql/client` `file::memory:` engine
(3.45.1). At base, 74 of the 108 bare patterns got at least one row
holding U+0000 wrong. Some examples:
- `$like: 'a'` returned `'a'` + U+0000 + `'b'`;
- `$like: ''` returned U+0000 + `'z'`;
- `$like: '_'` missed a lone U+0000;
- `$ilike: 'A_B'` missed `'a'` + U+0000 + `'b'`.
**Base vs head, every stored value without U+0000 and NULL: 0 of 16640
cells moved, on any face.**
## What changes
`likePatternPredicate`'s SQLite arm in `sql-driver.ts` now calls
`sqliteLikePatternMatch`. `RemoteTransport.pushLikePattern` in
`remote-transport.ts` restates the same rule for the remote transport.
The PR's own parity test holds the two to the same predicate text and
bindings, the precedent `pushLike` already follows. Sharing one compiler
would need a new export from `driver-sql`'s `index.ts`, which is outside
this claim's file surface. How a pattern compiles:
- **A literal prefix followed only by `%`** (`'ab%'`, `'%'`, `'%%'`)
keeps the bare `col GLOB ?` with the same binding as before. A prefix
free of U+0000 cannot be changed by the cut, the same argument
`$startsWith` stands on.
- **Every other pattern** compiles to `CASE WHEN instr(CAST(col AS
BLOB), X'00') > 0 THEN (a recursive CTE that replaces each U+0000 with
one stand-in character, then GLOB) ELSE col GLOB ? END`.
- A value without U+0000 takes the `ELSE` arm: the same `GLOB` as
before.
- NULL also takes the `ELSE` arm and stays NULL, so the NULL-safe `$not`
wrapper composes unchanged.
- **A case-exact pattern with a literal prefix** leads with `col GLOB
'PREFIX*'`, where PREFIX is the text before the first unescaped
wildcard. Every value the pattern matches satisfies that conjunct, cut
or not, and it keeps the index range. `$ilike` has no index range to
keep, since `lower(col)` is not an indexed column, so it gets no
conjunct.
**Why the rewrite is exact.** The stand-in is the smallest code point
from U+0001 up that is not a literal character of the pattern, not an
ASCII letter and not a surrogate. The pattern holds no U+0000, because
`hasNulInLikePattern` refuses it at every door. So a U+0000 in the value
can only be matched by `%` or `_`. The stand-in is one character, as
U+0000 is, and no literal position of the pattern matches it, just as
none matched U+0000. Because it is not a letter, the `$ilike` fold
cannot turn it into one. So the rewritten value matches exactly when the
original does.
**H2 is measured, not assumed.** On all three engines, `replace(v,
char(0), char(1))` returns its input unchanged, `GLOB` and `LIKE` cut at
U+0000, `lower()` keeps the bytes past U+0000, and `instr()` over BLOB
finds U+0000. A correlated recursive CTE inside a scalar subquery works
on all three and splices every U+0000, with NULL staying NULL. The
earlier sentinel measurement (223 of 2392 cells wrong) is consistent
with this: `replace()` never reached a U+0000 there.
## H3: characters outside the Basic Multilingual Plane under `_` (older
than this PR, not widened)
The spec's regex has no `u` flag, so `_` in `formula` and
`driver-memory` matches one UTF-16 unit, while `GLOB`'s `?` matches one
code point. At base and at head alike, 20 of 390 cells over such values
without U+0000 differ from `formula` on every SQLite face (17 queries).
For example, `$like: '_'` returns a stored '😀' on SQLite and not in
`formula`. This PR's U+0000 arm uses `GLOB` too, so it follows the
code-point reading, consistent with the arm for values without U+0000.
On the astral value holding U+0000, 0 of 130 head cells differ from a
code-point oracle (the spec's regex compiled with the `u` flag). The
divergence is reported to the seat as an out-of-scope finding and not
fixed here.
## H4: plans and cost
`EXPLAIN QUERY PLAN` of the statements the real compilers emit (the
`SqlDriver` compile, and the remote statement captured from a recording
libSQL client) over `plan_probe(id integer primary key, v text)` with an
index on `v`. Identical on better-sqlite3, sql.js and libSQL:
| filter | before | after |
|---|---|---|
| `$like 'ab%'` | SEARCH USING COVERING INDEX (v greater-than ? AND v
less-than ?) | same, same statement |
| `$like 'ab_'`, `'ab%cd'`, `'abc'`, `'a%b%'` | SEARCH USING COVERING
INDEX (range) | the same SEARCH, plus a correlated scalar subquery |
| `$like '%ab'`, `'_b'`, `''` | SCAN | SCAN, plus a correlated scalar
subquery |
| `$like '%'`, `$ilike 'ab%'` | SCAN | SCAN, same statement |
| `$ilike 'ab_'` | SCAN | SCAN, plus a correlated scalar subquery |
| `$eq`, `$startsWith` (controls) | SEARCH | unchanged |
Cost is the median of 7 runs of `count(*)` over 10,000 rows, base `GLOB`
against head. This is a shared box, so read the ratios:
| rows holding U+0000 | `$like '%ab%c'` | `$like '_b%'` | `$like 'ab_%'`
| `$like 'ab%'` | `$ilike '%AB%C'` |
|---|---|---|---|---|---|
| 0% | 1.4–1.8x | 1.7–2.4x | 1.1–2.2x | 1.0x | 1.3–1.4x |
| 10% | 4.2–4.7x | 5.4–7.8x | 3.3–3.7x | 0.9–1.0x | 2.4–2.8x |
| 100% (1–3 each) | 25–37x (23–34 ms) | 26–44x (25–31 ms) | 23–33x (5–7
ms) | 0.7–1.1x | 12–14x (25–35 ms) |
The cost for values without U+0000 is the per-row `instr()` guard. A
text-`instr` spelling of it is also correct on all three engines, but it
measured slower than the BLOB one. Every head count equals `formula`'s
count, and base counts differ wherever U+0000 rows exist.
## H5: the remote face, executed
`TursoDriver` remote over a real `@libsql/client` engine runs the whole
grid in the probe (0 of 3380 cells differ at head) and the whole pin
suite in CI. Its predicate text and bindings equal the local compiler's,
modulo identifier quoting. A real Turso server, and its wire encoding of
a string holding U+0000, are NOT MEASURED.
## H6: two pending changesets, two DELIBERATE CORRECTIONS (please
confirm)
This PR makes one sentence false in each of two pending changesets, and
it corrects each sentence in place. One line changes in each file, and
every other line of both files is byte-identical.
- `Check Changeset` goes RED on both by design.
`scripts/check-empty-changeset.mjs`'s foreign-changeset rule is
content-blind: any `M` row on a changeset present at the merge base is
refused, whichever card owns it.
- Both are the **DELIBERATE CORRECTION** class. Its remedy is to confirm
the correction here. ⛔ Restoring either file would put a false sentence
back.
1. `.changeset/20024-sqlite-glob-stored-nul.md` is this card's own item
1 entry. The last bullet changed (line 29 of 29):
- Before: "- `$like` and `$ilike` still compile to `GLOB` on SQLite, so
they still read a stored value only up to its first U+0000. A pattern
holding U+0000 is no longer cut there: every driver that answers `$like`
now refuses it (`INVALID_FILTER` / 400), by its own entry in this
release."
- After: "- `$like` and `$ilike` are outside this entry. Two other
entries in this release cover them: on SQLite they now read the whole
stored value as well, and every driver that answers `$like` refuses a
pattern holding U+0000 (`INVALID_FILTER` / 400)."
2. `.changeset/20041-like-nul-pattern-refused.md` is #20041's entry. The
first bullet under **Not changed here** changed (line 36 of 44). The
seat's claim amendment 5830697185 on #20024 (ruling A) added this bullet
to this PR's surface.
- Before: "- A pattern without U+0000 matched against a STORED value
that holds U+0000 still differs from `formula` on the SQLite faces,
because `GLOB` also reads the stored value only up to its first U+0000.
No refusal of the pattern can reach that half."
- After: "- A pattern without U+0000 matched against a STORED value that
holds U+0000 is not refused: it is well formed, and on the SQLite faces
it reads the whole stored value, by its own entry in this release."
- The new sentence is true at head `89e02a2521`:
- The refusal doors read only the pattern (a non-string pattern, a
dangling escape, or a pattern holding U+0000).
- The three pin suites pass there: 65 of 65, 22 of 22 and 23 of 23. They
include patterns without U+0000 over stored values holding U+0000 on
every SQLite face, and every such case returns `formula`'s rows and none
is refused.
- "Its own entry" is `.changeset/20024-like-stored-nul.md`.
The new entry is `.changeset/20024-like-stored-nul.md`.
## Tests (at `7434e6646e`; the code is identical at head)
- New suites:
- `sql-driver-20024-like-stored-nul.test.ts`: 65 of 65 pass. It has 29
literal cases, each pinned and checked against `formula`, a 67-pattern
grid held to `formula` cell by cell, the exact compiled statements, and
an `EXPLAIN QUERY PLAN` pin.
- `sqlite-wasm-20024-like-stored-nul.test.ts`: 22 of 22 pass (19 literal
cases, the grid, a plan pin, and the stored-bytes premise).
- `turso-20024-like-stored-nul.test.ts`: 23 of 23 pass. It runs local,
remote over the stub, and remote over a real libSQL engine, plus the
grid and the emitter-parity case. `driver-turso` does not depend on
`formula`, so its grid is held to the spec's `matchesLikePattern`.
- Full suites:
- `pnpm --filter @objectstack/driver-sql exec vitest run
--maxWorkers=2`: 187 files passed and 11 skipped; 3089 tests passed and
170 skipped.
- `@objectstack/driver-sqlite-wasm`: 34 of 34 files, 644 of 644 tests.
- `@objectstack/driver-turso`: 70 of 70 files, 1686 of 1686 tests.
- No existing pin moved: before the new suites were added, all three
full suites were green against the new emitters.
- `pnpm --filter … run typecheck` exits 0 for all three packages, and
`tsc --listFiles` counts each new suite once.
- Ablations, each with a prediction made from the base statement or a
simulation, a mutation proved on disk through
`scripts/ablation-replace.mjs`, a `driver-sql` rebuild, and
`ablation-dist-preflight` (marker present, then `--absent` after
restore). Each restore was proved as the blob hash equal to HEAD, with
`git status --porcelain` empty.
- (1) Dispatch reverted in both emitters, which gives the base
statement. Predicted and observed: driver-sql 25 failed / 40 passed,
sqlite-wasm 15 / 7, turso 16 / 7.
- (2) Stand-in fixed at U+0001 in both emitters. Predicted and observed:
4 / 61, 2 / 20, 2 / 21. The failures are the U+0001 cases, the grid, and
the stand-in statement pin.
- (3) Remote emitter only. Predicted and observed: turso 17 failed / 6
passed, including the parity test.
## Gates (the full union at `b424b90f7c`; the changeset gates re-run at
`89e02a2521`)
At `89e02a2521`, the head that adds the second correction:
- `node scripts/check-empty-changeset.mjs --base origin/main` exits 1.
This is expected and names both files: "This PR changes a changeset it
did not add: .changeset/20024-sqlite-glob-stored-nul.md present on the
merge base and CHANGED by this PR ...
.changeset/20041-like-nul-pattern-refused.md present on the merge base
and CHANGED by this PR". Both are the H6 DELIBERATE CORRECTIONS.
- `node scripts/check-changeset-no-major.mjs --base origin/main
--event`, over this body, exits 0: "This diff introduces no `major`
bump.", and "LEVEL AXIS: this PR declares clause-② `no`, so no package
here is declared to have grown a published surface" (declaration line
`Clause-②: no`, no direction arm).
- `node scripts/check-adr-0087-registration.mjs --base origin/main`
exits 0: "check-adr-0087-registration: this PR adds no declared-breaking
changeset (3 non-breaking changeset(s) seen)."
- `node scripts/check-issue-citations.mjs --base fe677ae` exits 0: 7
citations across 2 files resolve.
At `b424b90f7c`:
- `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` derives 62 commands. All 62 were run and
reconciled with `--ran`: 62 derived, 62 run, 0 NOT MEASURED, 0 UNRUN.
- Every command exits 0 except `node scripts/check-empty-changeset.mjs
--base origin/main`, which exits 1 for the H6 correction above.
- `check:dual-build-cjs-loads`, `check:lean-entry-closure` and
`check:type-check-debt` first exited 3 (PREREQUISITE NOT MET). They exit
0 after `turbo run build` over `./packages/*` and `./packages/*/*` (72
tasks). The dist-reading gates were re-run over that build and are green
too.
- `check:driver-conformance`: 50 covered cells, 0 DEBT, 0 exempt.
- `node scripts/check-issue-citations.mjs --base fe677ae`: 7
citations resolve.
- The rosters beside these paths are green: `check-changeset-fixed`,
`check:authz-resolver`, `check:error-code-casing`,
`check:filter-alias-parity`.
- ESLint, narrowed: `eslint --no-inline-config --format json` over the 5
changed `.ts` files gives 5 files, 0 errors, 0 warnings, 0 ignored.
- The population is every changed `.ts` file, each resolving a config
under `--print-config`.
- `eslint.config.mjs` never enables type-aware linting, so no untouched
file's verdict can move. The full `pnpm lint` run is CI's.
## Compile surfaces
1. `driver-sql` `likePatternPredicate`, SQLite arm: TOUCHED. Measured on
better-sqlite3 3.53.4. `SqliteWasmDriver` (sql.js 3.49.1) and
`TursoDriver` local inherit it, and each is measured by the grid and by
its own suite. The Postgres and MySQL arms are not touched and NOT
MEASURED (no live server here).
2. turso `RemoteTransport.pushLikePattern`: TOUCHED. Measured on
`makeLibsqlSqliteStub` and on a real `@libsql/client` engine (3.45.1). A
real Turso server is NOT MEASURED.
3. and 4. service-analytics `compileScopedFilterToSql` /
`lowerAnalyticsWhere`: not touched. `$like` / `$ilike` are in their
unevaluated set (`preview-unevaluable-operator.test.ts`), so these faces
do not compile them. Rows are NOT MEASURED.
5. `formula` `matchesFilterCondition`: not touched; it is the oracle.
- Half face: objectql `having-filter`: not touched, and `$like` does not
occur in `having-filter.ts`.
- Thawed faces:
- `driver-memory` `checkCondition`: not touched. Measured on the same
grid: 0 of 7670 cells differ from `formula`.
- `driver-mongodb`: not touched, NOT MEASURED (no mongod).
## Acceptance notes
- H3's divergence outside the Basic Multilingual Plane is reported to
the seat with its measured reach, and is not fixed here.
- The #20041 changeset: its line 32 ("every `$like` / `$ilike` pattern
without U+0000 answers exactly as before") is scoped to #20041's own
change and is left as it is. The corrected line 36, in the same section,
states what this PR changes for stored values.
- Draft PR #20104 edits `RemoteTransport` elsewhere in
`remote-transport.ts`. This PR's hunk is inside `pushLikePattern` only.
Written by `session_01Bvd69VPa6puiNzzPUroDBx` (os-dev, `mode:subagent`)
for the `domain:engine` seat.
---------
Co-authored-by: Claude <noreply@anthropic.com>
1 parent a08e059 commit f09d412
8 files changed
Lines changed: 1004 additions & 12 deletions
File tree
- .changeset
- packages/drivers
- driver-sqlite-wasm/src
- driver-sql/src
- driver-turso/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
| 29 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
| 36 | + | |
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| |||
0 commit comments