Repository navigation
fix(driver-sql, driver-turso): on SQLite, $like / $ilike read the whole stored value past a U+0000 (#20024) - #20140
Conversation
…e $like / $ilike (WIP) Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx Co-authored-by: Claude <noreply@anthropic.com>
…y's last bullet Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx Co-authored-by: Claude <noreply@anthropic.com>
…over a stored U+0000 on every SQLite face Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx Co-authored-by: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx Co-authored-by: Claude <noreply@anthropic.com>
…ed token Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx Co-authored-by: Claude <noreply@anthropic.com>
📓 Docs Drift Check8 anchor(s) derived from 2 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run. What this run could not see
Coarse fallback — 13 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): Which tree this was computed onThis run read A worktree cut from an older # while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 3a266c7e739752a9d4b2d04b9fe0c918ab8af37c && git checkout 3a266c7e739752a9d4b2d04b9fe0c918ab8af37c
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin a08e059c61f6ea4784d400c40ddedd3c81495bef 89e02a2521f36984272b9b2401822c88f71ad0e5 && git checkout -B drift-repro a08e059c61f6ea4784d400c40ddedd3c81495bef && git merge --no-ff 89e02a2521f36984272b9b2401822c88f71ad0e5
node scripts/docs-audit/affected-docs.mjs --json a08e059c61f6ea4784d400c40ddedd3c81495bef |
#20024 makes false Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx Co-authored-by: Claude <noreply@anthropic.com>
Contract reviewServed-tier: Scope: 8 files (+1004/−12) on merge base
Compile surfaces:
Measured in detached worktrees at head and base, with the reviewer's own builds, grid and probes. ① Derived judgments
② Semver level
③ Boundary flags
Implemented-by: VERDICT: PASS |
Fixes #20024
Clause-②: no
Item 2 (ii) of #20024, direction C (landing record 5823785768). On the SQLite faces, a
$like/$ilikepattern without U+0000 over a stored value holding U+0000 now gets the answerlikePatternToRegexSourcegives, which is also what@objectstack/formulaanddriver-memoryreturn. Item 1 landed as PR #20038 and item 2 (i) as PR #20124 (#20041). This is the card's last item, so the PR closes it.patchfor@objectstack/driver-sql,@objectstack/driver-sqlite-wasm(it inherits the fix) and@objectstack/driver-turso. Nothing that was accepted is refused now: the refusal doors,hasNulInLikePatternandhasDanglingLikeEscapeare untouched. The grid, plan and cost numbers below were taken atb424b90f7cor earlier. Head89e02a2521differs fromb424b90f7cby one changeset line only, and its emitter code is byte-identical to7434e6646e; every commit since then changed changeset prose only.The defect, measured at base
fe677aeeedglob()reads the stored value as a C string, only up to its first U+0000. The grid has 59 stored values (27 holding U+0000, leading, interior, trailing, doubled, next to multi-byte and U+0001 characters, plus NULL). It runs 130 queries against them: 81$likeand 27$ilikebare patterns, and 22$not/$or/$andcompositions. That is 7670 cells per face, each compared withformula. The four faces answered identically:formulaformulaThe four faces are
SqlDriveron better-sqlite3 (SQLite 3.53.4),SqliteWasmDriveron sql.js (3.49.1),TursoDriverlocal, andTursoDriverremote over a real@libsql/clientfile::memory:engine (3.45.1). At base, 74 of the 108 bare patterns got at least one row holding U+0000 wrong. Some examples:$like: 'a'returned'a'+ U+0000 +'b';$like: ''returned U+0000 +'z';$like: '_'missed a lone U+0000;$ilike: 'A_B'missed'a'+ U+0000 +'b'.Base vs head, every stored value without U+0000 and NULL: 0 of 16640 cells moved, on any face.
What changes
likePatternPredicate's SQLite arm insql-driver.tsnow callssqliteLikePatternMatch.RemoteTransport.pushLikePatterninremote-transport.tsrestates the same rule for the remote transport. The PR's own parity test holds the two to the same predicate text and bindings, the precedentpushLikealready follows. Sharing one compiler would need a new export fromdriver-sql'sindex.ts, which is outside this claim's file surface. How a pattern compiles:%('ab%','%','%%') keeps the barecol GLOB ?with the same binding as before. A prefix free of U+0000 cannot be changed by the cut, the same argument$startsWithstands on.CASE WHEN instr(CAST(col AS BLOB), X'00') > 0 THEN (a recursive CTE that replaces each U+0000 with one stand-in character, then GLOB) ELSE col GLOB ? END.ELSEarm: the sameGLOBas before.ELSEarm and stays NULL, so the NULL-safe$notwrapper composes unchanged.col GLOB 'PREFIX*', where PREFIX is the text before the first unescaped wildcard. Every value the pattern matches satisfies that conjunct, cut or not, and it keeps the index range.$ilikehas no index range to keep, sincelower(col)is not an indexed column, so it gets no conjunct.Why the rewrite is exact. The stand-in is the smallest code point from U+0001 up that is not a literal character of the pattern, not an ASCII letter and not a surrogate. The pattern holds no U+0000, because
hasNulInLikePatternrefuses it at every door. So a U+0000 in the value can only be matched by%or_. The stand-in is one character, as U+0000 is, and no literal position of the pattern matches it, just as none matched U+0000. Because it is not a letter, the$ilikefold cannot turn it into one. So the rewritten value matches exactly when the original does.H2 is measured, not assumed. On all three engines,
replace(v, char(0), char(1))returns its input unchanged,GLOBandLIKEcut at U+0000,lower()keeps the bytes past U+0000, andinstr()over BLOB finds U+0000. A correlated recursive CTE inside a scalar subquery works on all three and splices every U+0000, with NULL staying NULL. The earlier sentinel measurement (223 of 2392 cells wrong) is consistent with this:replace()never reached a U+0000 there.H3: characters outside the Basic Multilingual Plane under
_(older than this PR, not widened)The spec's regex has no
uflag, so_informulaanddriver-memorymatches one UTF-16 unit, whileGLOB's?matches one code point. At base and at head alike, 20 of 390 cells over such values without U+0000 differ fromformulaon every SQLite face (17 queries). For example,$like: '_'returns a stored '😀' on SQLite and not informula. This PR's U+0000 arm usesGLOBtoo, so it follows the code-point reading, consistent with the arm for values without U+0000. On the astral value holding U+0000, 0 of 130 head cells differ from a code-point oracle (the spec's regex compiled with theuflag). The divergence is reported to the seat as an out-of-scope finding and not fixed here.H4: plans and cost
EXPLAIN QUERY PLANof the statements the real compilers emit (theSqlDrivercompile, and the remote statement captured from a recording libSQL client) overplan_probe(id integer primary key, v text)with an index onv. Identical on better-sqlite3, sql.js and libSQL:$like 'ab%'$like 'ab_','ab%cd','abc','a%b%'$like '%ab','_b',''$like '%',$ilike 'ab%'$ilike 'ab_'$eq,$startsWith(controls)Cost is the median of 7 runs of
count(*)over 10,000 rows, baseGLOBagainst head. This is a shared box, so read the ratios:$like '%ab%c'$like '_b%'$like 'ab_%'$like 'ab%'$ilike '%AB%C'The cost for values without U+0000 is the per-row
instr()guard. A text-instrspelling of it is also correct on all three engines, but it measured slower than the BLOB one. Every head count equalsformula's count, and base counts differ wherever U+0000 rows exist.H5: the remote face, executed
TursoDriverremote over a real@libsql/clientengine runs the whole grid in the probe (0 of 3380 cells differ at head) and the whole pin suite in CI. Its predicate text and bindings equal the local compiler's, modulo identifier quoting. A real Turso server, and its wire encoding of a string holding U+0000, are NOT MEASURED.H6: two pending changesets, two DELIBERATE CORRECTIONS (please confirm)
This PR makes one sentence false in each of two pending changesets, and it corrects each sentence in place. One line changes in each file, and every other line of both files is byte-identical.
Check Changesetgoes RED on both by design.scripts/check-empty-changeset.mjs's foreign-changeset rule is content-blind: anyMrow on a changeset present at the merge base is refused, whichever card owns it..changeset/20024-sqlite-glob-stored-nul.mdis this card's own item 1 entry. The last bullet changed (line 29 of 29):$likeand$ilikestill compile toGLOBon SQLite, so they still read a stored value only up to its first U+0000. A pattern holding U+0000 is no longer cut there: every driver that answers$likenow refuses it (INVALID_FILTER/ 400), by its own entry in this release."$likeand$ilikeare outside this entry. Two other entries in this release cover them: on SQLite they now read the whole stored value as well, and every driver that answers$likerefuses a pattern holding U+0000 (INVALID_FILTER/ 400).".changeset/20041-like-nul-pattern-refused.mdis spec + drivers: a$like/$ilikepattern holding U+0000 is cut at the NUL on the SQLite faces and answers a different question; refuse it at every face through one spec predicate besidehasDanglingLikeEscape#20041's entry. The first bullet under Not changed here changed (line 36 of 44). The seat's claim amendment 5830697185 on driver-sql / driver-turso (SQLite faces):GLOBstill cuts a STORED value at its first U+0000, so a NUL-free$contains/$notContains/$icontains/$endsWithanswers wrongly on it; a$like/$ilikepattern holding U+0000 is cut the same way #20024 (ruling A) added this bullet to this PR's surface.formulaon the SQLite faces, becauseGLOBalso reads the stored value only up to its first U+0000. No refusal of the pattern can reach that half."89e02a2521:formula's rows and none is refused..changeset/20024-like-stored-nul.md.The new entry is
.changeset/20024-like-stored-nul.md.Tests (at
7434e6646e; the code is identical at head)sql-driver-20024-like-stored-nul.test.ts: 65 of 65 pass. It has 29 literal cases, each pinned and checked againstformula, a 67-pattern grid held toformulacell by cell, the exact compiled statements, and anEXPLAIN QUERY PLANpin.sqlite-wasm-20024-like-stored-nul.test.ts: 22 of 22 pass (19 literal cases, the grid, a plan pin, and the stored-bytes premise).turso-20024-like-stored-nul.test.ts: 23 of 23 pass. It runs local, remote over the stub, and remote over a real libSQL engine, plus the grid and the emitter-parity case.driver-tursodoes not depend onformula, so its grid is held to the spec'smatchesLikePattern.pnpm --filter @objectstack/driver-sql exec vitest run --maxWorkers=2: 187 files passed and 11 skipped; 3089 tests passed and 170 skipped.@objectstack/driver-sqlite-wasm: 34 of 34 files, 644 of 644 tests.@objectstack/driver-turso: 70 of 70 files, 1686 of 1686 tests.pnpm --filter … run typecheckexits 0 for all three packages, andtsc --listFilescounts each new suite once.scripts/ablation-replace.mjs, adriver-sqlrebuild, andablation-dist-preflight(marker present, then--absentafter restore). Each restore was proved as the blob hash equal to HEAD, withgit status --porcelainempty.Gates (the full union at
b424b90f7c; the changeset gates re-run at89e02a2521)At
89e02a2521, the head that adds the second correction:node scripts/check-empty-changeset.mjs --base origin/mainexits 1. This is expected and names both files: "This PR changes a changeset it did not add: .changeset/20024-sqlite-glob-stored-nul.md present on the merge base and CHANGED by this PR ... .changeset/20041-like-nul-pattern-refused.md present on the merge base and CHANGED by this PR". Both are the H6 DELIBERATE CORRECTIONS.node scripts/check-changeset-no-major.mjs --base origin/main --event, over this body, exits 0: "This diff introduces nomajorbump.", and "LEVEL AXIS: this PR declares clause-②no, so no package here is declared to have grown a published surface" (declaration lineClause-②: no, no direction arm).node scripts/check-adr-0087-registration.mjs --base origin/mainexits 0: "check-adr-0087-registration: this PR adds no declared-breaking changeset (3 non-breaking changeset(s) seen)."node scripts/check-issue-citations.mjs --base fe677aeeedexits 0: 7 citations across 2 files resolve.At
b424b90f7c:node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstackderives 62 commands. All 62 were run and reconciled with--ran: 62 derived, 62 run, 0 NOT MEASURED, 0 UNRUN.node scripts/check-empty-changeset.mjs --base origin/main, which exits 1 for the H6 correction above.check:dual-build-cjs-loads,check:lean-entry-closureandcheck:type-check-debtfirst exited 3 (PREREQUISITE NOT MET). They exit 0 afterturbo run buildover./packages/*and./packages/*/*(72 tasks). The dist-reading gates were re-run over that build and are green too.check:driver-conformance: 50 covered cells, 0 DEBT, 0 exempt.node scripts/check-issue-citations.mjs --base fe677aeeed: 7 citations resolve.check-changeset-fixed,check:authz-resolver,check:error-code-casing,check:filter-alias-parity.eslint --no-inline-config --format jsonover the 5 changed.tsfiles gives 5 files, 0 errors, 0 warnings, 0 ignored..tsfile, each resolving a config under--print-config.eslint.config.mjsnever enables type-aware linting, so no untouched file's verdict can move. The fullpnpm lintrun is CI's.Compile surfaces
driver-sqllikePatternPredicate, SQLite arm: TOUCHED. Measured on better-sqlite3 3.53.4.SqliteWasmDriver(sql.js 3.49.1) andTursoDriverlocal inherit it, and each is measured by the grid and by its own suite. The Postgres and MySQL arms are not touched and NOT MEASURED (no live server here).RemoteTransport.pushLikePattern: TOUCHED. Measured onmakeLibsqlSqliteStuband on a real@libsql/clientengine (3.45.1). A real Turso server is NOT MEASURED.compileScopedFilterToSql/lowerAnalyticsWhere: not touched.$like/$ilikeare in their unevaluated set (preview-unevaluable-operator.test.ts), so these faces do not compile them. Rows are NOT MEASURED.formulamatchesFilterCondition: not touched; it is the oracle.having-filter: not touched, and$likedoes not occur inhaving-filter.ts.driver-memorycheckCondition: not touched. Measured on the same grid: 0 of 7670 cells differ fromformula.driver-mongodb: not touched, NOT MEASURED (no mongod).Acceptance notes
$like/$ilikepattern holding U+0000 is cut at the NUL on the SQLite faces and answers a different question; refuse it at every face through one spec predicate besidehasDanglingLikeEscape#20041 changeset: its line 32 ("every$like/$ilikepattern without U+0000 answers exactly as before") is scoped to spec + drivers: a$like/$ilikepattern holding U+0000 is cut at the NUL on the SQLite faces and answers a different question; refuse it at every face through one spec predicate besidehasDanglingLikeEscape#20041's own change and is left as it is. The corrected line 36, in the same section, states what this PR changes for stored values.RemoteTransportelsewhere inremote-transport.ts. This PR's hunk is insidepushLikePatternonly.Written by
session_01Bvd69VPa6puiNzzPUroDBx(os-dev,mode:subagent) for thedomain:engineseat.