Skip to content

Commit 3172831

Browse files
os-zhuangos-steveclaude
authored
fix(drivers): text-operator case folding is the contract's answer, not the dialect's (#6518) (#6706)
* fix(drivers): 文本算子的大小写由契约决定,不再由方言决定 (#6518) `$contains` 族与 `$icontains` 的大小写答案此前取决于底层数据库的 `LIKE` 语义,两个方向都是**过匹配**——返回了 filter 排除的行,在 ADR-0021 的 RLS 读作用域上是越权而非「宽松过滤」(#3948): | | `$contains` 族(契约要求大小写敏感,#4706 Q2=A) | `$icontains`(只折叠 ASCII,#4706 Q1=A) | |:--|:--|:--| | SQLite / turso / sqlite-wasm | ❌ `LIKE` 自带 ASCII 折叠 | ✅ `lower()` 只折 ASCII | | Postgres | ✅ `LIKE` 恰好大小写精确 | ❌ `LOWER()` 折全 Unicode | | MySQL | ❌ 随 collation | ❌ `LOWER()` 折全 Unicode | 横着读:**每个方言恰好在另一个方言犯错的那一半上是对的**——这就是两半都 无法从单一后端上被发现的原因。 下译改为按方言选构造,仍然只有一个发射点(转义与折叠是同一条代码路径, 未转义的通配符是 P0 —— #5567): - **SQLite 族 → `GLOB`**。`LIKE` 的 ASCII 折叠无法按语句关闭 (`PRAGMA case_sensitive_like` 是连接级全局开关),实测 `CAST(col AS BLOB) LIKE ?` 一行都不匹配。`GLOB` 大小写精确,并自带转义类 (`*` / `?` / `[` 写作自闭合字符类,因为 SQLite 语法不给 `GLOB` 任何 `ESCAPE` 子句)。 - **Postgres → `LIKE` 不变**,只把折叠从 `LOWER()` 换成显式 `translate()`; live PG 16 实测 `LOWER('CAFÉ')` 就是 `'café'`。 - **MySQL → `CAST(… AS BINARY)` 上的 `LIKE`**,比较按字节,collation 不再 参与;折叠同样按字节做,因 UTF-8 自同步故恰为 ASCII-only。 - 其余 client 保留改动前的 `LIKE`/`LOWER()` 形状,并记为残留而非静默。 turso 的 remote transport 自带一份编译器,同批移动;两个 transport 由新的 `turso-local-remote-text-parity.test.ts` 用共享 `FILTER_TEXT_CASES` 钉在同 一组行上。driver-sql / -sqlite-wasm / -turso 三格的 `FILTER_TEXT_CASES` DEBT 随之清偿;driver-memory / driver-mongodb 属 #5499 冻结族,如实留 DEBT 并改指新填的后继单 #6682。 Fixes #6518 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AYvVzTaCWDCC32e9Le9zXn * test(plugin-auth): restate the case-semantics note now that $contains is case-exact on sqlite (#6518) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AYvVzTaCWDCC32e9Le9zXn --------- Co-authored-by: os-zhuang <steve@objectstack.ai> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent be8561d commit 3172831

18 files changed

Lines changed: 1349 additions & 282 deletions
Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
---
2+
"@objectstack/driver-sql": minor
3+
"@objectstack/driver-sqlite-wasm": minor
4+
"@objectstack/driver-turso": minor
5+
---
6+
7+
fix(drivers): text-operator case folding is the CONTRACT's answer, not the dialect's (#6518)
8+
9+
The `$contains` family and `$icontains` returned **different rows on different
10+
databases** for the same filter, because case sensitivity was decided by whatever
11+
`LIKE` happened to mean on the dialect underneath. Both directions **over-matched**
12+
— they returned rows the filter excludes, which on an ADR-0021 RLS read scope is
13+
over-reach rather than a loose filter (#3948):
14+
15+
| | `$contains` / `$notContains` / `$startsWith` / `$endsWith` — case-SENSITIVE (#4706 Q2 = A) | `$icontains` — folds ASCII ONLY (#4706 Q1 = A) |
16+
|:--|:--|:--|
17+
| SQLite / turso / sqlite-wasm | ❌ `LIKE` folds ASCII | ✅ `lower()` is ASCII-only |
18+
| Postgres | ✅ `LIKE` is case-exact | ❌ `LOWER()` folds all of Unicode |
19+
| MySQL | ❌ follows the column's collation | ❌ `LOWER()` folds all of Unicode |
20+
21+
Read across: **each dialect was already right on the half another one got wrong**,
22+
which is why neither half could be found from one backend alone.
23+
24+
## What now runs
25+
26+
The construct is chosen per dialect, in one emitter, so the escaping and the fold
27+
stay a single code path (an unescaped wildcard is a filter bypass, P0 — #5567):
28+
29+
- **SQLite family → `GLOB`.** `LIKE`'s ASCII fold cannot be switched off per
30+
statement (`PRAGMA case_sensitive_like` is connection-global, so one query would
31+
redefine every other query on the connection), and `CAST(col AS BLOB) LIKE ?` was
32+
measured to match *nothing at all*. `GLOB` is case-exact and brings its own
33+
escaped class — `*`, `?`, `[` as the self-closing classes `[*]`, `[?]`, `[[]`,
34+
because SQLite's grammar gives `GLOB` no `ESCAPE` clause. `$icontains` keeps
35+
`lower()` on both operands, still ASCII-only.
36+
- **Postgres → `LIKE`, unchanged.** Only the fold moved, from `LOWER()` to an
37+
explicit `translate()` over the 26 ASCII letters. Measured on a live PostgreSQL
38+
16 (ICU database): `LOWER('CAFÉ')` is `'café'` — the over-fold — while the
39+
`translate()` form leaves `É` alone.
40+
- **MySQL → `LIKE` over `CAST(… AS BINARY)`**, so the comparison is byte-wise and
41+
no collation decides the case; `$icontains` folds byte-wise over the same binary
42+
rendering, which is ASCII-only because UTF-8 is self-synchronising.
43+
- **Any other client** keeps the previous `LIKE` / `LOWER()` shape — it is the only
44+
form that still runs there — and is recorded as residue rather than left to be
45+
discovered.
46+
47+
`driver-turso`'s remote transport carries the twin (it compiles filters itself and
48+
inherits nothing), and the two transports are now held to the same rows by a
49+
parity suite that runs the shared `FILTER_TEXT_CASES` on both.
50+
51+
## Behaviour change — read this before upgrading
52+
53+
A filter whose comparand's case did not match the stored text used to match on
54+
SQLite/turso/sqlite-wasm and may have matched on MySQL. It no longer does:
55+
56+
```ts
57+
// rows: { id: '1', name: 'ACME Corp' }, { id: '2', name: 'acme corp' }
58+
{ name: { $contains: 'acme' } } // was ['1','2'] on SQLite → now ['2'] everywhere
59+
{ name: { $icontains: 'acme' } } // ['1','2'] — unchanged, and now correct on PG/MySQL too
60+
{ name: { $icontains: 'café' } } // was ['3','4'] on PG/MySQL → now ['4'] everywhere
61+
```
62+
63+
If you were relying on `$contains` to ignore case, **write `$icontains`** — that is
64+
the operator for it, and it now folds the same ASCII-only range on every backend.
65+
Result sets only ever get NARROWER, never wider, so a filter that was already
66+
correct stays correct.
67+
68+
## Why `minor` rather than `major`
69+
70+
No declared surface moves. `$contains` still exists, still takes the same
71+
comparand, and `filter.zod.ts` is untouched — the case-sensitivity this delivers
72+
was **already published** as the contract by #5701 (`FILTER_TEXT_CASES`, one
73+
release earlier in this same v17 major), and the drivers were the half that had
74+
not caught up. This is Prime Directive #12 applied in the direction it points:
75+
declared = enforced. It is graded the way its sibling #5702/#6549 was graded for
76+
the same operator family in the same rc cycle, and it registers nothing in the
77+
ADR-0087 registries because it retires no authorable key.
78+
79+
## What is deliberately NOT in this change
80+
81+
`driver-memory` and `driver-mongodb` still fold case on their query paths — they
82+
are the #5499 frozen family, so their `FILTER_TEXT_CASES` cells stay honest DEBT
83+
and are tracked as #6682 (case sensitivity) and #6520 (`$icontains`). The
84+
`service-analytics` SQL compilers were measured already compliant: they emit
85+
Postgres-shaped statements, where `LIKE` is case-exact, and that assumption is now
86+
written down and pinned rather than implied.

‎packages/drivers/driver-sql/src/sql-driver-icontains-and-retired-operators.test.ts‎

Lines changed: 124 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -2,44 +2,50 @@
22

33
/**
44
* [#5702] `$icontains` on the SQL family, and the retirement of `$regex` /
5-
* `$options` — the DRIVER half of the #4706 ruling (its contract half is #5701).
5+
* `$options` — the DRIVER half of the #4706 ruling (its contract half is #5701)
6+
* — plus [#6518] the case-sensitivity half #5702 could not deliver.
67
*
7-
* ## What this pins, and why it is not the shared text case-set
8+
* ## This cell now answers the WHOLE shared case-set
89
*
9-
* `@objectstack/spec/data` carries a canonical text case-set whose rows this
10-
* file reuses (`FILTER_TEXT_ROWS` — the same nine, so a verdict here is
11-
* comparable to one anywhere else). It deliberately does NOT import that
12-
* case-set's CASES export, because `scripts/check-driver-conformance.mjs`
13-
* judges a cell covered by that import and this driver does not yet answer the
14-
* whole table: five of its cases require the `$contains` family to be
15-
* case-SENSITIVE (#4706 Q2 = A), which SQLite's `LIKE` is not, and which cannot
16-
* be fixed in this driver alone — `read-scope-sql` and `service-analytics`
17-
* compile the same predicate for RLS and for the analytics face, so a
18-
* driver-only change would give ONE permission rule two row sets (#3948). That
19-
* work is filed separately and the driver's DEBT row stays open for it.
10+
* `FILTER_TEXT_CASES` is imported and executed below, which is what
11+
* `scripts/check-driver-conformance.mjs` reads as coverage. #5702 deliberately
12+
* imported only `FILTER_TEXT_ROWS` and left the CASES export alone, because
13+
* five of its cases require the `$contains` family to be case-SENSITIVE (#4706
14+
* Q2 = A) and SQLite's `LIKE` folds ASCII — an import then would have flipped
15+
* the cell to "covered" while five cases went unanswered, which is a gate
16+
* reporting success over a standard nobody runs.
2017
*
21-
* Importing the case-set here would flip the cell to "covered" while five of
22-
* its cases were unanswered — a gate reporting success over a standard nobody
23-
* runs, which is exactly the failure the gate exists to prevent.
18+
* #6518 is what makes the import honest: `textMatchPredicate` emits `GLOB` on
19+
* the SQLite dialects (case-exact, carrying GLOB's own `[*]` / `[?]` / `[[]`
20+
* escapes because SQLite's grammar gives it no `ESCAPE` clause), keeps `LIKE`
21+
* on Postgres (already case-exact) with the `$icontains` fold moved off the
22+
* Unicode-folding `LOWER()` onto an ASCII-only `translate()`, and compares over
23+
* `CAST(… AS BINARY)` on MySQL. The per-dialect reasoning and the measurements
24+
* behind each cell live on that function.
2425
*
2526
* ## The reverse verification, direction decided BEFORE it was run
2627
*
2728
* - **Refusal face** — predicted RED, measured RED. Restoring the deleted
2829
* `case '$regex':` fallthrough makes every assertion below that reads `code`
2930
* / `status` fail, because the filter compiles again and nothing throws.
30-
* - **`$icontains` face on SQLite** — predicted red, and the prediction needed
31-
* a correction that is recorded here rather than smoothed over: deleting the
32-
* `case '$icontains':` arm turns these cases red LOUDLY (the operator falls to
33-
* `default:` and is refused), but deleting only the `LOWER()` fold does NOT,
34-
* for any comparand. SQLite's `LIKE` folds ASCII by itself, so on this
35-
* dialect the fold is unobservable in rows. The compiled-SQL case at the end
36-
* is what pins it, and it is the only thing here that can.
31+
* - **`$icontains` face on SQLite** — #5702 recorded that deleting only the
32+
* fold changed NO row here, because `LIKE` folded ASCII by itself, so the
33+
* compiled-SQL case was the only witness it could offer. #6518 retires that
34+
* caveat, and the retirement was predicted before it was run: under `GLOB`
35+
* the fold is load-bearing in ROWS. Measured — dropping the column-side
36+
* `lower()` turns `$icontains: 'acme'` and `$icontains: 'ACME'` alike from
37+
* `['1','2']` into `['2']`, and `$icontains: 'CAFÉ'` from `['3']` into `[]`.
38+
* - **`$contains` case-sensitivity** — predicted RED on exactly the five case
39+
* rows before the change, measured RED: reverting the sqlite arm of
40+
* `textMatchPredicate` to `LIKE` fails those five case rows and NO others in
41+
* the table (10 reds in this file: the five, plus the five blocks below that
42+
* name the construct directly).
3743
*/
3844

3945
import type { Knex } from 'knex';
4046
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
4147
import type { DriverOptions, FilterCondition } from '@objectstack/spec/data';
42-
import { FILTER_TEXT_ROWS } from '@objectstack/spec/data';
48+
import { FILTER_TEXT_CASES, FILTER_TEXT_ROWS } from '@objectstack/spec/data';
4349
import { SqlDriver } from './sql-driver.js';
4450

4551
/** The error a refused filter produced — never a bare `toThrow()` (see below). */
@@ -206,26 +212,105 @@ describe('[#5702] SqlDriver — $icontains, and the retired $regex/$options', ()
206212
await expect(driver.find('txt', { where: { name: { $regex: 'zzz' } } }, BYPASS)).rejects.toThrow();
207213
});
208214

209-
// ── The fold, where it is actually observable on SQLite ────────────────────
215+
// ── [#6518] The case-SENSITIVE family (#4706 Q2 = A) ───────────────────────
210216

211-
it('compiles LOWER() on both operands, and $contains on neither', async () => {
212-
// On SQLite this is the ONLY witness to the fold: `LIKE` already folds ASCII
213-
// here, so `$contains` and `$icontains` select identical rows for every
214-
// comparand and a dropped `LOWER()` changes no answer. It changes the SQL,
215-
// and it changes the answer on Postgres (whose LIKE is case-exact), so the
216-
// statement is what has to be pinned.
217+
it('$contains is case-SENSITIVE in both directions', async () => {
218+
// The defect this closes: on `origin/main` both lines answered ['1','2'],
219+
// because SQLite's `LIKE` folds ASCII whatever the contract says. Over-
220+
// matching, not a near miss — the caller asked for one row and got two.
221+
expect(await ids({ name: { $contains: 'acme' } })).toEqual(['2']);
222+
expect(await ids({ name: { $contains: 'ACME' } })).toEqual(['1']);
223+
});
224+
225+
it('the case rule holds under negation too', async () => {
226+
// Row 1 is EXCLUDED from the negation only if the positive form excluded
227+
// it. A folding backend drops row 1 here and looks "stricter", which is the
228+
// reading that hides the defect.
229+
expect(await ids({ name: { $notContains: 'acme' } }))
230+
.toEqual(['1', '3', '4', '5', '6', '7', '8', '9']);
231+
});
232+
233+
it('$startsWith and $endsWith are case-SENSITIVE', async () => {
234+
expect(await ids({ name: { $startsWith: 'ACME' } })).toEqual(['1']);
235+
expect(await ids({ name: { $endsWith: 'corp' } })).toEqual(['2']);
236+
});
237+
238+
it('$contains and $icontains no longer answer identically on SQLite', async () => {
239+
// #5702 measured that they DID, for every comparand, and recorded it as the
240+
// reason `$icontains` was unobservable on this dialect. That is the fact
241+
// #6518 changes, so it is asserted rather than left as a comment: the two
242+
// operators are now distinguishable in rows on the dialect where they were
243+
// not.
244+
expect(await ids({ name: { $contains: 'acme' } })).not
245+
.toEqual(await ids({ name: { $icontains: 'acme' } }));
246+
});
247+
248+
// ── [#6518] GLOB's metacharacters are escaped, exactly as LIKE's were ──────
249+
250+
for (const [label, comparand] of [
251+
['*', 'a*b'],
252+
['?', 'a?b'],
253+
['[', 'a[b'],
254+
] as const) {
255+
it(`treats "${label}" as a literal character, not a GLOB metacharacter`, async () => {
256+
// The new operator brings a NEW escaped character class, and an
257+
// unescaped `*` is the same filter bypass an unescaped `%` is under LIKE:
258+
// measured on this fixture, the unescaped pattern `*a*b*` returns rows
259+
// 7, 8 and 9 where the escaped one returns none of them. No fixture row
260+
// holds these characters, so the observable claim is that the pattern
261+
// stays literal and selects nothing rather than expanding.
262+
expect(await ids({ name: { $contains: comparand } })).toEqual([]);
263+
expect(await ids({ name: { $icontains: comparand } })).toEqual([]);
264+
});
265+
}
266+
267+
it('compiles the case-exact SQLite construct, on both operators', async () => {
217268
// Identifier quoting is the dialect's (knex renders backticks on the sqlite
218-
// clients), so the assertion is on the SHAPE, not on one dialect's quotes.
219-
const unquote = (sql: string) => sql.replace(/[`"\[\]]/g, '');
269+
// clients), so the assertion is on the SHAPE. `[` / `]` are NOT stripped
270+
// here the way #5702's version stripped them: under GLOB they are the
271+
// escape mechanism, so erasing them would erase what is being pinned.
272+
const unquote = (sql: string) => sql.replace(/[`"]/g, '');
220273

221274
const icontainsSql = unquote(driver.compileWhere({ name: { $icontains: 'acme' } }));
222-
expect(icontainsSql).toContain('LOWER(name) LIKE LOWER(');
223-
expect(icontainsSql).toContain('ESCAPE');
224-
// The escaped pattern still travels as the comparand, wildcards and all.
225-
expect(icontainsSql).toContain('%acme%');
275+
expect(icontainsSql).toContain('lower(name) GLOB lower(');
276+
expect(icontainsSql).toContain('*acme*');
277+
// GLOB has no ESCAPE clause in SQLite's grammar; emitting one is a syntax
278+
// error, so its absence is part of the construct rather than a detail.
279+
expect(icontainsSql).not.toContain('ESCAPE');
226280

227281
const containsSql = unquote(driver.compileWhere({ name: { $contains: 'acme' } }));
228-
expect(containsSql).toContain('name LIKE');
229-
expect(containsSql).not.toContain('LOWER');
282+
expect(containsSql).toContain('name GLOB');
283+
expect(containsSql).not.toContain('lower');
284+
expect(containsSql).not.toContain('LIKE');
285+
});
286+
287+
// ── The shared standard, executed ──────────────────────────────────────────
288+
289+
/**
290+
* [#6518] Every case in `FILTER_TEXT_CASES`, on the live SQLite cell.
291+
*
292+
* The invariant this enforces is the case-set's own: a backend answers with
293+
* the SAME ROW SET, or REFUSES with `INVALID_FILTER` — never a third, quieter
294+
* answer. So the rejection cases assert `code` AND `status` (ADR-0112) plus
295+
* the prescription the message must carry, and never a bare `toThrow()`.
296+
*
297+
* The blocks above are not redundant with this loop: they name the SQLite
298+
* construct and its escaped class, which the shared table deliberately does
299+
* not know about — it encodes the contract, not any dialect's spelling.
300+
*/
301+
describe('FILTER_TEXT_CASES — the shared standard', () => {
302+
for (const testCase of FILTER_TEXT_CASES) {
303+
it(testCase.name, async () => {
304+
if (testCase.expectRejection) {
305+
const err = await refusalOf(testCase.filter);
306+
expect(err.code).toBe(testCase.code);
307+
expect(err.status).toBe(400);
308+
for (const mention of testCase.mustMention) expect(err.message).toContain(mention);
309+
return;
310+
}
311+
expect(await ids(testCase.filter), testCase.note ?? testCase.name)
312+
.toEqual([...testCase.expected]);
313+
});
314+
}
230315
});
231316
});

0 commit comments

Comments
 (0)