Repository navigation
Commit 226e00c
Fixes #20040
Clause-②: no (narrowing)
## What this changes
The analytics `where` door (`lowerAnalyticsWhere` in
`packages/services/service-analytics/src/strategies/filter-normalizer.ts`)
now refuses a `$null` or `$exists` flag whose value is not a boolean. It
refuses before any lowering, in the door's `INVALID_FILTER` / 400
envelope, with the message kept.
- `FieldOperatorsSchema` declares both flags `z.boolean()`. The #5347 /
#5369 rulings refuse a non-boolean flag in every position and on every
backend, because the backends read one in opposite directions.
`driver-sql` refuses it (`nonBooleanNullComparandError` /
`nonBooleanExistsComparandError`), and so does the read-scope compiler
(#6387, `assertBooleanFlagComparands`).
- This door read the flag by identity (`=== true` for `$null`, `===
false` for `$exists`), so every other value lowered to `set`, IS NOT
NULL.
How:
- `normalizeWhereComparands` gains a third pass,
`assertBooleanNullFlags`, after the shared comparand-shape face and the
shared comparand-type face.
- It walks their traversal (`forEachWhereFieldEntry`), so it finds a
flag in every position they judge: `$and` / `$or` members, `$not`, field
entries, and a nested relation's entries.
- It reports a flag at the path those faces give it
(`where.acct.stage.$null`).
- The message names the operator, the field and the path: `[analytics]
Operator "$null" on field "stage" requires a boolean comparand (true or
false). Received a string ("x") at where.stage.$null. …`
- Its main clause is `driver-sql`'s, word for word through "(true or
false)".
- The rest says what this door used to do and how to repair it. It
carries no tracker number.
- **Reuse.** `@objectstack/spec`, `core` and `types` publish no
predicate or sentence for this refusal (searched; none exists).
- So the one check lives in this file, and its docblock cites #5347 /
#5369 and the sibling faces.
- It imports no driver and adds no spec export.
- The received value is rendered with this package's own `shapePreview`
(`comparand-shape.ts`).
- `true` / `false` lower exactly as before.
- The `FilterArray` spelling needs no pass (measured, below):
- `parseFilterAST` writes a hard-coded boolean for `is_null` /
`is_not_null`, and refuses `$null` / `exists` as array operators;
- `isFilterAST` refuses an embedded object.
### The order chosen, and why
- **After the shape and type faces.** A flag the type face refuses keeps
the face's sentence: `undefined`, a plain object, a `Map`, a bigint
beyond 2^53. That is the sentence the `FilterArray` spelling and the
engine seam give. A shape or type defect elsewhere in the same `where`
is still answered first. Ablation C pins this order.
- **Before the `undefined` gate (#6386) and everything else `buildNode`
reaches.** Two readers see the flag before the identity read in
`fieldLeaves`:
- the #5146 `$not` rewrite, which classifies every field spec through
`nullValueSatisfiesOperator` / `operatorIsNullTotal` (the flag readers
at the old `filter-normalizer.ts:1411` / `:1464`);
- the draft preview, which evaluates what `normalizeWhereComparands`
returns and never reaches `fieldLeaves`.
The `undefined` gate skips both flags by name, so the two gates never
judge one value.
- **One precedence change, same envelope.** A `where` carrying a
non-boolean flag and a defect only the lowering diagnoses is now
answered with the flag. Those defects are a mixed `$` / non-`$` wrapper
(#6444), an operator outside the vocabulary, and a zero-operator
constraint. The code and status stay `INVALID_FILTER` / 400. #20035 took
the same precedence for the type face over #6444. It is pinned, with
boolean controls that keep the old sentences.
- **The preview.** Before the fix it refused every flag ("not
evaluated", 400) while the published faces served the cell, so the two
answered it differently. It reaches the new pass through
`normalizeWhereComparands` and now refuses this cell in the door's
words, byte for byte. `preview-evaluator.ts` is not edited. A boolean
flag still gets the preview's own "not evaluated" refusal.
## Measured first (recorded on this branch as `12e3081867`, test-only,
before any source change)
Base `origin/main` `8d76c2d38c`.
- **Data.** A real sql.js engine (`driver-sqlite-wasm`) over rows r1
'won', r2 NULL and r3 'lost' (field `stage`), with a real `ObjectQL`
behind the ObjectQL face. The echo's SQL was run on the same database.
- **HTTP legs.** Read and measured only, through throwaway harnesses
(not committed) over a real `AnalyticsService`:
- `packages/runtime`'s dispatcher route `POST /api/v1/analytics/query`,
native and ObjectQL-only;
- `packages/rest`'s `POST /analytics/dataset/query`, with the flag in
`selection.runtimeFilter` and in an inline `dataset.filter`.
| `{ stage: { FLAG: V } }` | native execute | echo (run) | ObjectQL
engine path | draft preview | `AnalyticsService.query` (native /
ObjectQL) | HTTP, both routes |
|:--|:--|:--|:--|:--|:--|:--|
| `$null` or `$exists`, V = `'x'`, `'false'`, `'true'`, `0`, `1`,
`null`, `[true]`, `{ $field: 'id' }`, a `Date`, `2n` | r1 r3 (IS NOT
NULL) | r1 r3 | r1 r3; the engine received `{ stage: { $ne: null } }` |
400, "not evaluated" | r1 r3 | **200**, r1 r3, 1 statement |
| the same under `$not` | r2 | r2 | r2 | 400, "not evaluated" | r2 |
200, r2 |
| V = `{ a: 1 }` or `undefined` | 400, type face | = | = | = | = | 400
(the plain object) |
| `$null: true` / `$exists: false` (control) | r2 | r2 | r2, `{ stage:
null }` | 400, "not evaluated" | r2 | 200, r2 |
| `$null: false` / `$exists: true` (control) | r1 r3 | r1 r3 | r1 r3, `{
stage: { $ne: null } }` | 400, "not evaluated" | r1 r3 | 200, r1 r3 |
- **Positions.** The top level, `$and`, `$or`, `$not`, and `$not` beside
`$ne`; every coerced cell answered the same in each. `{ $not: { stage: {
$null: 'true' } } }` served the NULL row the author excluded.
- **`$null: null` / `$exists: null` are reachable**, in process and over
HTTP (JSON `null`). Both read as IS NOT NULL and answered 200:
- the `undefined` gate (#6386) skips both flags by name;
- the type face accepts `null`, which is in its accepted set;
- the shape face's null carve-outs cover `$in` / `$nin` members,
`$between` endpoints and ordering comparands, not the flags.
- **The card's HTTP 200 is confirmed on `main`.**
`FilterConditionSchema` types a field entry as `z.unknown()`, so the
body parse admitted every non-boolean flag on both routes.
- **The `FilterArray` spelling carries no flag value.**
- `['stage', 'is_null', 'x']` gives `notSet` and `['stage', 'isnotnull',
0]` gives `set`; the third member is filler.
- `['stage', '$null', 'x']`, `'null'`, `'exists'` and `'$exists'` are
refused as not a filter.
- `isFilterAST` answers false for every array embedding an object node
(5 shapes probed).
- **Other analytics paths through the same lowering** (the card's
check):
- dataset scope filters and measure filters reach it through the
strategies' `normalizeAnalyticsFilterTree` calls;
- the preview reaches it through `normalizeWhereComparands`;
- the read-scope door does not use it.
## After (head `66266c2a3b`)
- **In process.** Every non-boolean row above is refused
`INVALID_FILTER` / 400 on all seven in-process columns, the preview
included, in the door's sentence (44 of 44 rows). Nothing reaches the
database or the engine: 0 statements and 0 `engine.aggregate` calls.
- **HTTP re-measure.** Both routes answer **400 `INVALID_FILTER`** for
every non-boolean cell, with the message kept and 0 statements. The
controls still answer 200 with the same rows.
- **Controls, byte for byte.** For `$null` / `$exists` × `true` /
`false`, at the top level, under `$not`, and under `$not` beside `$ne`,
I captured:
- the compiled tree;
- the native SQL and its params;
- the echo SQL and its params;
- the engine `where`.
The capture taken on `8d76c2d38c` and the one taken after the fix are
byte-identical (10,328 bytes, the same sha256). The test pins them as
literals.
- **The read scope keeps its own answer**, untouched:
- on native, the echo and `AnalyticsService.query` (native):
`READ_SCOPE_COMPILE_FAILED` / 500, withheld, in `read-scope-sql`'s
sentence;
- on the ObjectQL face, `driver-sql` answers it: `INVALID_FILTER` / 400,
with the policy content withheld.
## Tests
New file
`packages/services/service-analytics/src/__tests__/where-boolean-flag-refusal.test.ts`,
63 cases. Every refusal asserts `code` + `status`.
- the door rows: 2 flags × 10 non-boolean values;
- every position, the nested-relation path, and the received value named
in the message;
- the `FilterArray` control;
- existing refusals keeping their order and sentence;
- the measured precedence against the lowering's own refusals, with
boolean controls;
- every face over a real engine, with statement and engine-call
counters;
- the preview, whose message is byte-equal to the door's;
- the byte-for-byte controls;
- the read-scope face on every sub-face, plus
`compileScopedFilterToSql`;
- stored dataset and measure filters through `queryDataset`.
**Pins re-judged, none deleted.**
- The census was taken before the change, over `service-analytics`,
`packages/rest` and `packages/runtime`, plus a repo-wide test scan.
- `rest` and `runtime` had 0 hits asserting this door's admission.
`rest-4xx-message-truncation.test.ts` copies a `driver-sql` message and
is not this door.
- One pin in `filter-normalizer-undefined-comparand.test.ts` asserted
the admission: the case "$null / $exists: an undefined flag is refused
by the type face; an accepted flag value reads as before".
- It asserted `{ $null: 'false' }` and `{ $exists: 'yes' }` lower to
`set`.
- **Flipped to refusals, with a note.** The `undefined` half is
unchanged, and the case title now says what it asserts.
**Runs.**
- Test-only commit `12e3081867`, before the fix: **53 failed | 10 passed
(63)**.
- Every door row was red.
- Green: the four controls, the type-face and shape-face order pins, the
`FilterArray` control and the three read-scope pins.
- Head `66266c2a3b`:
- `pnpm --filter @objectstack/service-analytics` suite: **125 files,
2949 tests passed**;
- `typecheck`: exit 0, and `tsc --listFiles` includes both touched test
files;
- consumer sanity run against the rebuilt `dist`, not a pin:
`packages/rest`'s 10 analytics route suites (148 passed) and
`packages/runtime`'s 3 analytics suites (36 passed).
### Ablations
Each ran through `scripts/ablation-replace.mjs` from committed state,
with the anchor hit ×1 and the restore proven (blob == HEAD, `git diff
HEAD` empty). `service-analytics` tests import the subject by relative
`src` paths, so no `dist` is in the path, except for leg F.
| leg | mutation | red / green | restore |
|:--|:--|:--|:--|
| A | the gate's one call site removed | **54 red / 43 green** over the
new file and the re-judged one: the 53 door rows from the before-fix run
plus the re-judged pin. Controls and read-scope pins stayed green, as
predicted | blob `a7bdec971782` == HEAD |
| B | `$exists` dropped from the gate's set | **27 red / 70 green**.
Every red case carries an `$exists` assertion (21 name it, 6 are mixed);
every `$null`-only case stayed green | == HEAD |
| C | the gate moved ahead of the shape and type faces | **11 red / 192
green** over the new file, the re-judged file and
`where-type-face-refusal.test.ts`: the order pins (2 + 2) and 7 existing
type-face pins. The first attempt was refused by the tool itself,
because the replacement contained the anchor, so the count could not
drop. Nothing ran, the restore was proven, and it was re-run with an
equivalent spelling | == HEAD |
| D | the gate widened to refuse `false` | **2 red / 61 green**: the two
`false` controls | == HEAD |
| E | `read-scope-sql.ts`'s `assertBooleanFlagComparands` call removed
(a transient ablation; the file is not in this diff) | **2 red / 61
green**: the read-scope pin and `compileScopedFilterToSql`. The
ObjectQL-face pin stayed green, as predicted, because `driver-sql`
answers there | blob `403440964f36` == HEAD |
| F | `driver-sql`'s `$null` gate removed and `driver-sql` rebuilt;
`ablation-dist-preflight --absent` passed (the marker is absent from all
6 built files) | **1 red / 62 green**: the ObjectQL-face read-scope pin
| blob `a89054ee3aab` == HEAD; rebuilt, and preflight shows the marker
present in 2 built files with the tree clean |
## Gates (head `66266c2a3b`)
- **Derivation.** `node scripts/pm/dispatch-gates.mjs --repo
objectstack-ai/objectstack --commands` derived **60 families** from the
real diff. That is 6 more than the dispatch-time list:
`check:engine-double-contract`, `check:objectql-double-limit`,
`check:query-options-erasure`, `check:type-check-coverage`,
`check:type-check-debt` and `check:where-matcher`.
- **60 of 60 exit 0.** Among them:
- `check-adr-0087-registration --base origin/main`, with the disposition
`not-required (no-migration-prescription)`;
- `check-changeset-no-major`, `check-empty-changeset` and
`check:nul-bytes`;
- `check:issue-citations`, which runs only its self-test. The real run
(`GITHUB_TOKEN="$GH_TOKEN" node scripts/check-issue-citations.mjs`)
judged 29 citations, and all resolve.
- **The two built-output gates.** `check:dual-build-cjs-loads` and
`check:type-check-debt` read built output. On the first head, before any
full build, both answered PREREQUISITE NOT MET, which is not a failure.
They ran green after the lint.yml build command (`turbo run build
--filter='./packages/*' --filter='./packages/*/*'`).
- **Reconciliation.** `--ran`: 60 derived, 60 run, 0 NOT MEASURED, 0
UNRUN. Every entry carries its exit code.
- **Merges.** `main` moved twice while this ran: `aa04ea2964` (objectql)
and `fa00ebf447` (runtime), both disjoint from this diff. Both were
merged, never rebased, and the whole union was re-run on the final head.
- **Lint, narrowed and proven.**
- `eslint --no-inline-config --format json` over the 3 changed `.ts`
files: 3 files, 0 errors, 0 warnings.
- `isPathIgnored` is false for all three. The changeset `.md` is ignored
and has no config, so it is outside the population.
- `eslint.config.mjs` never enables type-aware linting (no
`parserOptions.project` and no `projectService`, as its line 327
states). So this diff cannot move the verdict of any untouched file.
## Changeset
`.changeset/20040-analytics-null-flag.md`: `minor` with `!` and a
`Clause-②: no (narrowing)` line, following the #20035 / #20068
precedents.
- **ADR-0087 marker.** It is an HTML-comment line, described here in
words so this body keeps it. It declares `not-required
(no-migration-prescription)`, and the gate accepts it.
- No ledger entry names the flags' boolean domain: the #5347 / #5369
refusals in `driver-sql` and the #6387 refusal were never registered.
- A non-boolean flag has no boolean it can be mechanically rewritten to.
- **Who is affected.** A text scan over 4598 tracked non-test files
found 22 matches for a non-boolean flag. 21 are code comments; the 22nd
is an operator-name lookup table in `driver-memory`, not a filter.
Stored datasets, widget filters and report runtime filters in a
deployment were NOT measured.
## Acceptance notes
- **The preview still refuses a BOOLEAN flag ("not evaluated").** It
evaluates no `$null` / `$exists` at all (the #19810 table), and that is
unchanged and not this card's to change. Widening that table is ruled to
land together with the conformance kits. A drafted chart carrying `{
$null: 'x' }` now gets the flag refusal first. Once repaired to `true`,
it gets the preview's own refusal, while publish serves it.
## Out-of-scope findings (reported to the seat, not filed here)
1. **Class (c).** The save door admits a non-boolean flag that every
query face refuses.
- Measured at `66266c2a3b`:
- `FilterConditionSchema.safeParse({ stage: { $null: 'x' } })` succeeds;
- `DatasetSchema.safeParse` with `filter: { stage: { $null: 'x' } }`
(also with `$exists: 'false'` and with `$null: null`) succeeds;
- the same save door refuses `$icontains: ''` (#19514).
- So a stored dataset or widget filter carrying the shape saves, and
then answers 400 on every query face: `driver-sql` since #5347, the
analytics door since this PR.
- The named producer is whoever authors a dataset `filter`:
`defineStack` metadata or the Studio.
- `Seam: spec:FilterConditionSchema (checkFilterConditionComparands) →
runtime:driver-sql reduceFilterKey | service-analytics
normalizeWhereComparands`
- Dedupe words: `FilterConditionSchema $null non-boolean admitted at
save` · `dataset filter $exists string saves then 400` ·
`checkFilterConditionComparands boolean flag`
---
_Generated by [Claude
Code](https://claude.ai/code/session_01Evb5jFDZGKQE9KG4jbMfMF)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
1 parent 55daf89 commit 226e00c
4 files changed
Lines changed: 718 additions & 9 deletions
File tree
- .changeset
- packages/services/service-analytics/src
- __tests__
- strategies
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
112 | 112 | | |
113 | 113 | | |
114 | 114 | | |
115 | | - | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
116 | 118 | | |
117 | 119 | | |
118 | 120 | | |
| |||
480 | 482 | | |
481 | 483 | | |
482 | 484 | | |
483 | | - | |
| 485 | + | |
484 | 486 | | |
485 | 487 | | |
486 | 488 | | |
| |||
496 | 498 | | |
497 | 499 | | |
498 | 500 | | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
499 | 512 | | |
500 | 513 | | |
501 | 514 | | |
502 | 515 | | |
503 | 516 | | |
504 | 517 | | |
505 | | - | |
506 | | - | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
507 | 524 | | |
508 | 525 | | |
509 | 526 | | |
| |||
0 commit comments