Skip to content

Commit 5dbeb7d

Browse files
fix(service-analytics): the ObjectQL strategy hands the engine its null tests as $null, so a $not over a multi-valued lookup gets the engine's rows (#20918) (#21036)
Fixes #20918 Clause-②: yes On the engine-aggregate (ObjectQL) strategy, a `$not` over a multi-valued lookup was refused `400 INVALID_FILTER`, while `engine.find` and the native strategy answered rows. It now gets the engine's rows. The fix is one producer spelling in `ObjectQLStrategy.convertFilter`. The `set` / `notSet` leaves reach the engine as `{ $null: false }` / `{ $null: true }`. They used to arrive as `{ $ne: null }` and the bare `null`, and `driver-sql` refuses both over a JSON column. No driver, engine or spec file is touched. ## Mechanism assumptions, measured All readings use one fixture: a ledger with `title` (text), `owner` (lookup) and `owners` (multiple lookup). The rows are a `['u1']`, b `['u2']`, c `['u2','u1']`, d `['u3']`, and e, which has `owner` and `owners` NULL. The composition is the shipped one: the real `SecurityPlugin`, `ObjectQL` and `SqlDriver` (better-sqlite3), queried as a member. The base is `9b0de7de7`, and "after" is this branch. - **G1, the reference.** `engine.find` for `{ $not: { owners: { $contains: 'u1' } } }` answers **b, d, e**. The single-valued control `{ $not: { owner: { $contains: 'u1' } } }` also answers b, d, e. `engine.aggregate` grouped by `title` answers the same rows for every `where` row below. - **G2, where the guard comes from.** The analytics `where` door adds `{ owners: { $null: false } }` beside the `$contains` inside the `$not` operand, and it does so twice: - the shared lowering, `lowerFilterCondition` rule 3 (`packages/spec/src/data/filter-lowering.ts:445-446`, `guardNullPolarity`'s `requireValue` arm), applied at `filter-normalizer.ts:2467`; - this package's own copy, `nullSafeNegationOperand` (`filter-normalizer.ts:1436`, then `guardFieldEntry` at `:1652`). The guard exists because of SQL's three-valued logic. `NOT (owners LIKE '%u1%')` is UNKNOWN when `owners` is NULL, and a `WHERE` keeps only TRUE, so row e would drop out of a negation that the JS backends answer TRUE for. #5146 ruled the JS answer canonical. A total leaf, `NOT (owners IS NOT NULL AND owners LIKE ...)`, keeps row e. Both copies of the guard become `set` leaves. `ObjectQLStrategy.convertFilter` turned `set` into `{ $ne: null }` at `objectql-strategy.ts:1920` on the base (not `:1872`), and `notSet` into the bare `null` at `:1921`. `notSet` carries the `allowNull` escape of `$not` operands and of `$ne` / `$nin` / `$notContains` (#5298). - **G3, the engine already carries the guard.** `resolveThenLowerWhere` (`packages/objectql/src/engine.ts:1204`) runs the same lowering on `where`, `aggregations[i].filter` and `having`. Measured: `engine.find` hands `driver-sql` `{ $not: { $and: [{ owners: { $null: false } }, { owners: { $contains: 'u1' } }] } }` and answers b, d, e. The NULL row e is IN the answer. That is the guard's intent: the negation answers a row with no value instead of dropping it. `driver-sql` applies `$null` / `$exists` on a JSON column (they are absent from `JSON_COLUMN_INCOMPATIBLE_OPERATORS`). It refuses `$ne` and the bare equality spelling for any comparand, `null` included. - **Which route, from measurement.** "Stop adding the guard" would not serve rows 5 to 7 of the table below, `{ owners: { $null: false } }`, `{ owners: { $exists: true } }` and `{ owners: { $null: true } }`. They carry no guard at all: the author's own null test becomes the same `set` / `notSet` leaf, so the defect is the leaf's spelling, not the guard. That route would also need the shared normalizer to diverge per strategy, because the native compile and the `/analytics/sql` echo read the same tree. The other route, "the guard in a form `driver-sql` applies", serves every row. It changes one producer, and it speaks the lowering's own spelling, so the engine now recognises the guard as one it carries. Measured on the base: the driver received three `owners` guards for the card's filter, one of them added by the engine. After the fix it receives two, and the engine adds none. (The two are this package's lowering and its hand copy, which is #5146's documented idempotent double.) - **G4, the native strategy is unchanged.** Every row below answers the same before and after. The native strategy compiles its own SQL, and `native-sql-strategy.ts` is untouched. - **G5, `Clause-②: yes`.** Widening: 9 `where` rows and 3 measure-filter rows go from 400 to rows on the ObjectQL path, at the cube read and at the dataset door. Narrowing: none measured. Every row answered before is answered with the same rows after: every single-valued row, `$contains` on the multi-valued field, the read scope, and the native strategy on all rows. The 10 re-judged pins below moved only the engine-input spelling. None of their row expectations moved. ## Per row The ObjectQL cell is the same at the cube read (`AnalyticsService.query`, which `POST /api/v1/analytics/query` relays) and at the dataset door (`POST /api/v1/analytics/dataset/query`, `selection.runtimeFilter`). The native strategy answered the same before and after on every row, so one column carries it. | # | filter (`where`) | engine (reference) | native, before = after | ObjectQL before | ObjectQL after | |--:|:--|:--|:--|:--|:--| | 1 | `{ $not: { owners: { $contains: 'u1' } } }` | b, d, e | b, d, e | 400 `INVALID_FILTER` (`$ne`) | **b, d, e** | | 2 | `{ $not: { owners: { $notContains: 'u1' } } }` | a, c | a, c | 400 (bare equality) | **a, c** | | 3 | `{ $not: { $or: [{ owners: { $contains: 'u1' } }, { title: 'b' }] } }` | d, e | d, e | 400 (`$ne`) | **d, e** | | 4 | `{ owners: { $notContains: 'u1' } }` | b, d, e | b, d, e | 400 (bare equality) | **b, d, e** | | 5 | `{ owners: { $null: false } }` | a, b, c, d | a, b, c, d | 400 (`$ne`) | **a, b, c, d** | | 6 | `{ owners: { $exists: true } }` | a, b, c, d | a, b, c, d | 400 (`$ne`) | **a, b, c, d** | | 7 | `{ owners: { $null: true } }` | e | e | 400 (bare equality) | **e** | | 8 | `{ owners: { $ne: null } }` | 400 (`$ne`) | a, b, c, d | 400 (`$ne`) | a, b, c, d (see Acceptance notes) | | 9 | `{ owners: null }` | 400 (bare equality) | e | 400 (bare equality) | e (see Acceptance notes) | | 10 | `{ owners: { $contains: 'u1' } }` | a, c | a, c | a, c | a, c | | 11 | `{ $not: { owner: { $contains: 'u1' } } }` (single-valued control) | b, d, e | b, d, e | b, d, e | b, d, e | | 12 | `{ $not: { owner: { $eq: 'u1' } } }` | b, d, e | b, d, e | b, d, e | b, d, e | | 13 | `{ $not: { title: { $contains: 'a' } } }` | b, c, d, e | b, c, d, e | b, c, d, e | b, c, d, e | | 14 | `{ owner: { $ne: null } }` | a, b, c, d | a, b, c, d | a, b, c, d | a, b, c, d | | 15 | `{ owner: null }` | e | e | e | e | | 16 | `{ owner: { $notContains: 'u1' } }` | b, d, e | b, d, e | b, d, e | b, d, e | Two more positions were measured, as the dataset door's count over all five rows: | position | filter | `engine.aggregate` filter | `engine.find` rows | native | ObjectQL before | ObjectQL after | |:--|:--|:--|:--|:--|:--|:--| | measure `filter` | `{ $not: { owners: { $contains: 'u1' } } }` | 5 | 3 | 3 | 400 | **3** | | measure `filter` | `{ owners: { $null: false } }` | 4 | 4 | 4 | 400 | **4** | | measure `filter` | `{ owners: { $notContains: 'u1' } }` | 5 | 3 | 3 | 400 | **3** | | measure `filter` | single-valued `$null: false` / `null` / `$not $contains` | 4 / 1 / 3 | same | same | same | same | | host read scope | `$not $contains` multi / `$null: false` multi / `owner $ne null` | | b,d,e / a,b,c,d / a,b,c,d | same | same | same | A read scope reaches the engine as written, not through `convertFilter`, so it was served before and is unchanged. ## What changed - `packages/services/service-analytics/src/strategies/objectql-strategy.ts`, `convertFilter`: `set` becomes `{ $null: false }` and `notSet` becomes `{ $null: true }`. The comment records why. - `.changeset/20918-objectql-not-multivalue-served.md`: `@objectstack/service-analytics` `minor`, `Clause-②: yes`. The `notSet` arm is fixed beside the `set` arm. It is the same defect class in the same function, and the card's scope ("a `$not` over a multi-valued column") includes row 2, which only the `notSet` arm serves. ## Pins, red first (`f45d92d43`) - `packages/rest/src/analytics-not-multivalue-null-guard.test.ts`: the shipped composition, both strategies, the cube read and the dataset door, against `engine.find`. It covers rows 1, 2, 4, 5 and 7, plus the single-valued control (row 11), and each row's reference rows are asserted too. Base: 5 red (ObjectQL, the multi-valued rows) and 7 green (native, and the control). Now 12 of 12 green. - `packages/services/service-analytics/src/__tests__/objectql-null-predicate-json-column.test.ts`: the ObjectQL face over a real `ObjectQL` on `SqliteWasmDriver`, against `engine.find`. It covers the same five multi-valued rows and three single-valued controls, plus the engine-input spelling of `set` / `notSet`. Base: 6 red and 3 green. Now 9 of 9 green. ## Pins re-judged Ten cases in five existing files recorded the old engine-input spelling, or read it through a test stand-in engine. Each was respelled, and no row expectation moved: | file | cases | disposition | |:--|--:|:--| | `filter-normalizer-not-null-safe.test.ts` | 2 | engine input respelled to `{ $null: false }` / `{ $null: true }` | | `filter-value-type-fidelity.test.ts` | 1 | respelled; its strict-equality stand-in engine reads the `$null` predicate | | `objectql-contains-canonical-operator.test.ts` | 2 | respelled; its regex-face stand-in reads `{ $null: flag }` where it read the bare `null` | | `objectql-daterange.test.ts` | 1 | respelled; its bridge reads `$null` | | `where-boolean-flag-refusal.test.ts` | 4 | the control families' engine-input table respelled; their real-engine rows are unchanged | ## Ablations, predicted before running The ablations ran at `b6c4893fd`. Between that commit and the head, `b5a2e4481`, nothing under `packages/services` or `packages/rest` changed. Each ablation: - mutated the committed `objectql-strategy.ts` through `scripts/ablation-replace.mjs` (anchor 1 to 0, blob `63ce11d10a04` to `75b05cae8989` / `aeab45c348ec` / `3f893a607ea2`); - rebuilt `@objectstack/service-analytics`, and passed `ablation-dist-preflight` with the old arm present in `dist/`; - ran the unit files and the route pin; - restored, proving the blob equal to HEAD `63ce11d10a04`, `git diff HEAD` empty and the tree clean, then rebuilt and passed `ablation-dist-preflight --absent` for each marker. Every observed count equals its prediction: | ablation | mutation | unit (the 6 touched files) | route pin (12) | |:--|:--|:--|:--| | A0 | both arms back to `{ $ne: null }` / `null` | 16 red. The full package suite also went 16 red and nothing else (151 files, 3451 passed) | 5 red (ObjectQL rows 1, 2, 4, 5, 7) | | A1 | the `set` arm only | 7 red: new pin 3, not-null-safe 2, boolean-flag 2 (`$null: false`, `$exists: true`) | 2 red (rows 1, 5) | | A2 | the `notSet` arm only | 13 red: new pin 4, not-null-safe 1, type-fidelity 1, contains-canonical 2, daterange 1, boolean-flag 4 | 3 red (rows 2, 4, 7) | ## Verification, at `b5a2e4481` (main `2f2fa11d7` merged, no rebase; it touched nothing on this surface) - `@objectstack/service-analytics`: `test` exit 0, 151 files, 3467 passed. `typecheck` exit 0, with 151 of 151 test files in the tsc program (`--listFiles`). - `@objectstack/rest`: `typecheck` exit 0 (`tsc --noEmit` and `check:test-typecheck`). Its 16 `analytics-*.test.ts` files: 225 passed and 3 skipped. - Consumer sweep, narrowed to the files that load this package: `@objectstack/runtime` `analytics-*` plus `cross-field-refusal-operand-withhold`, 5 files, 38 passed and 4 skipped. - Gates: `dispatch-gates --commands` derived 62. All 62 were run as one locked sequential script, plus the 4 roster families (`check-changeset-fixed`, `check:authz-resolver`, `check:error-code-casing`, `check:filter-alias-parity`), and each exit code was captured before any pipe. - 61 exit 0. `check:type-check-debt` first exited 3 (prerequisite: `@objectstack/runtime` and `@objectstack/driver-turso` unbuilt). It was re-run after building that closure and exited 0: "none above its recorded number". - `check:dual-build-cjs-loads` exited 3 (prerequisite: 38 packages with no `dist/`, a whole-workspace build). It is NOT MEASURED here. - `dispatch-gates --ran`: 62 derived, 61 run, 1 NOT-MEASURED, 0 UNRUN. - The 4 roster families exit 0. - Lint, narrowed and proved at `b5a2e4481`. The population is the 8 changed `.ts` files, and none is ignored by eslint's own config (`isPathIgnored` false for all 8). `parserOptions.project` and `projectService` are unset for all 8, so no type-aware rule runs and no untouched file's verdict can move. `eslint --no-inline-config --format json` over them: 8 files, 0 errors, 0 warnings. ## Acceptance notes - **Rows 8 and 9 differ from the engine.** `{ owners: { $ne: null } }` and the bare `{ owners: null }` are null tests at the analytics door: since #5332 the normalizer reads them as `set` / `notSet`. So both strategies answer them now (the native one already did). `engine.find` refuses them, because `driver-sql`'s JSON-column gate reads the operator, not the comparand. Triage ruled out a driver special case for `$ne: null` on a JSON column, so this difference stays. `{ $null: false }` / `{ $null: true }` is the spelling every face reads alike. - **Measure `filter` against the engine's aggregation evaluator.** For the multi-valued `$not $contains` and `$notContains` measure filters, the analytics answer (3) equals `engine.find` and the native strategy. `engine.aggregate`'s per-aggregation `filter` answers 5. That is the engine's aggregation evaluator, the position of #20873 (`domain:engine`), which triage kept as a neighbour. #20873 is not addressed here. - **Surface.** The claim named new pin files. This PR also edits the five existing pin files above, all under `service-analytics/src/__tests__/`, because their assertions recorded the spelling the fix changes. None of them is a file of #20912 or #20889. - **The G3 wording.** The PM's assumption read "excludes NULL rows". Measured, the guard makes the negation ANSWER the NULL row (row e is in b, d, e), and the engine already does that. - **NOT MEASURED:** - PostgreSQL and MySQL; - driver-memory, driver-mongodb and driver-turso behind the ObjectQL strategy. They compile `$null` on every path today because the engine's own lowering emits it, but that is reasoning, not a measurement; - an external object's datetime column on the legacy-repair path, where `{ f: null }` and `{ f: { $null: true } }` take different `driver-sql` arms; - the dogfood and integration lanes, the full `@objectstack/rest` `local` suite, the whole-workspace typecheck, and `check:dual-build-cjs-loads`. All are left to CI. --- _Generated by [Claude Code](https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent d830d71 commit 5dbeb7d

9 files changed

Lines changed: 535 additions & 29 deletions
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
'@objectstack/service-analytics': minor
3+
---
4+
5+
fix(service-analytics): on the engine-aggregate path, a `$not`, `$notContains` or null test over a multi-valued lookup now gets the engine's rows instead of `400 INVALID_FILTER` (#20918)
6+
7+
Clause-②: yes
8+
9+
**What changed.** `POST /api/v1/analytics/query` and `POST /api/v1/analytics/dataset/query`, when served by the engine-aggregate strategy (a query with a granularity, or a host with no raw SQL), used to refuse these filters on a SQL driver when the field is a multi-valued lookup (or any other JSON-stored multi-value field). The engine's `find()` and the native-SQL strategy answered them:
10+
11+
- `{ $not: { owners: { $contains: 'u1' } } }`, and any `$not` whose operand tests such a field;
12+
- `{ owners: { $notContains: 'u1' } }`;
13+
- `{ owners: { $null: false } }`, `{ owners: { $exists: true } }`, `{ owners: { $null: true } }`, and the same tests in a dataset measure's own `filter`.
14+
15+
Each now answers the rows the native-SQL strategy answers. Where `engine.find()` serves the same filter, those are its rows too.
16+
17+
**Why.** The analytics `where` door adds a NULL test beside each leaf of a `$not` operand, so that a row holding no value is still answered by the negation. It adds a NULL alternative to the negative-polarity operators too. The engine-aggregate strategy passed both tests to the engine as `{ $ne: null }` and the bare `{ field: null }`. `driver-sql` refuses both spellings over a JSON column, so the whole filter was refused. They now reach the engine as `{ $null: false }` and `{ $null: true }`. The engine's own filter lowering writes the same tests in those spellings for every driver, and `driver-sql` applies them on a JSON column.
18+
19+
**Unchanged.** Every filter on a single-valued field gets the same rows as before. The native-SQL strategy and the `POST /api/v1/analytics/sql` echo are unchanged: they compile their own SQL. A read scope is unchanged too.
20+
21+
**One difference from the engine remains.** `{ owners: { $ne: null } }` and the bare `{ owners: null }` are null tests at the analytics door. Both strategies now answer them, the native strategy as before. `engine.find()` refuses them, because `driver-sql` reads `$ne` and the bare equality as value comparisons on a JSON column. `{ $null: false }` / `{ $null: true }` is the spelling both read the same way.
Lines changed: 277 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,277 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* [#20918] A `$not` over a multi-valued lookup gets the engine's rows on BOTH
5+
* analytics strategies, at the cube read and at the dataset door.
6+
*
7+
* The `where` door makes every leaf of a `$not` operand total before negating
8+
* it (#5146): `{ $not: { owners: { $contains: 'u1' } } }` is lowered with a
9+
* `{ owners: { $null: false } }` conjunct beside the `$contains`, so a row
10+
* whose `owners` holds no value is answered by the negation instead of falling
11+
* into SQL's UNKNOWN. The engine-aggregate strategy handed that guard to the
12+
* engine as `{ $ne: null }`, and the negative-polarity escape (#5298) as the
13+
* bare `{ owners: null }`. `driver-sql` refuses both over a multi-valued
14+
* lookup's JSON column, so the engine-aggregate path answered
15+
* `400 INVALID_FILTER` where the engine and the native strategy answer rows.
16+
* The guard now reaches the engine as `{ $null: false }` / `{ $null: true }`,
17+
* the spellings the engine's own lowering emits for the same guard.
18+
*
19+
* The composition is the shipped one: `SecurityPlugin` over a real `ObjectQL`
20+
* on a real `SqlDriver` (SQLite), and `AnalyticsServicePlugin` over the same
21+
* engine as its `'data'` service. Two compositions, one per strategy:
22+
*
23+
* - `native` — the plugin's own capabilities, so `NativeSQLStrategy` answers
24+
* (it compiles its own SQL, and its answers are unchanged by this fix);
25+
* - `objectql` — the capabilities narrowed to the engine-aggregate path, so
26+
* `ObjectQLStrategy` answers every query.
27+
*
28+
* Two faces per composition: the cube read (`AnalyticsService.query`, what
29+
* `POST /api/v1/analytics/query` relays) over the object's inferred cube, and
30+
* the dataset door through this package's own route
31+
* (`POST /api/v1/analytics/dataset/query`, the caller's filter as
32+
* `selection.runtimeFilter`).
33+
*
34+
* ## Measured on the base (`9b0de7de7`), this fixture
35+
*
36+
* | `where` | `engine.find` | native | objectql (base) |
37+
* |:--|:--|:--|:--|
38+
* | `{ $not: { owners: { $contains: 'u1' } } }` | b, d, e | b, d, e | 400 `INVALID_FILTER` (`$ne`) |
39+
* | `{ $not: { owners: { $notContains: 'u1' } } }` | a, c | a, c | 400 (bare equality) |
40+
* | `{ owners: { $notContains: 'u1' } }` | b, d, e | b, d, e | 400 (bare equality) |
41+
* | `{ owners: { $null: false } }` | a, b, c, d | a, b, c, d | 400 (`$ne`) |
42+
* | `{ owners: { $null: true } }` | e | e | 400 (bare equality) |
43+
* | `{ $not: { owner: { $contains: 'u1' } } }` (single-valued) | b, d, e | b, d, e | b, d, e |
44+
*
45+
* The engine's answer, computed in the same test over the same rows, is the
46+
* reference for every cell. SQLite only: the strategy's spelling is the
47+
* change, and the driver that refused it is `driver-sql`.
48+
*/
49+
50+
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest';
51+
import type { FilterCondition } from '@objectstack/spec/data';
52+
import { PermissionSetSchema } from '@objectstack/spec/security';
53+
import { ObjectQL } from '@objectstack/objectql';
54+
import { SqlDriver } from '@objectstack/driver-sql';
55+
import { SecurityPlugin } from '@objectstack/plugin-security';
56+
import { AnalyticsServicePlugin, type AnalyticsService } from '@objectstack/service-analytics';
57+
import { RestServer } from './rest-server';
58+
59+
const OBJECT = 'rest_an_notmv_ledger';
60+
const OWNER = 'rest_an_notmv_owner';
61+
62+
const SYS_CTX = { isSystem: true, userId: 'usr_system' };
63+
64+
const MEMBER_SET = PermissionSetSchema.parse({
65+
name: 'member_default',
66+
label: 'Member',
67+
objects: { '*': { allowRead: true, allowCreate: true, allowEdit: true, allowDelete: true } },
68+
});
69+
70+
const MEMBER_CTX = { userId: 'usr_member', positions: [], permissions: [MEMBER_SET.name], posture: 'MEMBER' };
71+
72+
const OWNERS = [
73+
{ id: 'u1', region: 'NA' },
74+
{ id: 'u2', region: 'EU' },
75+
{ id: 'u3', region: 'AP' },
76+
];
77+
/** `e` holds no value in either relation: the row the `$not` guard is about. */
78+
const ROWS = [
79+
{ id: 'd1', title: 'a', owner: 'u1', owners: ['u1'] },
80+
{ id: 'd2', title: 'b', owner: 'u2', owners: ['u2'] },
81+
{ id: 'd3', title: 'c', owner: 'u1', owners: ['u2', 'u1'] },
82+
{ id: 'd4', title: 'd', owner: 'u3', owners: ['u3'] },
83+
{ id: 'd5', title: 'e', owner: null, owners: null },
84+
];
85+
const TITLE_OF = new Map(ROWS.map((r) => [r.id, r.title]));
86+
87+
/**
88+
* Conditions the engine SERVES, with the rows it answers on this fixture
89+
* (asserted too, so a moved reference cannot pass silently). Every face must
90+
* give the engine's rows.
91+
*/
92+
const SERVED: ReadonlyArray<readonly [string, FilterCondition, string[]]> = [
93+
['$not over $contains, multi-valued', { $not: { owners: { $contains: 'u1' } } }, ['b', 'd', 'e']],
94+
['$not over $notContains, multi-valued', { $not: { owners: { $notContains: 'u1' } } }, ['a', 'c']],
95+
['$notContains, multi-valued', { owners: { $notContains: 'u1' } }, ['b', 'd', 'e']],
96+
['has a value, multi-valued', { owners: { $null: false } }, ['a', 'b', 'c', 'd']],
97+
['has no value, multi-valued', { owners: { $null: true } }, ['e']],
98+
['CONTROL $not over $contains, single-valued', { $not: { owner: { $contains: 'u1' } } }, ['b', 'd', 'e']],
99+
];
100+
101+
/** The inline dataset the dataset door queries: the ledger, grouped by title. */
102+
const DATASET = {
103+
name: 'not_multivalue_inline',
104+
label: 'Not multi-valued inline',
105+
object: OBJECT,
106+
dimensions: [{ name: 'title_dim', field: 'title', type: 'string' }],
107+
measures: [{ name: 'row_count', aggregate: 'count' }],
108+
};
109+
110+
const quiet: any = { debug() {}, info() {}, warn() {}, error() {}, child() { return quiet; } };
111+
112+
function createMockServer() {
113+
const noop = () => {};
114+
return { get: noop, post: noop, put: noop, delete: noop, patch: noop, use: noop, listen: async () => {}, close: async () => {} };
115+
}
116+
117+
function mockProtocol() {
118+
return {
119+
getDiscovery: async () => ({ version: 'v0', routes: { data: '', metadata: '' } }),
120+
getMetaTypes: async () => [],
121+
getMetaItems: async () => [],
122+
};
123+
}
124+
125+
function makeRes() {
126+
const res: any = {
127+
statusCode: 200,
128+
body: undefined as any,
129+
header: () => res,
130+
status: (code: number) => { res.statusCode = code; return res; },
131+
json: (body: unknown) => { res.body = body; return res; },
132+
end: () => res,
133+
};
134+
return res;
135+
}
136+
137+
type Thrown = { code?: string; status?: number; statusCode?: number; message?: string } | null;
138+
const envelopeOf = (e: Thrown) => ({ code: e?.code, status: e?.status ?? e?.statusCode });
139+
const titlesOf = (rows: ReadonlyArray<Record<string, unknown>>, key: string) =>
140+
rows.map((r) => String(r[key])).sort();
141+
142+
interface Harness {
143+
engine: ObjectQL;
144+
service: AnalyticsService;
145+
dataset: (runtimeFilter: FilterCondition) => Promise<{ status: number; body: any }>;
146+
}
147+
148+
/** Boot the shipped composition; `caps` narrows the strategy set to the engine-aggregate path. */
149+
async function boot(caps?: 'objectql'): Promise<Harness> {
150+
const engine = new ObjectQL({ logger: quiet } as any);
151+
engine.registerDriver(
152+
new SqlDriver({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true } as any),
153+
true,
154+
);
155+
await engine.init();
156+
engine.registerApp({
157+
id: 'com.objectstack.qa.analytics-not-multivalue-20918',
158+
name: 'Analytics $not over a multi-valued lookup',
159+
version: '1.0.0',
160+
type: 'plugin',
161+
scope: 'system',
162+
objects: [
163+
{
164+
name: OWNER,
165+
label: 'Owner',
166+
sharingModel: 'public_read_write',
167+
fields: { region: { name: 'region', type: 'text' } },
168+
},
169+
{
170+
name: OBJECT,
171+
label: 'Ledger',
172+
sharingModel: 'public_read_write',
173+
fields: {
174+
title: { name: 'title', type: 'text' },
175+
owner: { name: 'owner', type: 'lookup', reference: OWNER },
176+
owners: { name: 'owners', type: 'lookup', reference: OWNER, multiple: true },
177+
},
178+
},
179+
],
180+
} as never);
181+
await engine.syncSchemas();
182+
183+
const services: Record<string, unknown> = {
184+
manifest: { register: vi.fn() },
185+
objectql: engine,
186+
data: engine,
187+
metadata: {
188+
get: async (_type: string, name: string) => engine.getSchema(name) ?? null,
189+
list: async () => [MEMBER_SET],
190+
},
191+
};
192+
const ctx: any = {
193+
logger: quiet,
194+
hook: () => {},
195+
registerService: (name: string, svc: unknown) => { services[name] = svc; },
196+
replaceService: (name: string, svc: unknown) => { services[name] = svc; },
197+
getService: (name: string) => {
198+
if (!(name in services)) throw new Error(`service not registered: ${name}`);
199+
return services[name];
200+
},
201+
};
202+
const security = new SecurityPlugin({ fallbackPermissionSet: 'member_default' });
203+
await security.init(ctx);
204+
await security.start(ctx);
205+
vi.spyOn((engine as unknown as { logger: { warn: () => void } }).logger, 'warn').mockImplementation(() => undefined);
206+
207+
await engine.insert(OWNER, OWNERS.map((r) => ({ ...r })), { context: SYS_CTX } as never);
208+
await engine.insert(OBJECT, ROWS.map((r) => ({ ...r })), { context: SYS_CTX } as never);
209+
210+
await new AnalyticsServicePlugin(
211+
caps === 'objectql'
212+
? { queryCapabilities: () => ({ nativeSql: false, objectqlAggregate: true, inMemory: false }) }
213+
: {},
214+
).init(ctx);
215+
const service = services.analytics as AnalyticsService;
216+
217+
const rest = new RestServer(
218+
createMockServer() as any, mockProtocol() as any, { api: { requireAuth: false } } as any,
219+
undefined, undefined, undefined, undefined, undefined, undefined, undefined,
220+
undefined, undefined, undefined, undefined,
221+
async () => service,
222+
);
223+
(rest as any).resolveExecCtx = async () => MEMBER_CTX;
224+
rest.registerRoutes();
225+
const route = rest.getRoutes().find((r: any) => r.method === 'POST' && r.path === '/api/v1/analytics/dataset/query');
226+
expect(route).toBeDefined();
227+
const dataset = async (runtimeFilter: FilterCondition) => {
228+
const res = makeRes();
229+
const selection = { measures: ['row_count'], dimensions: ['title_dim'], runtimeFilter };
230+
const body = JSON.parse(JSON.stringify({ dataset: DATASET, selection }));
231+
await route!.handler({ method: 'POST', params: {}, headers: {}, body, query: {} } as any, res);
232+
return { status: res.statusCode, body: res.body };
233+
};
234+
return { engine, service, dataset };
235+
}
236+
237+
/** The engine's answer for a `where`, as the member: its titles, or its refusal. */
238+
async function engineAnswer(engine: ObjectQL, where: FilterCondition) {
239+
return engine.find(OBJECT, { where, context: MEMBER_CTX } as never).then(
240+
(rows: any[]) => ({ titles: rows.map((r) => TITLE_OF.get(r.id) as string).sort() }),
241+
(e: Thrown) => ({ refused: envelopeOf(e) }),
242+
);
243+
}
244+
245+
const cubeQuery = (where: FilterCondition) =>
246+
({ cube: OBJECT, measures: ['count'], dimensions: ['title'], where }) as never;
247+
248+
for (const strategy of ['native', 'objectql'] as const) {
249+
describe(`[#20918] $not over a multi-valued lookup answers the engine's rows — ${strategy} composition`, () => {
250+
let h: Harness;
251+
252+
beforeAll(async () => {
253+
h = await boot(strategy === 'objectql' ? 'objectql' : undefined);
254+
}, 60_000);
255+
256+
afterAll(async () => {
257+
try { await h?.engine.destroy(); } catch { /* noop */ }
258+
});
259+
260+
for (const [label, where, rows] of SERVED) {
261+
it(`${label}: the cube read and the dataset door answer the engine's rows`, async () => {
262+
const reference = await engineAnswer(h.engine, where);
263+
expect(reference, `${label}: the engine`).toEqual({ titles: rows });
264+
265+
const cube = await h.service.query(cubeQuery(where), MEMBER_CTX as never).then(
266+
(r) => ({ titles: titlesOf(r.rows, 'title') }),
267+
(e: Thrown) => ({ refused: envelopeOf(e), message: e?.message }),
268+
);
269+
expect(cube, `${label}: the cube read`).toEqual(reference);
270+
271+
const ds = await h.dataset(where);
272+
expect(ds.status, `${label}: the dataset door ${JSON.stringify(ds.body)}`).toBe(200);
273+
expect(titlesOf(ds.body.rows, 'title_dim'), `${label}: the dataset door`).toEqual(rows);
274+
});
275+
}
276+
});
277+
}

‎packages/services/service-analytics/src/__tests__/filter-normalizer-not-null-safe.test.ts‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -601,7 +601,8 @@ describe('[#5325] analytics `where` — NULL-safe `$not` and the boolean identit
601601
// The guard travels as one more conjunct inside the `$not`, so any driver
602602
// behind the engine — NULL-safe or not — admits the same rows. Rendering it
603603
// only in the SQL strategy would have made the answer depend on the driver.
604-
expect(JSON.stringify(lastEngineFilter)).toContain('$ne');
604+
// [#20918] It travels in the engine's own spelling, `{ $null: false }`.
605+
expect(JSON.stringify(lastEngineFilter)).toContain('{"stage":{"$null":false}}');
605606
expect(JSON.stringify(lastEngineFilter)).toContain('$not');
606607
});
607608

@@ -890,13 +891,15 @@ describe('[#5325] analytics `where` — NULL-safe `$not` and the boolean identit
890891

891892
it('the ObjectQL path hands the engine a null predicate, not `\'\'`', async () => {
892893
expect(await engineIds({ stage: { $eq: null } })).toEqual(['3', '4']);
893-
// `convertFilter` maps `notSet` to a bare `null` — `{stage: null}`, the
894-
// spelling every driver reads as IS NULL. It used to receive
895-
// `{stage: ''}` (via `coerceFilterValueForObjectQL('')`), i.e. the empty
896-
// string compared against stored `null` — never a match on any driver.
897-
expect(lastEngineFilter).toEqual({ stage: null });
894+
// `convertFilter` maps `notSet` to `{ $null: true }` and `set` to
895+
// `{ $null: false }`, the engine's own null predicates (#20918; they were
896+
// the bare `null` and `{ $ne: null }`, which `driver-sql` refuses over a
897+
// JSON column). It used to receive `{stage: ''}` (via
898+
// `coerceFilterValueForObjectQL('')`), i.e. the empty string compared
899+
// against stored `null` — never a match on any driver.
900+
expect(lastEngineFilter).toEqual({ stage: { $null: true } });
898901
expect(await engineIds({ stage: { $ne: null } })).toEqual(['1', '2']);
899-
expect(lastEngineFilter).toEqual({ stage: { $ne: null } });
902+
expect(lastEngineFilter).toEqual({ stage: { $null: false } });
900903
});
901904

902905
it('the echoed display SQL renders the predicate it executes', async () => {

‎packages/services/service-analytics/src/__tests__/filter-value-type-fidelity.test.ts‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -605,9 +605,14 @@ function makeEngine(captured: Array<Record<string, unknown> | undefined>) {
605605
): Promise<Array<Record<string, unknown>>> => {
606606
captured.push(options.filter);
607607
const filtered = ENGINE_ROWS.filter((row) =>
608-
Object.entries(options.filter ?? {}).every(
609-
([field, cond]) => (row as Record<string, unknown>)[field] === cond,
610-
),
608+
Object.entries(options.filter ?? {}).every(([field, cond]) => {
609+
const stored = (row as Record<string, unknown>)[field];
610+
// [#20918] The null predicate, in the engine's own `{ $null: flag }`
611+
// spelling: it asks about presence, never compares a value.
612+
const flag = (cond as { $null?: unknown } | null)?.$null;
613+
if (typeof flag === 'boolean') return (stored === null) === flag;
614+
return stored === cond;
615+
}),
611616
);
612617
return [{ order_count: filtered.length }];
613618
};
@@ -652,9 +657,10 @@ describe('[#5526] analytics engine path — the comparand reaches engine.aggrega
652657

653658
it('a real null comparand is still the null predicate, not a value', async () => {
654659
const { captured, result } = await run(null);
655-
// `convertFilter` maps `notSet` to a bare `null` — the spelling every driver
656-
// reads as IS NULL (#5332 / #5525), reached without entering `values`.
657-
expect(captured[0]).toEqual({ code: null });
660+
// `convertFilter` maps `notSet` to `{ $null: true }` — the engine's own IS
661+
// NULL predicate (#5332 / #5525; the bare `null` before #20918), reached
662+
// without entering `values`.
663+
expect(captured[0]).toEqual({ code: { $null: true } });
658664
expect(result.rows).toEqual([{ order_count: 1 }]);
659665
});
660666
});

0 commit comments

Comments
 (0)