fix(service-analytics): every authorable filter operator reaches the query (#4128)

Closes the CAUSE behind the $between defect, not just that instance.
`normalizeAnalyticsFilters` skipped any operator missing from its map, and a
skipped predicate does not narrow a query — it widens it: the SQL stays valid
and returns rows the author excluded. Four operators from the spec's
authorable vocabulary sat in that state, and a fifth was mapped wrongly:

- $startsWith / $endsWith were dropped. Both strategies now compile them —
  anchored LIKE on the raw-SQL path, the canonical operators (which every
  driver implements directly) on the ObjectQL path, so an anchored match does
  not depend on regex dialect.
- $null was dropped. It is what the console emits for an "is empty" filter,
  so such a widget showed every row.
- $exists was mapped value-INDEPENDENTLY to `set`, so {$exists: false}
  compiled to IS NOT NULL — the inverse of what it asks. It and $null are now
  resolved explicitly: a key→name map cannot express an operator whose meaning
  flips with its value, which is exactly how that inversion got in.
- $notContains reached ObjectQLStrategy, which had no arm for it and fell to a
  `default` returning a bare value — compiling "does not contain x" as
  "equals x".
- Unknown operators now THROW on both surfaces rather than being dropped
  (normalizer) or reinterpreted as an equality (ObjectQL). #3948's call for
  the same shape.

$or / $not remain skipped — expressing them needs a recursive WHERE builder
rather than the flat array the strategies consume. That gap is declared in the
module doc rather than silent.

filter-operator-coverage.test.ts runs the whole vocabulary against a real
SQLite engine and asserts ROW IDS; six of its cases fail without this change.
A dropped predicate is invisible to the SQL-string assertions the strategies'
other suites use, which is how these survived. service-analytics 413 green.

Closes #4128.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqqZmPS5a4gJGBoCTwipFr

Author: claude

.changeset/analytics-filter-operator-coverage.md

@@ -0,0 +1,42 @@
+---
+"@objectstack/service-analytics": patch
+---
+
+fix(service-analytics): every authorable filter operator now reaches the query (#4128)
+
+Closes the cause behind the `$between` defect rather than just that instance.
+`normalizeAnalyticsFilters` skipped any operator missing from its map, and a
+skipped predicate does not narrow a query — it **widens** it: the compiled SQL
+stays valid and returns rows the author excluded. Four operators from the
+spec's authorable vocabulary sat in that state, plus one that was mapped
+incorrectly.
+
+- **`$startsWith` / `$endsWith`** were dropped entirely. Both strategies now
+  compile them — anchored `LIKE 'x%'` / `LIKE '%x'` on the raw-SQL path, and
+  the canonical `$startsWith` / `$endsWith` operators (which every driver
+  implements directly) on the ObjectQL path, so an anchored match does not
+  depend on regex dialect.
+- **`$null`** was dropped. It is the shape the console emits for an "is empty"
+  / "is not empty" filter, so such a widget was showing every row. Now compiles
+  to `IS NULL` / `IS NOT NULL` per its boolean.
+- **`$exists`** was mapped value-*independently* to `set`, so `{$exists: false}`
+  compiled to `IS NOT NULL` — the exact inverse of what it asks for. It and
+  `$null` are now resolved explicitly, because a key→name map cannot express an
+  operator whose meaning flips with its value.
+- **`$notContains`** reached the ObjectQL strategy, which had no arm for it and
+  fell through to a `default` returning a bare value — compiling "does not
+  contain x" as "**equals** x".
+- **Unknown operators now throw** on both surfaces instead of being silently
+  dropped (normalizer) or reinterpreted as an equality (ObjectQL strategy). An
+  operator outside the vocabulary is a caller error, and a loud one beats a
+  silently widened read — the call driver-memory made for the same shape in
+  #3948.
+
+Still declared as a gap, but no longer a silent one: `$or` / `$not` are skipped,
+since expressing them needs a recursive WHERE builder rather than the flat
+array the strategies consume.
+
+Cover is `filter-operator-coverage.test.ts`, which runs the whole vocabulary
+against a real SQLite engine and asserts **row ids** — six of its cases fail
+without this change. A dropped predicate is invisible to the SQL-string
+assertions the strategies' other suites use, which is how these survived.

packages/services/service-analytics/src/__tests__/filter-operator-coverage.test.ts

@@ -0,0 +1,193 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * Every authorable filter operator reaches the query — the closure of #4128.
+ *
+ * `normalizeAnalyticsFilters` maps the spec's `FilterCondition` vocabulary onto
+ * the internal pipeline form. Whatever it fails to map used to be `continue`d,
+ * and that is not "unsupported": the predicate DISAPPEARS, the compiled SQL
+ * stays valid, and the query returns rows the author excluded. It reads as a
+ * chart drawn over the whole dataset (#3650's symptom) and is invisible to a
+ * test that asserts the emitted SQL string — which is what every other suite
+ * for these strategies does, and why four operators sat broken behind one that
+ * had already been found (`$between`, ADR-0053 D-A3.1).
+ *
+ * So this file asserts ROW IDS, against a real SQLite (`sql.js`, the pure-WASM
+ * engine `driver-sql` itself falls back to — see the note in
+ * `read-scope-sql-conformance.test.ts` for why not `better-sqlite3`). A dropped
+ * predicate cannot hide from it: the row set is simply wrong.
+ *
+ * The table below is the whole authorable vocabulary of `filter.zod.ts`, so a
+ * new operator added to the spec without a home in the pipeline fails here
+ * rather than silently widening a customer's dashboard.
+ */
+
+import { describe, it, expect, beforeAll, afterAll } from 'vitest';
+import type { Cube } from '@objectstack/spec/data';
+import type { AnalyticsQuery, FilterCondition } from '@objectstack/spec/data';
+import type { StrategyContext } from '@objectstack/spec/contracts';
+
+import { NativeSQLStrategy } from '../strategies/native-sql-strategy.js';
+import { normalizeAnalyticsFilters } from '../strategies/filter-normalizer.js';
+
+interface Row {
+  id: string;
+  name: string | null;
+  score: number;
+}
+
+/** `n_null` carries a NULL name — the row the null predicates turn on. */
+const ROWS: Row[] = [
+  { id: 'a_alpha', name: 'alpha-one', score: 10 },
+  { id: 'b_alphex', name: 'alphex-two', score: 20 },
+  { id: 'c_beta', name: 'beta-one', score: 30 },
+  { id: 'd_null', name: null, score: 40 },
+];
+
+const CUBE: Cube = {
+  name: 'ops',
+  title: 'Ops',
+  sql: 'ops',
+  measures: { total: { name: 'total', label: 'Total', type: 'count', sql: '*' } },
+  dimensions: {
+    id: { name: 'id', label: 'Id', type: 'string', sql: 'id' },
+    name: { name: 'name', label: 'Name', type: 'string', sql: 'name' },
+    score: { name: 'score', label: 'Score', type: 'number', sql: 'score' },
+  },
+  public: false,
+} as unknown as Cube;
+
+/**
+ * One case per operator in `filter.zod.ts`'s authorable vocabulary.
+ * `expected` is the row ids the filter MUST return — the full row set means
+ * the predicate went missing.
+ */
+const CASES: Array<{ op: string; filter: FilterCondition; expected: string[]; note?: string }> = [
+  { op: '$eq', filter: { name: { $eq: 'alpha-one' } }, expected: ['a_alpha'] },
+  { op: '$ne', filter: { name: { $ne: 'alpha-one' } }, expected: ['b_alphex', 'c_beta'] },
+  { op: '$gt', filter: { score: { $gt: 20 } }, expected: ['c_beta', 'd_null'] },
+  { op: '$gte', filter: { score: { $gte: 20 } }, expected: ['b_alphex', 'c_beta', 'd_null'] },
+  { op: '$lt', filter: { score: { $lt: 20 } }, expected: ['a_alpha'] },
+  { op: '$lte', filter: { score: { $lte: 20 } }, expected: ['a_alpha', 'b_alphex'] },
+  { op: '$in', filter: { name: { $in: ['alpha-one', 'beta-one'] } }, expected: ['a_alpha', 'c_beta'] },
+  { op: '$nin', filter: { name: { $nin: ['alpha-one'] } }, expected: ['b_alphex', 'c_beta'] },
+  { op: '$between', filter: { score: { $between: [20, 30] } }, expected: ['b_alphex', 'c_beta'], note: '#4128: was dropped → every row.' },
+  { op: '$contains', filter: { name: { $contains: 'one' } }, expected: ['a_alpha', 'c_beta'] },
+  {
+    op: '$notContains',
+    filter: { name: { $notContains: 'one' } },
+    expected: ['b_alphex'],
+    note: 'On the ObjectQL path this had no arm and fell to the default, compiling "does not contain" as an EQUALITY.',
+  },
+  {
+    op: '$startsWith',
+    filter: { name: { $startsWith: 'alpha' } },
+    expected: ['a_alpha'],
+    note: '#4128: was dropped → every row. b_alphex proves the anchor is a prefix, not a substring.',
+  },
+  {
+    op: '$endsWith',
+    filter: { name: { $endsWith: '-one' } },
+    expected: ['a_alpha', 'c_beta'],
+    note: '#4128: was dropped → every row.',
+  },
+  {
+    op: '$null: true',
+    filter: { name: { $null: true } },
+    expected: ['d_null'],
+    note: '#4128: was dropped → every row. This is the shape the console emits for an "is empty" filter.',
+  },
+  { op: '$null: false', filter: { name: { $null: false } }, expected: ['a_alpha', 'b_alphex', 'c_beta'] },
+  { op: '$exists: true', filter: { name: { $exists: true } }, expected: ['a_alpha', 'b_alphex', 'c_beta'] },
+  {
+    op: '$exists: false',
+    filter: { name: { $exists: false } },
+    expected: ['d_null'],
+    note: 'Was mapped value-INDEPENDENTLY to `set`, so it compiled to IS NOT NULL — the exact inverse of what it asks.',
+  },
+  { op: 'implicit equality', filter: { name: 'beta-one' }, expected: ['c_beta'] },
+  { op: 'bare null', filter: { name: null }, expected: ['d_null'] },
+  {
+    op: '$and',
+    filter: { $and: [{ score: { $gte: 20 } }, { name: { $contains: 'one' } }] },
+    expected: ['c_beta'],
+  },
+];
+
+/** Point sql.js at the `.wasm` shipped inside its own package (Node-safe). */
+async function locateWasm(): Promise<((file: string) => string) | undefined> {
+  try {
+    const { createRequire } = await import('node:module');
+    const require = createRequire(import.meta.url);
+    const pkgJsonPath = require.resolve('sql.js/package.json');
+    const { dirname, join } = await import('node:path');
+    const dir = dirname(pkgJsonPath);
+    return (file: string) => join(dir, 'dist', file);
+  } catch {
+    return undefined;
+  }
+}
+
+describe('analytics filters — every authorable operator reaches the query (#4128)', () => {
+  let db: any;
+  let ctx: StrategyContext;
+
+  beforeAll(async () => {
+    const mod: any = await import('sql.js');
+    const initSqlJs = mod.default ?? mod;
+    const locateFile = await locateWasm();
+    const SQL = await initSqlJs(locateFile ? { locateFile } : undefined);
+
+    db = new SQL.Database();
+    db.run(`CREATE TABLE "ops" ("id" TEXT PRIMARY KEY, "name" TEXT, "score" INTEGER);`);
+    const insert = db.prepare(`INSERT INTO "ops" ("id","name","score") VALUES (?,?,?)`);
+    for (const r of ROWS) insert.run([r.id, r.name, r.score]);
+    insert.free();
+
+    ctx = {
+      getCube: (name: string) => (name === 'ops' ? CUBE : undefined),
+      queryCapabilities: () => ({ nativeSql: true, objectqlAggregate: false, inMemory: false }),
+      executeRawSql: async (_object: string, sql: string, params: unknown[]) => {
+        const stmt = db.prepare(sql.replace(/\$\d+/g, '?'));
+        stmt.bind(params as any[]);
+        const out: Record<string, unknown>[] = [];
+        while (stmt.step()) out.push(stmt.getAsObject());
+        stmt.free();
+        return out;
+      },
+    } as StrategyContext;
+  });
+
+  afterAll(() => {
+    db?.close();
+  });
+
+  for (const c of CASES) {
+    it(`${c.op} narrows to the rows it names`, async () => {
+      const result = await new NativeSQLStrategy().execute(
+        { cube: 'ops', measures: ['total'], dimensions: ['id'], where: c.filter } as AnalyticsQuery,
+        ctx,
+      );
+      const got = result.rows.map((r) => String(r.id)).sort();
+      expect(got, c.note).toEqual(c.expected);
+      // Belt and braces: a predicate that silently vanished returns the whole
+      // fixture, which is the one wrong answer that looks like a working query.
+      expect(got.length, `${c.op} matched every row — the predicate was dropped`).toBeLessThan(ROWS.length);
+    });
+  }
+
+  it('an operator outside the vocabulary throws instead of widening the query', () => {
+    // The failure mode this whole file exists to prevent: silently returning
+    // rows the filter excludes. A typo'd or non-spec operator is a caller
+    // error, and a loud one — the same call driver-memory made in #3948.
+    expect(() =>
+      normalizeAnalyticsFilters({ where: { name: { $sortOf: 'alpha' } } }),
+    ).toThrow(/Unsupported filter operator "\$sortOf"/);
+  });
+
+  it('a malformed $between throws rather than binding a half-open guess', () => {
+    expect(() => normalizeAnalyticsFilters({ where: { score: { $between: [10] } } })).toThrow(
+      /two-element/,
+    );
+  });
+});

packages/services/service-analytics/src/strategies/filter-normalizer.ts

@@ -17,23 +17,31 @@
  * spec is honoured: dashboard metadata is authored once in the
  * canonical MongoDB form and the server normalizes at the boundary.
  *
- * # Coverage, stated honestly — an unmapped operator WIDENS the query
+ * # Coverage — a dropped predicate WIDENS the query, so nothing is dropped
  *
- * Dropping a predicate is not "not supporting" it: the compiled SQL stays
- * valid and simply returns more rows, which reads as a chart drawn over the
- * whole dataset (#3650's symptom) and is invisible to any test that asserts
- * the emitted SQL string. So what this maps is a capability claim:
+ * Failing to map an operator is not "not supporting" it: the predicate simply
+ * disappears, the compiled SQL stays valid, and the query returns rows the
+ * author excluded. It reads as a chart drawn over the whole dataset (#3650's
+ * symptom) and is invisible to any test that asserts the emitted SQL string.
+ * `$between`, `$startsWith`, `$endsWith` and `$null` each sat broken that way
+ * (#4128), so what this maps is now a complete capability claim over
+ * `filter.zod.ts`'s authorable vocabulary:
  *
  *   - mapped 1:1 — `$eq` `$ne` `$gt` `$gte` `$lt` `$lte` `$in` `$nin`
- *     `$contains` `$notContains` `$exists`, plus `null` → `notSet`;
- *   - lowered — `$and` (flattened in place), and `$between`, which becomes
- *     its two bounds so each strategy's existing upper-bound handling
- *     applies the calendar-day whole-day rule (see the note at the lowering);
- *   - NOT covered, and silently dropped today — `$startsWith` `$endsWith`
- *     `$null` `$regex`, and the `$or` / `$not` combinators (the latter two
- *     deliberately, pending recursive WHERE building). Tracked in #4128,
- *     which also carries the case for turning the fallback into a throw the
- *     way driver-memory did in #3948.
+ *     `$contains` `$notContains` `$startsWith` `$endsWith`;
+ *   - value-DEPENDENT, so resolved explicitly rather than through the map —
+ *     `$null` and `$exists`, whose meaning flips with their boolean;
+ *   - lowered — `$and` (flattened in place), and `$between`, which becomes its
+ *     two bounds so each strategy's existing upper-bound handling applies the
+ *     calendar-day whole-day rule (see the note at the lowering);
+ *   - anything else THROWS. An operator outside the vocabulary is a caller
+ *     error, and a loud one beats a silently widened read — the call
+ *     driver-memory made for the same shape in #3948.
+ *
+ * The one remaining gap is declared, not silent: the `$or` / `$not`
+ * combinators are still skipped, because expressing them needs a recursive
+ * WHERE builder rather than this flat array. Row-result cover for everything
+ * above lives in `filter-operator-coverage.test.ts`.
  */
 
 export interface NormalizedAnalyticsFilter {
@@ -42,6 +50,14 @@ export interface NormalizedAnalyticsFilter {
   values: string[];
 }
 
+/**
+ * The value-INDEPENDENT operators: the pipeline name depends only on the key.
+ *
+ * `$null` and `$exists` are deliberately absent — their meaning flips with
+ * their boolean value, which a key→name map cannot express. Putting `$exists`
+ * here anyway is what made `{$exists: false}` compile to `IS NOT NULL`, the
+ * exact inverse of what it asks for; both are handled explicitly below.
+ */
 const MONGO_TO_CUBE_OP: Record<string, string> = {
   $eq: 'equals',
   $ne: 'notEquals',
@@ -53,7 +69,8 @@ const MONGO_TO_CUBE_OP: Record<string, string> = {
   $nin: 'notIn',
   $contains: 'contains',
   $notContains: 'notContains',
-  $exists: 'set',
+  $startsWith: 'startsWith',
+  $endsWith: 'endsWith',
 };
 
 /**
@@ -133,8 +150,36 @@ function flattenCondition(cond: Record<string, unknown>, out: NormalizedAnalytic
             out.push({ member: key, operator: 'lte', values: [stringifyForCube(v[1])] });
             continue;
           }
+
+          // The two null predicates read their BOOLEAN, not just their key —
+          // which is why neither can live in MONGO_TO_CUBE_OP. `$null: true`
+          // asks for IS NULL (`notSet`), `$null: false` for IS NOT NULL
+          // (`set`); `$exists` is the mirror image. `$null` is the shape the
+          // console emits for an "is empty" / "is not empty" filter
+          // (`is_null`/`is_not_null` normalise to it in `filter.zod.ts`), so
+          // dropping it silently meant such a widget showed every row.
+          if (opKey === '$null' || opKey === '$exists') {
+            const isNull = opKey === '$null' ? wrapper[opKey] === true : wrapper[opKey] === false;
+            out.push({ member: key, operator: isNull ? 'notSet' : 'set', values: [] });
+            continue;
+          }
+
           const cubeOp = MONGO_TO_CUBE_OP[opKey];
-          if (!cubeOp) continue;
+          if (!cubeOp) {
+            // NEVER drop: a missing predicate does not narrow the query, it
+            // WIDENS it — the compiled SQL stays valid and simply returns rows
+            // the author excluded, which is indistinguishable from a
+            // legitimately broad query and invisible to any test that asserts
+            // the emitted SQL. That failure mode is #3650's, and skipping
+            // unmapped operators is how `$between` reproduced it (#4128).
+            // driver-memory made the same call for the same reason in #3948.
+            throw new Error(
+              `[analytics] Unsupported filter operator "${opKey}" on "${key}". ` +
+              `Supported: ${Object.keys(MONGO_TO_CUBE_OP).join(', ')}, $between, $null, $exists ` +
+              `(and $and; $or/$not are not yet compiled by the analytics strategies). ` +
+              `Dropping it would silently widen the query to rows the filter excludes.`,
+            );
+          }
           const v = wrapper[opKey];
           const values = Array.isArray(v)
             ? v.map(stringifyForCube)

packages/services/service-analytics/src/strategies/native-sql-strategy.ts

@@ -484,6 +484,14 @@ export class NativeSQLStrategy implements AnalyticsStrategy {
     const opMap: Record<string, string> = {
       equals: '=', notEquals: '!=', gt: '>', gte: '>=', lt: '<', lte: '<=',
       contains: 'LIKE', notContains: 'NOT LIKE',
+      startsWith: 'LIKE', endsWith: 'LIKE',
+    };
+    /** The LIKE pattern each string operator wraps its comparand in. */
+    const likePattern: Record<string, (v: string) => string> = {
+      contains: (v) => `%${v}%`,
+      notContains: (v) => `%${v}%`,
+      startsWith: (v) => `${v}%`,
+      endsWith: (v) => `%${v}`,
     };
 
     // Null predicates and the LIKE family read the column as stored — the former
@@ -504,8 +512,11 @@ export class NativeSQLStrategy implements AnalyticsStrategy {
     const sqlOp = opMap[operator];
     if (!sqlOp || !values || values.length === 0) return null;
 
-    if (operator === 'contains' || operator === 'notContains') {
-      params.push(`%${values[0]}%`);
+    // The LIKE family reads the column as stored — a substring/prefix/suffix
+    // match is on the raw text — so it keeps the un-normalised reference.
+    const pattern = likePattern[operator];
+    if (pattern) {
+      params.push(pattern(values[0]));
       return `${rawCol} ${sqlOp} $${params.length}`;
     }