Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .changeset/19757-equality-slot-array-refused.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
---
"@objectstack/spec": minor
---
Expand Down Expand Up @@ -32,7 +32,7 @@
- Every scalar equality comparand is untouched. That includes `null`: `{ field: null }` and `{ field: { $eq: null } }` are the has-no-value predicate.
- A `{ $field }` reference on an equality spelling still lowers to `$eq` and passes.
- A field spec with no `$` key (`{ author: { tags: ['a'] } }`) is still not descended into.
- The schema doors are not touched. `FilterConditionSchema` still parses `{ field: [...] }`, because `FieldOperatorsSchema.$eq` is `z.any()`. A document carrying the shape therefore still publishes, and is refused when it is queried.
- This change does not touch the schema doors. A separate change in this release does: `FilterConditionSchema` and `FieldOperatorsSchema.$eq` now refuse the same shape when a document is saved, with this refusal's sentence (the location is carried by the issue's path instead). Its changeset and the ADR-0087 entry `filter-equality-array-comparand-refused-at-save` describe it.
- `ViewFilterRule` already refused an array on every scalar view operator at authoring time.

## FROM → TO
Expand Down
2 changes: 1 addition & 1 deletion .changeset/19888-analytics-implicit-array.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
---
"@objectstack/service-analytics": minor
---
Expand Down Expand Up @@ -27,4 +27,4 @@

Each list is now handed to the shared face's equality arm before any node is built. Both spellings of one condition therefore get the same refusal, with the same wording, path and `$in` prescription. The draft-data preview runs the same gate, so a drafted chart refuses what the published chart refuses. Before, it compared each row against the list's string form.

Who is affected: nothing in this repository's examples, seeds or docs authors the shape (measured over `examples/**`, `packages/**` and the fenced code in the docs). Stored datasets, dashboard widget filters, report runtime filters and measure filters in a deployment were NOT measured. The authoring schema still admits the shape, so such a document still publishes, and it is refused when it is charted. The refusal names the field and the path. `$ne` with a list is not part of the ruling and is not judged here. The list operators (`$in`, `$nin`, `$between`) keep their lists, and every scalar, `null` included, compiles as before.
Who is affected: nothing in this repository's examples, seeds or docs authors the shape (measured over `examples/**`, `packages/**` and the fenced code in the docs). Stored datasets, dashboard widget filters, report runtime filters and measure filters in a deployment were NOT measured. In this release the authoring schema refuses the shape too, when such a document is saved (a separate change in `@objectstack/spec`, ADR-0087 entry `filter-equality-array-comparand-refused-at-save`). One position is judged only here: a list inside a nested relation, which this door flattens to a dotted member. A document carrying that still publishes, and it is refused when it is charted. The refusal names the field and the path. `$ne` with a list is not part of the ruling and is not judged here. The list operators (`$in`, `$nin`, `$between`) keep their lists, and every scalar, `null` included, compiles as before.
46 changes: 46 additions & 0 deletions .changeset/19889-filter-schema-door-array-equality.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
---
"@objectstack/spec": minor
---

fix(spec)!: a filter carrying an array in the equality slot is refused when it is saved, in the query face's own words (#19889)

**BREAKING** — an accept-set narrowing of published authoring schemas, shipped as `minor` under the repo's launch-window convention for accept-set narrowings. Ruled on #19889 (record 5805248669, letter A). The hand-migration prescription is registered under protocol major 18 as `filter-equality-array-comparand-refused-at-save`.

## What changes

`FilterConditionSchema` now refuses an ARRAY in the equality slot at parse: the implicit form `{ field: [...] }` and the explicit form `{ field: { $eq: [...] } }`, the empty array included, at any depth under `$and` / `$or` / `$not`. `FieldOperatorsSchema.$eq` refuses an array comparand too, and so do its documentation copy `EqualityOperatorSchema.$eq` and the `NormalizedFilter` AST that validates against it.

The comparand-shape face (`assertListComparandShapes`) refuses both shapes on every query, from the equality-slot change earlier in this release. Before this change the schema accepted them, so a dataset or dashboard filter carrying one published clean and then failed every query that used it. Measured on `origin/main` `a0920b42dc`: `DatasetSchema.safeParse` with `filter: { stage: ['won', 'lost'] }` answered `success: true`, and so did a measure `filter` of `{ stage: { $eq: ['won', 'lost'] } }`.

The schema door prints the face's sentence: the field, the received list, and the two operators a list in that slot stood in for. Both doors import it from one builder. The face adds `at where.<field>`. The schema door leaves that out, because the issue's `path` already says where it is (`filter.stage`, `measures.0.filter.stage.$eq`).

Every schema that carries a `FilterCondition` refuses it on parse. That covers the dataset `filter` and measure `filter`, the dashboard widget `filter` and options-source `filter`, the report and joined-report-block `runtimeFilter`, the field `relatedListFilter` and rollup `summaryOperations.filter`, the solution-blueprint summary `filter`, the analytics query `where`, the dataset selection `runtimeFilter`, the query `where` and `having`, the data-engine aggregate call's `having`, the aggregation `filter`, and the query-filter `where`. So `defineStack`, `os validate` and a save through the metadata protocol (`422 INVALID_METADATA`) refuse such a document at the filter's path.

Two request doors parse these carriers, and they now answer before the analytics compiler does. The REST dataset selection (its `runtimeFilter`) and the analytics query body (its `where`) answer `VALIDATION_FAILED` / 400 with the sentence at the field. Before, the compiler answered `INVALID_FILTER` / 400.

## What does NOT change

- **Nothing stored is rewritten, and nothing is dropped.** The parse fails and strips nothing. The read path does not re-validate stored rows, so a stored document keeps loading, and its next save is refused. Such a filter has failed every query since the equality-slot change, so the refusal is a repair.
- **The reach is the face's, and no wider.** A field spec with no `$` key, such as the nested-relation condition `{ account: { region: ['a'] } }`, is not judged, because the face does not judge it either. The analytics `where` door does refuse that shape when a dataset or measure filter is charted.
- **The data-engine calls' `where` option still parses.** Its type is a union whose first arm is an open record. The face refuses the shape when the call runs.
- **`$ne` carrying an array is not judged.**
- The list operators keep their arrays, `$in: []` and `$nin: []` included. Every scalar, `null`, a `Date` and a `{ $field }` reference pass as before.
- **The published JSON Schema cannot state the check.** `z.toJSONSchema()` has no projection for it, so `data/FieldOperators`, `data/EqualityOperator` and `data/NormalizedFilter` still read `{}` at `$eq`. The three sites are declared in `dropped-refinements.baseline.json`.

## FROM → TO

| you wrote | write instead |
|:--|:--|
| `{ stage: ['won', 'lost'] }`, meaning "one of these values" | `{ stage: { $in: ['won', 'lost'] } }` |
| `{ tags: ['a'] }`, meaning "the stored list holds `a`" on a multi-value field | `{ tags: { $contains: 'a' } }` |
| `{ tags: ['a', 'b'] }`, meaning "the stored list holds `a` or `b`" | `{ $or: [{ tags: { $contains: 'a' } }, { tags: { $contains: 'b' } }] }` |
| `{ stage: ['won'] }`, meaning one value | `{ stage: 'won' }` |
| `{ stage: { $eq: [...] } }` | any of the rows above |

## Who is affected, measured

Nothing shipped in this repository carries the shape. A brace-matched scan of every filter-carrier literal (`filter`, `where`, `runtimeFilter`, `having`, `relatedListFilter`) in `packages/**`, `examples/**`, `apps/**`, `content/docs/**` and `skills/**` read 4709 literals across 7785 files and found 20 field entries whose value opens an array. Sixteen are test fixtures, and four are not filter carriers (a realtime subscription filter and a plugin-permission filter). A grep for `$eq` followed by an array found 24 lines: prose, MongoDB aggregation expressions, and one door-refusal conformance row. Deployed datasets, dashboards and reports were NOT measured. Validating each stack, or re-saving each document, finds every instance the surface above lists.

Clause-②: no (narrowing) — nothing is widened. No key is added, removed or renamed, no exported symbol moves, and the operator vocabulary is unchanged. One comparand shape in one slot, which every query door already refused, is now refused on save as well.

<!-- adr-0087: registered filter-equality-array-comparand-refused-at-save -->
11 changes: 9 additions & 2 deletions packages/spec/dropped-refinements.baseline.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
"description": "Shrink-only ledger of every PUBLISHED JSON Schema that is STILL WIDER than the Zod type it was generated from, because a rule written as `.refine()` reaches the runtime and not the file (#18670). `z.toJSONSchema()` has no arm for a `custom` check: a plain record, the same record with a `.refine()`, and the same record with an ABORTING `.refine()` all project byte-identically (measured on zod 4.4.3, the version packages/spec resolves). So a document one of these files ACCEPTS can still be refused at parse time, and an author -- or an AI -- validating against packages/spec/json-schema/** finds out a release later. Each `sites` path is a position under that schema at which a refinement is dropped; the same paths are written onto the artifact itself as `x-dropped-refinements`. Item 2 closed the first patterns: a refinement DECLARED through the closed list in src/shared/refinement-projection.ts is emitted into the published file, reads `projected` rather than `dropped`, and its row LEAVES this ledger in the same PR -- which is why the ledger shrinks and never grows on a repair. Every refinement outside that closed list stays here, and adding an arm to the list is a public-contract decision, not a refactor. Hand-edited on purpose and with no `gen:` script: a generator would let a new gap be admitted by running a command instead of by a decision, which is the silence this ledger exists to end. Adding, removing or moving a site fails packages/spec/scripts/build-schemas.ts until the line moves with it, and the failure prints the corrected entry in full. ⛔ Do not delete or weaken a refinement to shorten this file -- the runtime rule is correct; it is the projection that is silent, and the remedy is to teach the closed list a NAMED pattern, never to drop the rule.",
"measured": {
"zod": "4.4.3",
"publishedSchemasWithDroppedRefinements": 206,
"droppedRefinementSites": 571,
"publishedSchemasWithDroppedRefinements": 207,
"droppedRefinementSites": 574,
"refinementSitesThatDidProject": 369,
"refinementSitesWithNoJsonFormToCompare": 0
},
Expand Down Expand Up @@ -605,6 +605,11 @@
"where.options[1].lazy"
]
},
"data/EqualityOperator": {
"sites": [
"$eq"
]
},
"data/Field": {
"sites": [
"",
Expand All @@ -616,6 +621,7 @@
"sites": [
"$between.items[0]",
"$between.items[1]",
"$eq",
"$in",
"$nin"
]
Expand Down Expand Up @@ -681,6 +687,7 @@
"sites": [
"lazy.$not.options[0].valueType.$between.items[0]",
"lazy.$not.options[0].valueType.$between.items[1]",
"lazy.$not.options[0].valueType.$eq",
"lazy.$not.options[0].valueType.$in",
"lazy.$not.options[0].valueType.$nin"
]
Expand Down
145 changes: 145 additions & 0 deletions packages/spec/src/data/filter-comparand-refusal-text.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.

/**
* [#19889] The words the two filter doors share — the comparand-shape FACE
* (`./filter-comparand-shape.ts`, which every query crosses) and the SCHEMA
* door (`./filter.zod.ts`, which every stored filter crosses on save).
*
* ## Why a module of its own
*
* The 2026-09-24 ruling on #19889 (letter A, record 5805248669) refuses an
* ARRAY in the EQUALITY slot at the schema door "with the SAME remedy text the
* shared compile face emits — one constant, two doors". The face already
* refused it (#19757, ruling 乙). So the text has to live where BOTH doors can
* import it, and neither door can be that place: `filter.zod.ts` imports the
* face, and the face cannot import `filter.zod.ts` back — the cycle the face's
* `LIST_COMPARAND_OPERATORS` docblock records. A module both import breaks the
* tie. It imports nothing.
*
* ## Why it is NOT in the `data` barrel
*
* Nothing here is a contract a consumer calls; the two refusals are, and they
* are already published (`assertListComparandShapes`, and the schemas
* themselves). Exporting the text would widen `@objectstack/spec/data` for no
* reader, so this file stays out of `./index.ts`, as
* `./currency-fraction-digits.ts` does. Consumers read the words where the
* refusals put them: the `Error` message at the face, the issue message at the
* schema door.
*/

/**
* The authoring spellings that lower to `$in`.
*
* The face's `LIST_COMPARAND_OPERATORS` reads its `$in` row from here, so the
* refusal of a malformed `$in` and the equality-slot refusal (which prescribes
* `$in`) name one spelling list. `filter-comparand-shape.test.ts` reconciles
* the list against `AST_OPERATOR_MAP`.
*/
export const IN_OPERATOR_SPELLINGS: readonly string[] = ['in'];

/**
* [#19757] The ARRAY-CONTAINMENT operator the equality-slot refusal prescribes,
* with its authoring spelling: `$contains`, which `FieldOperatorsSchema`
* declares as a MEMBERSHIP test on a `multiple: true` / JSON-stored column
* ("the stored list holds this value"). The other half of the prescription is
* `$in`, read off {@link IN_OPERATOR_SPELLINGS}. Reconciled by pin in
* `filter-comparand-shape.test.ts`.
*/
export const ARRAY_CONTAINMENT_OPERATOR = { op: '$contains', spellings: ['contains'] } as const;

/**
* A short, bounded rendering of the offending value.
*
* Bounded because the value came off the wire and a filter comparand can be
* arbitrarily large; the message is for a human reading a 400, not a dump.
*
* The whole message has a second, harder bound: `rest-server.ts` TRUNCATES a
* declared-4xx message at `CLIENT_MESSAGE_MAX` (500) before it reaches the
* client (#5423). Everything a caller needs in order to act — operator, field,
* received value, position, corrected shape — is therefore front-loaded, and
* `filter-comparand-shape.test.ts` pins the assembled length under that bound so
* a later edit cannot silently push the tail off the wire.
*
* Moved here from the face (#19889) so the schema door renders a received list
* with the same characters the face does.
*/
export function shapePreview(value: unknown): string {
let text: string;
try {
text = JSON.stringify(value) ?? String(value);
} catch {
text = String(value);
}
return text.length > 60 ? `${text.slice(0, 59)}…` : text;
}

/**
* The ONE remedy for an array in the equality slot: the two operators an author
* holding a list was reaching for, by their spec spellings and their authoring
* spellings, then the sentence a caller cannot infer from a status code.
*
* - `$in` — "one of these values" (authoring spelling `in`).
* - `$contains` — "the stored list holds a value" on a multi-value field
* (authoring spelling `contains`), and an `$or` of those for any-of.
*
* The corrected shapes are written WITHOUT the field wrapper and with `…` for
* the value: the field is the subject of the sentence in front of this one,
* and the 500-char client bound buys more as prescription than as a second
* echo of the field name or the received list.
*
* The closing sentence is the face's, and it is kept at the schema door too:
* a refused filter is one that was NOT applied, and the reason for refusing
* rather than dropping it is that a dropped filter shows MORE rows than the
* author asked for — the ruling's own ground for never dropping one silently.
*/
export const ARRAY_EQUALITY_COMPARAND_REMEDY: string =
`For "one of these values" use {"$in": […]} (authoring: ${IN_OPERATOR_SPELLINGS.join(', ')}); ` +
`for "the stored list holds a value" on a multi-value field, ` +
`{"${ARRAY_CONTAINMENT_OPERATOR.op}": "…"} ` +
`(authoring: ${ARRAY_CONTAINMENT_OPERATOR.spellings.join(', ')}), an $or of those for any-of. ` +
`The filter was NOT applied, and an unapplied filter would have returned the ` +
`UNFILTERED result set.`;

/**
* Where the refused array sits, as far as the calling door can see.
*
* - `op: '$eq'` — the explicit spelling. The `field` is optional because one
* door cannot know it: `FieldOperatorsSchema.$eq` is judged inside a
* single field's operator map, whose key belongs to the enclosing record.
* - no `op` — the implicit form `{ field: [...] }`, which only exists as a
* field entry, so the field is always known.
* - `path` — the face's dotted location (`where.stage`). The schema door
* passes none: a zod refinement cannot see where it sits in the document,
* and the issue it raises carries that location as its own `path`.
*/
export type ArrayEqualityComparandSite =
| { readonly op: '$eq'; readonly field?: string; readonly path?: string }
| { readonly op?: undefined; readonly field: string; readonly path?: string };

/**
* The refusal of an ARRAY as an EQUALITY comparand, as both doors print it.
*
* The leading sentence is `driver-memory`'s `arrayComparandError` verbatim, so
* one condition keeps one wording across packages (#5240's rule, applied
* across packages). Then the received list, bounded by {@link shapePreview};
* then the location when the door knows it; then
* {@link ARRAY_EQUALITY_COMPARAND_REMEDY}.
*
* For a given field, value and operator the two doors therefore print the same
* characters, with one difference: the face adds ` at <path>` and the schema
* door does not, because the schema door's issue carries that location as its
* own `path`. `filter-equality-array-schema-door.test.ts` pins exactly that.
*/
export function arrayEqualityComparandMessage(
value: unknown,
site: ArrayEqualityComparandSite,
): string {
const position = site.op
? `Operator "${site.op}"${site.field === undefined ? '' : ` on field "${site.field}"`}`
: `The implicit-equality comparand on field "${site.field}"`;
const location = site.path === undefined ? '' : ` at ${site.path}`;
return (
`${position} requires a single comparable value, but received an array ` +
`(${shapePreview(value)})${location}. ${ARRAY_EQUALITY_COMPARAND_REMEDY}`
);
}
Loading
Loading