Merge remote-tracking branch 'origin/main' into claude/issue-20437-replica-needs-syncurl

Author: claude

.changeset/20309-number-arm-numeric-string-grammar.md

@@ -0,0 +1,92 @@
+---
+"@objectstack/objectql": minor
+---
+
+fix(objectql)!: a number, currency, percent, rating, slider or progress field reads a string by the platform's numeric grammar and stores the number it denotes (#20309)
+
+Clause-②: no (narrowing)
+
+**BREAKING**: shipped as `minor` under the launch-window convention
+(`check-changeset-no-major` refuses `major` until GA; breaking-ness is carried by
+this banner and the ADR-0087 disposition below, never by the level). The
+narrowing: a string that `Number()` reads as a finite number but the platform's
+numeric grammar does not is now refused with `400 VALIDATION_FAILED` /
+`invalid_number`. It used to be accepted.
+
+**What a caller sees, before → after.** One of these strings written to one of
+those fields: `201`, stored as sent (memory kept the string; SQLite kept
+`'0x10'` as TEXT and the others as numbers by column affinity) → `400
+VALIDATION_FAILED` with the field code `invalid_number`, nothing stored. The
+REST create, batch, update and updateMany routes all answer it, and `validate`
+(the dry run) predicts it. The forms:
+
+- a radix literal: `'0x10'`, `'0X1A'`, `'0o17'`, `'0b101'`;
+- a whitespace-padded number: `' 12 '`, `'12\n'`, `'\t-3'`;
+- a spelling that is not a JSON number: `'+5'`, `'.5'`, `'5.'`, `'007'`.
+
+The fix, when a write is refused: send a JS number, or the number's plain JSON
+spelling — `'16'`, `'12'`, `'-3'`, `'5'`, `'0.5'`, `'7'`. `String(n)` of any
+finite number always qualifies, exponent forms included (`'1e-7'`, `'1e+21'`).
+
+## What was wrong
+
+This is the separate change the earlier #20309 note (arrays, booleans and
+objects refused) left open. The record validator judged a string by `Number()`
+while the write carried the string itself, so an accepted string reached the
+driver as sent:
+
+- **memory** stored `'12'` as the string `'12'` and read it back as a string;
+- **SQLite** stored `'0x10'` as the TEXT `'0x10'` (read back as `16`), and the
+  other accepted strings as numbers through the column's affinity.
+
+One write, two stored shapes, depending on the backend.
+
+## What changes
+
+- A string is judged by `parseNumericString` from `@objectstack/spec/data`, the
+  one numeric grammar the filter door also reads: the whole string is a JSON
+  number literal naming a finite double. Its case table,
+  `NUMERIC_STRING_GRAMMAR_CASES`, decides every form. No second grammar lives in
+  the engine.
+- An admitted string is stored as the number it denotes, on every backend:
+  `'12'` is written as `12`, `'1e3'` as `1000`. The rewrite runs at the write
+  door, before the middleware, the hooks, the `readonlyWhen` locks and
+  validation read the payload, so a `before*` hook now sees the number. The
+  caller's own object is not mutated.
+- `min`, `max`, `scale` and `precision` read that number, exactly as they read
+  a number: `'12.50'` passes `scale: 1` (it is `12.5`), and `'150'` over
+  `max: 100` is `max_value`.
+- This holds on every engine, REST, batch and updateMany door, and in
+  `validate` (the dry run). The server `/import` route is unchanged: its own
+  cell reader turns a numeric cell into a number before the write, so the
+  grammar never sees a string from it.
+- A blank is still `null` before the check (#20308). `summary` is still not
+  judged. A number, and an array, boolean or object, are answered as before.
+
+## Who sends numeric strings
+
+objectui's CSV import wizard, on its legacy per-row fallback (`legacyImport`,
+used only when the connected client cannot reach the server `/import` route),
+posts each raw cell to `create` after a client check of
+`!isNaN(Number(value))`. Its parser trims cells, so of the refused forms it can
+send the radix literals and the non-JSON spellings. Those rows now fail with
+`invalid_number` instead of storing a string. The fix there is the wizard's
+default path: import through the server `/import` route, whose cell reader
+converts the number before the write. Every interactive form widget sends a JS
+number or `null`, and is unaffected.
+
+## Rows already stored
+
+This judges new writes only; a stored value is never re-read by the check. On
+memory, an accepted string stayed a string until the record is next written.
+On SQLite, the earlier #20309 note's query finds a numeric column holding TEXT
+(such as `'0x10'`):
+
+```sql
+SELECT id, "FIELD" FROM "OBJECT" WHERE typeof("FIELD") = 'text';
+```
+
+OBJECT is the object name and FIELD is the field name. Nothing here rewrites
+such a cell; decide its number by hand.
+
+<!-- adr-0087: not-required (no-migration-prescription) Nothing authored moves: `packages/spec` is untouched and no metadata key is added, removed or reshaped, so `objectstack migrate meta` has nothing to rewrite and the ledger has no row to gain. What narrows is the set of caller-written string VALUES a number-typed field accepts at the write door, judged by the spec's existing numeric grammar; stored rows are never re-read, and a caller that sends a number, a blank or a grammar-admitted string is unaffected. The other categories are closed on facts: the package publishes (not `unpublished`); no ADR-0087 id covers a write-door value check (not `registered` / `already-registered`); and the change is runtime behaviour, not a TypeScript declaration (not `runtime-interface-only` / `type-surface-only`). -->

.changeset/20351-number-comparand-door.md

@@ -0,0 +1,35 @@
+---
+"@objectstack/objectql": minor
+---
+
+fix(objectql)!: a string compared against a number field must be a number: a non-numeric one is refused with `INVALID_FILTER` / 400 on `where`, a per-aggregation `filter` and `having`, and a numeric one is narrowed to its number (#20351)
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (no-migration-prescription) a refusal of a filter COMPARAND at the engine's query door: no authorable key, spelling or stored shape moves, `packages/spec` is untouched (the grammar, the verdict and the words shipped with the spec contract), and no stored row is read or rewritten. What is refused is a string that names no number, compared against a declared numeric field or a numeric aggregated column; which number the caller meant is not something a ledger entry can decide. The other categories are closed on facts: the package publishes (not `unpublished`); no ADR-0087 id covers a filter comparand (not `registered` / `already-registered`); and the change is runtime behaviour, not a declaration (not `runtime-interface-only` / `type-surface-only`). -->
+
+**BREAKING**: this narrows what a filter may compare a number field with. A string the platform's numeric grammar does not read as a number used to answer 200 with no rows (every row under `$ne`) on memory and SQLite and a 500 on PostgreSQL; it now answers 400, before any read, on every driver. It ships as `minor` under the launch-window convention for accept-set narrowings. `@objectstack/objectql`'s root exports are unchanged.
+
+FROM a string that is not a JSON number spelling of a finite number (`"abc"`, `""`, `" 12 "`, `"0x10"`, `"1,000"`, `"+5"`, `"007"`, `"Infinity"`, a `{placeholder}`), compared against a `number`, `currency`, `percent`, `rating`, `slider`, `progress` or `summary` field (or a `count` / `sum` / `avg`, or a numeric `min` / `max` / groupBy column in `having`) at the implicit comparand, `$eq` / `$ne` / `$gt` / `$gte` / `$lt` / `$lte`, or a member of `$in` / `$nin` / `$between` → TO `INVALID_FILTER` / 400, naming the field, its declared type, the comparand, its position and what is wrong with it. The fix is one line: send the number (`12`, `-3.5`, `1e3`) or a string of exactly that spelling (`"12"`).
+
+Measured through `engine.find` / `engine.aggregate` and `POST /data/:object/query` (the two doors agree), three rows (5, 12, 30):
+
+| position | comparand on a `number` field | before: memory · SQLite · PostgreSQL 16 | now, on all three |
+|:--|:--|:--|:--|
+| `where` | `$gt` / `$eq` / implicit / a `$in` member `"abc"`; `$eq ""` | no rows · no rows · `DATABASE_ERROR` (500) | `INVALID_FILTER` / 400 |
+| `where` | `$ne "abc"` | every row · every row · 500 | `INVALID_FILTER` / 400 |
+| `where`, over REST | `$gt "{current_user_id}"` (resolved to the user's id) | no rows · no rows · 500 | `INVALID_FILTER` / 400 |
+| per-aggregation `filter` | `$gt "abc"` (`$ne "abc"`) | count 0 (3), on all three | `INVALID_FILTER` / 400 |
+| `having` on `sum(amount)` | `$gt "abc"` (`$ne "abc"`) | no group (every group), on all three | `INVALID_FILTER` / 400 |
+| `where` | `$gt "12"` / `$eq "12"` | **no rows** · 1 row · 1 row | 1 row, the number's answer |
+
+What changes:
+
+- A new door at the engine's single filter collection point, after the temporal-comparand door. It reads `@objectstack/spec/data`'s published contract (`numberComparandDoorVerdict` over `NUMERIC_VALUE_TYPES`, the numeric grammar, and `numberComparandRefusalMessage` for the words); the engine carries no numeric grammar of its own.
+- It runs on `where` in both spellings (the filter object and the `FilterArray` sugar) on `find`, `findOne`, `count`, `aggregate`, `update` and `delete`, and on `IObjectQLEngine.judgeFilter`; on each per-aggregation `filter`, against the object's declared fields; and on `having`, over the columns the engine classes numeric.
+- A numeric string is rewritten to its number, copy-on-write, before any driver or in-memory evaluator reads it. InMemoryDriver used to compare `"12"` as a string and match nothing; it now matches what `12` matches, as SQLite and PostgreSQL already did.
+- A `{placeholder}` compared against a number field is refused unresolved: every filter token resolves to an id or a date, never a number.
+
+**Who is affected.** A caller that compares a number field with a string that is not a plain number, through any door that reaches the engine: a REST query parameter (`?amount=abc`), a `where` / `$filter` / `filter` body of `POST /data/:object/query`, or an in-process engine call. A caller that sends a number, or a numeric string such as `"12"` or `"1e3"`, is unaffected, except that memory now answers it as the other drivers do.
+
+**Unchanged.** A numeric comparand: the `$gt 10` controls at `where`, the per-aggregation `filter` and `having` answered identically before and after on memory, SQLite and PostgreSQL, through the engine and REST. Not judged by this door, so the filter reaches the driver as written (pinned per case in the engine suite): a string compared against a non-numeric field; `$null` / `$exists` / `$empty`; the text operators (a text operator over a number field keeps its own refusal); a `{ $field }` reference; a dotted key; a key that names no declared field. A `formula` field is still refused one door earlier with `INVALID_FIELD`. RLS, sharing and tenant predicates the security layer composes onto a query are not judged by this door; a policy predicate is judged at authoring where the host hands the rule the engine's `judgeFilter`. A boolean or a `Date` compared against a number field is outside this contract (it judges strings) and keeps its old answer, measured: `$gt true` no rows on memory, every row on SQLite and a 500 on PostgreSQL; a `Date` no rows on memory and SQLite and a 500 on PostgreSQL.

.changeset/20358-aggregate-honours-search.md

@@ -0,0 +1,17 @@
+---
+"@objectstack/spec": minor
+"@objectstack/objectql": minor
+"@objectstack/metadata-protocol": patch
+---
+
+A grouped or aggregated query now honours `search`: the groups and every aggregated number are computed over the searched rows, exactly the rows the same query without `groupBy` / `aggregations` returns.
+
+Clause-②: yes (widening) — `EngineAggregateOptionsSchema` gains two OPTIONAL keys, `search` and `searchFields`, so the accept set of the aggregate options grows. Nothing previously admitted is refused, no key is renamed or retired, and no producer is required to write them.
+
+`QuerySchema.search` (ADR-0061) is declared on the query beside `groupBy` and `aggregations`, with no carve-out. Until now, `POST /data/:object/query` accepted a body such as `{ groupBy: ["business_unit"], aggregations: [{ function: "count", alias: "count" }], search: "harbour" }` and answered it with the UNSEARCHED groups — no error and no warning — while the same body without `groupBy` / `aggregations` returned only the searched rows. A grouped list view under a toolbar search would therefore show group headers that ignore what the user typed.
+
+- **`@objectstack/spec`** — `EngineAggregateOptionsSchema` declares `search` (the bare string, or the structured `FullTextSearchSchema` form) and `searchFields`, identically to `EngineQueryOptionsSchema`. A parse used to strip them.
+- **`@objectstack/objectql`** — `engine.aggregate()` (and `ctx.api.object(name).aggregate()`) accepts the two keys it used to refuse as unknown options, and expands them through the same ADR-0061 expansion `find()` uses: the same server-resolved searchable fields, the same `searchFields` narrowing, AND-ed with `where` before the security middlewares run. There is one expander, not two. It applies on both aggregate paths, native `driver.aggregate()` and the in-memory lowering. A key the verb still does not execute, such as `$search`, is refused as before.
+- **`@objectstack/metadata-protocol`** — `findData`'s grouped branch passes `search` / `searchFields` to `engine.aggregate()`. `searchFields` is validated on that branch exactly as on the flat one: a column search cannot scan is `400 INVALID_FIELD`.
+
+Nothing to migrate. A caller that worked around the gap, for example by grouping a page of searched rows on the client, can send the grouped query with its `search` instead.

.changeset/20439-hook-condition-expression-row.md

@@ -0,0 +1,11 @@
+---
+'@objectstack/spec': patch
+---
+
+`hook.form.ts`'s `condition` row now declares `language: 'expression'`, matching the CEL predicate `HookSchema.condition` actually is.
+
+Clause-②: no
+
+The row previously declared `language: 'javascript'` — the same declared language as a real script row (`body.source`) — over a field that is `EvaluatedExpressionInputSchema`, a CEL predicate. A consumer keyed on the row's declared language could not tell the predicate apart from a script. The `helpText` moves from "Optional formula — skip the hook when this evaluates to false" to "CEL predicate — the hook runs only when TRUE", matching the phrasing every sibling predicate row (`field.form.ts` / `object.form.ts`'s `visibleWhen` / `readonlyWhen` / `requiredWhen`, and the formula `expression` row) already uses.
+
+No key is added, removed, narrowed or widened, and no parse verdict changes — `type: 'code'` and `language` are already-declared form-DSL vocabulary. `metadata-form-declared-rows.pin.test.ts` pins the new value, with a control against a sibling predicate row.

.changeset/20445-analytics-empty-operator-arms.md

@@ -0,0 +1,20 @@
+---
+'@objectstack/service-analytics': minor
+---
+
+fix(service-analytics): both analytics filter faces answer `$empty` by the field's declared type (#20445)
+
+Clause-②: yes (widening)
+
+`$empty: true | false` is declared by `@objectstack/spec` (`FieldOperatorsSchema`) with a per-type meaning: a text-like field is empty when it is null or `''`, a multi-value field (multiselect, checkboxes, tags, or a select / radio / lookup / user / file / image with `multiple: true`) when it is null or `[]`, and every other type only when it is null. `$empty: false` is the exact complement. Both of this package's filter faces now answer it by that table, through the spec's one expansion (`expandEmptyOperator`), instead of refusing it:
+
+- **The analytics `where`** (`/analytics/query`, `/analytics/sql`, dataset filters): `NativeSQLStrategy` and the `ObjectQLStrategy` SQL echo compile the field's declared row; the ObjectQL execute path hands `{ $empty }` to the data engine, which answers it once the engine's own arm lands (until then the engine refuses it, `INVALID_FILTER` / 400, as it does today).
+- **Row-level read scopes** compiled to SQL (`compileScopedFilterToSql`): same rows, in the read-scope envelope.
+
+A multi-value field's empty list is tested with a JSON function per SQL dialect (`json_array_length` on SQLite, a `jsonb` comparison on Postgres, `JSON_LENGTH` on MySQL).
+
+**Refused, never guessed** — `INVALID_FILTER` / 400 on the `where` face, `READ_SCOPE_COMPILE_FAILED` / 500 on a read scope — when the host cannot name the field's declared type (no `sourceFieldMeta` wired, or no such field), when a multi-value field's datasource dialect is unknown, and when the flag is not a boolean (`$empty: 'true'` is refused like a non-boolean `$null`).
+
+Host API (two new optional members, hence `minor`): `AnalyticsServiceConfig.sourceFieldMeta` may now answer `multiple` beside `type`, and `AnalyticsServicePlugin` relays it from the field definition; `compileScopedFilterToSql` takes an optional `declaredValueShape` option. A host whose `sourceFieldMeta` answers `type` but not `multiple` has every multi-capable field it declared `multiple: true` (select / radio / lookup / user / file / image) read as single-valued, which is the null-only row. On such a field a read scope's `$empty: false` then admits rows holding `[]`, and `$empty: true` misses them. Relay the field's `multiple` from its definition to get the list row.
+
+`$empty` stays staged: it is not in `FILTER_OPERATORS`, and the view operators `is_empty` / `is_not_empty` still lower to `$null`.

.changeset/20466-gantt-timezone-describe.md

@@ -0,0 +1,20 @@
+---
+"@objectstack/spec": patch
+---
+
+fix(spec): correct `GanttConfig.timeZone`'s describe — persisted gantt drops on a `date` field are not "real instants" (#20466)
+
+Clause-②: no
+
+The `GanttConfig` `timeZone` member's `.describe()` said "persisted data stays real
+instants" for every field. That is false for a `Field.date` column: per the spec's own
+storage rule (`temporalStorageForm` / ADR-0053), a gantt drop on a `date` field writes the
+calendar day it landed on, as a timezone-naive `YYYY-MM-DD`, while a `datetime` field
+still writes the real instant. Only the false clause is replaced — "a datetime value is
+still written as the real instant, and a date value as the calendar day it was dropped on
+in this zone's calendar (`YYYY-MM-DD`)" — the rest of the describe, and every other
+member, is unchanged.
+
+No key moves and no verdict moves: this corrects a published describe's prose to match
+the contract it already had, it does not add, remove or re-scope anything authorable. The
+JSON Schema and reference docs regenerate from the corrected source.