|
| 1 | +--- |
| 2 | +'@objectstack/spec': minor |
| 3 | +--- |
| 4 | + |
| 5 | +feat(spec)!: an analytics dataset dimension's and measure's `field` is a column reference — a SQL expression there is refused at parse, as it already is on the cube members a dataset compiles to (#21220) |
| 6 | + |
| 7 | +Clause-②: yes (narrowing) |
| 8 | + |
| 9 | +**BREAKING** — shipped as `minor` under the launch-window convention |
| 10 | +(`check-changeset-no-major` refuses `major` until GA; breaking-ness is carried by |
| 11 | +this banner, the `(narrowing)` arm above and the ADR-0087 disposition below, |
| 12 | +never by the level). |
| 13 | + |
| 14 | +`DatasetDimensionSchema.field` and `DatasetMeasureSchema.field` — the `field` of |
| 15 | +every entry in an ADR-0021 dataset's `dimensions` and `measures` — admit a column |
| 16 | +reference only: a field of the dataset's object (`amount`), or a relationship path |
| 17 | +of bare identifiers ending in one (`account.amount`, `account.owner.region`); a |
| 18 | +measure also admits `'*'` for a count, and a count may still omit `field`. Any |
| 19 | +other value — an arithmetic, an aggregate, a `CASE`, a subquery, a function call, |
| 20 | +a quoted or `$`-prefixed spelling, a padded or empty string, a broken path — is |
| 21 | +refused at `dimensions.N.field` / `measures.N.field` with a prescription, and so |
| 22 | +is `'*'` on a dimension. |
| 23 | + |
| 24 | +Why: the dataset layer was declared to take no raw SQL (ADR-0021 "zero raw SQL / |
| 25 | +zero raw expressions") and `field` was documented as a field or a relationship |
| 26 | +path, but it was a bare string and parsed anything. The analytics dataset door |
| 27 | +already refused an expression `field` on every query (`PERMISSION_DENIED` / 403, |
| 28 | +inline or saved), so such a dataset could be saved and never answered — declared, |
| 29 | +never enforced (ADR-0049). That door never judged an empty `field`: it skips one, |
| 30 | +which is how a `count` measure with `field: ''` kept counting rows on SQLite's |
| 31 | +native-SQL path (the D2 repair below). The accept set is the one the cube members a dataset |
| 32 | +compiles to already hold: the dataset compiler copies `field` into the member's |
| 33 | +`sql` verbatim, and both now read one shared declaration. `'*'` is refused on a |
| 34 | +dimension because grouping by every column is no axis — both analytics strategies |
| 35 | +answered such a dimension `500`. The rule is a `pattern` in the published JSON |
| 36 | +Schema too, so a document validated against `json-schema/**` is judged as the |
| 37 | +parse judges it. |
| 38 | + |
| 39 | +## FROM → TO |
| 40 | + |
| 41 | +``` |
| 42 | +FROM defineDataset({ name: 'task_metrics', label: 'Task Metrics', object: 'task', |
| 43 | + dimensions: [{ name: 'priority', field: 'priority' }], |
| 44 | + measures: [ |
| 45 | + { name: 'task_count', aggregate: 'count', field: '' }, |
| 46 | + { name: 'done_points', aggregate: 'sum', |
| 47 | + field: "CASE WHEN status = 'done' THEN points ELSE 0 END" }, |
| 48 | + ] }) |
| 49 | + -> parsed; the dataset door refused the expression on every query |
| 50 | +TO -> ZodError at measures.0.field and measures.1.field (invalid_format): |
| 51 | + `measures[].field` is a column reference: a field of the dataset's object … |
| 52 | +
|
| 53 | + defineDataset({ name: 'task_metrics', label: 'Task Metrics', object: 'task', |
| 54 | + dimensions: [{ name: 'priority', field: 'priority' }], |
| 55 | + measures: [ |
| 56 | + { name: 'task_count', aggregate: 'count' }, |
| 57 | + { name: 'done_points', aggregate: 'sum', field: 'points', filter: { status: 'done' } }, |
| 58 | + ] }) |
| 59 | +``` |
| 60 | + |
| 61 | +A conditional count or sum is a measure with its own structured `filter`; a |
| 62 | +ratio, sum, difference or product of measures is `derived: { op, of: [...] }` |
| 63 | +over measures named in the same dataset. **Mind the scale:** a `derived` ratio is |
| 64 | +a 0–1 fraction, so an expression that multiplied by 100 returned percentage |
| 65 | +points — pair the ratio with a `%` numeral pattern. A dimension that bucketed a |
| 66 | +column with an expression has no expression form: group by the column itself, or |
| 67 | +keep the bucket as a field of the object and name that field. |
| 68 | + |
| 69 | +**The one-line fix:** parse each dataset; every refusal at `…field` is one member |
| 70 | +to change — name the column, omit `field` on a plain count (never `field: ''`), |
| 71 | +or move the computation to a measure `filter` or a `derived` measure. The one |
| 72 | +mechanical case is done for you: `os migrate meta --from 17` lists, and every |
| 73 | +stored-row rehydration replays, the D2 conversion |
| 74 | +`dataset-count-measure-empty-field-removed`, which drops a `count` measure's empty |
| 75 | +`field` (it still counts rows). Nothing else has a mechanical rewrite. |
| 76 | + |
| 77 | +**What an author who still writes it sees.** `DatasetSchema`, `defineStack({ |
| 78 | +datasets })` (`STACK_SCHEMA_INVALID` / 422), the `dataset` write door and |
| 79 | +`POST /api/v1/analytics/dataset/query` (which parses every dataset it is handed, |
| 80 | +inline or saved, and now answers `400 VALIDATION_FAILED` at the path where it |
| 81 | +answered `403 PERMISSION_DENIED` before) refuse the member at its `field` path |
| 82 | +with the prescription. `tsc` does not: the key's type is still `string`. |
| 83 | + |
| 84 | +## The retirement kit |
| 85 | + |
| 86 | +- **Schema.** `ui/dataset.zod.ts` holds both keys to the pattern; the pattern is |
| 87 | + declared once, in the non-public `data/analytics-column-reference.ts`, and the |
| 88 | + cube layer's `CUBE_MEMBER_SQL` is that same `RegExp`. A dimension's pattern is |
| 89 | + the same column path without the `'*'` arm. A column reference parses |
| 90 | + byte-identically to before. |
| 91 | +- **ADR-0087.** D2 carries the one lossless repair: the conversion |
| 92 | + `dataset-count-measure-empty-field-removed` (`retiredFromLoadPath`, so an author |
| 93 | + is refused at parse while stored rows and `os migrate meta` replay it) drops a |
| 94 | + `count` measure's `field: ''`, which compiles to `COUNT(*)` without it. The D3 |
| 95 | + entry `dataset-member-field-expression-refused`, linked to that conversion and |
| 96 | + with its step-18 rationale fragment, carries the rest — a non-count measure or a |
| 97 | + dimension with `''` and every expression have no mechanical rewrite into a |
| 98 | + column. No `RETIRED_KEYS_BY_MAJOR` row: no key left the shape, so the |
| 99 | + authorable-surface, api-surface and JSON-schema manifest ratchets are |
| 100 | + unchanged. |
| 101 | +- **Liveness.** The `dataset` ledger rows `dimensions.field` and |
| 102 | + `measures.field` stay `live`, re-verified, with the narrowing recorded. |
| 103 | +- **Docs.** The `ui/dataset` reference page is regenerated. |
| 104 | +- **Runtime.** Unchanged: the analytics dataset door's refusal stays as defence |
| 105 | + in depth for a dataset that reaches the service without meeting the parse — a |
| 106 | + row stored before this change, which the build probe hands over as read. |
| 107 | + |
| 108 | +## Reach, measured |
| 109 | + |
| 110 | +- This repository: no authored dataset carries a non-column `field` — the |
| 111 | + examples, `platform-objects`, the hand-written docs and the published skills |
| 112 | + were read. Two test fixtures that sent an expression `field` on purpose were |
| 113 | + re-pinned: the service door's test builds them unparsed, and the REST route's |
| 114 | + test now expects the route's `400`. |
| 115 | +- Studio's dataset inspector (objectui) seeds a new dimension or measure row with |
| 116 | + `field: ''`. A plain count saved that way parsed before; its query answered |
| 117 | + `500` on the ObjectQL path, while SQLite's native-SQL path accepted the |
| 118 | + `COUNT()` it compiled to. A row already stored that way is repaired on load by |
| 119 | + the D2 conversion above. A NEW save of that shape is refused at the save door |
| 120 | + with the prescription to omit the key, because the write path parses with the |
| 121 | + current schema and replays no conversion; the producer-side change is |
| 122 | + objectui's. |
| 123 | +- Out-of-repo authored datasets: NOT MEASURED. |
| 124 | + |
| 125 | +<!-- adr-0087: registered dataset-member-field-expression-refused, dataset-count-measure-empty-field-removed --> |
0 commit comments