fix(metadata-protocol): refuse filter and sort on the stored metadata body column, and reconcile the shared seam

Builds on the merged #21086 seam. Two parts:

- F3 (maintainer ruling A): the generic data door refuses a filter or sort on
  the stored body column of sys_metadata / sys_metadata_history, the sibling of
  its groupBy refusal — a predicate on the body evaluates it row by row (a
  withheld credential is otherwise recoverable by probing) and a sort orders by
  the same stored bytes, so neither is evaluated. Same family, shape and code
  (INVALID_FIELD / 400). Aggregation per-measure filters are covered too.

- Seam reconciliation: the family's object set, its predicate, the column names
  and the body redactor now have ONE definition in @objectstack/spec/kernel;
  metadata-protocol's data-door wrappers (projection, dropType, the grouping and
  filter/sort refusals) consume it instead of a private copy, so the audit,
  analytics and realtime exits cannot drift from the data door about what a
  credential is.

Adds the family enumeration pin and the per-surface tests (audit writer and
migration, analytics refusal, realtime event, data-door filter/sort).

Co-authored-by: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MRdbfpy4sQT8bUjmMhxsN7

Author: claude

.changeset/21120-stored-metadata-body-closeout.md

@@ -0,0 +1,23 @@
+---
+'@objectstack/metadata-protocol': minor
+'@objectstack/service-analytics': minor
+'@objectstack/plugin-audit': minor
+'@objectstack/objectql': minor
+---
+
+fix(security)!: stored metadata bodies are projected or refused at the audit, analytics, realtime and data-door filter/sort exits too
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (no-migration-prescription) further read/copy/evaluate exits for a stored metadata body (sys_metadata / sys_metadata_history), each routed through the one shared redactor or refused: the audit/activity write-time copy is projected, a data-door filter or sort on the body column is refused (the sibling of the already-registered-as-not-required groupBy refusal), an analytics query member on the body column is refused, and a data.record.* realtime event body is projected. No authorable key, spelling, export or stored shape moves, and no stored row is read differently by any metadata consumer; the published surfaces gain and lose nothing. The other categories are closed on facts: the packages publish (not `unpublished`); no ADR-0087 id covers a filter/sort target, an analytics member or an event body (not `registered` / `already-registered`); and the change is runtime behaviour, not a declaration (not `runtime-interface-only` / `type-surface-only`). -->
+
+**BREAKING**: this narrows what three doors accept or serve for the two stored-metadata tables — the generic data door refuses a filter or sort on the body column, the analytics door refuses it as a dimension / measure / filter / sort member, and the realtime event and the audit/activity copy now carry the body as its type's read projection instead of the stored bytes. It ships as `minor` under the launch-window convention for accept-set narrowings. No export or published type changes.
+
+**What changes.**
+
+- **Audit / activity copy (`@objectstack/plugin-audit`).** The audit writer copies a `sys_metadata` / `sys_metadata_history` row into `sys_audit_log.new_value` / `old_value` and `sys_activity.metadata`. That copy now projects the body through the shared redactor, so stored credential material is withheld from the second store too. A new `os migrate audit-metadata-bodies` command rewrites the copies already at rest (dry run by default, `--apply` to write, idempotent).
+- **Analytics (`@objectstack/service-analytics`).** A query naming the stored body column of these objects as a dimension, measure, filter or sort is refused with `400 INVALID_FIELD`, before any strategy runs — the posture analytics already takes for a member it will not evaluate.
+- **Realtime (`@objectstack/objectql`).** A `data.record.*` event projects its `after` / `changes` body through the same redactor, so a subscriber to these objects' events receives no stored credential.
+- **Data door filter / sort (`@objectstack/metadata-protocol`).** A filter or sort on the body column is refused with `400 INVALID_FIELD`, the same family and shape as the existing groupBy refusal.
+
+**What stays answerable.** Every scalar column of these objects — `type`, `name`, `scope`, `state`, timestamps — is still grouped, filtered, sorted, counted and served; only the body column is affected. Every other object is unchanged.

.changeset/21120-stored-metadata-body-seam.md

@@ -0,0 +1,21 @@
+---
+'@objectstack/spec': minor
+'@objectstack/cli': minor
+---
+
+feat(spec,cli): shared seam for projecting stored metadata bodies, and the audit rewrite command
+
+Clause-②: no
+
+`@objectstack/spec/kernel` gains the family-wide primitives for the
+stored-metadata-body security invariant, beside the per-type redactor registry
+they build on: `STORED_METADATA_BODY_OBJECTS` / `isStoredMetadataBodyObject`,
+the `STORED_METADATA_BODY_COLUMN` / `STORED_METADATA_TYPE_COLUMN` names, and
+`redactStoredMetadataBody` / `redactStoredMetadataRow` / `redactStoredMetadataRows`.
+These project a stored row's body through the one `getMetadataTypeRedactor`
+definition, so every surface that serves, copies or evaluates such a body shares
+one rule rather than a copy per package. Additive — no existing export changes.
+
+`@objectstack/cli` gains `os migrate audit-metadata-bodies`, the one-off rewrite
+of at-rest metadata-body copies in `sys_audit_log` / `sys_activity` (dry run by
+default, `--apply` to write, idempotent).

packages/metadata-protocol/src/metadata-redaction.ts

@@ -68,8 +68,23 @@
 
 import { getMetadataTypeRedactor } from '@objectstack/spec/kernel';
 import type { MetadataTypeRedactor } from '@objectstack/spec/kernel';
+// [#21120] The family-wide stored-metadata-body primitives — the object set,
+// the object predicate, the column names and the body redactor — live in
+// `@objectstack/spec/kernel`, reachable by every surface in the family
+// (service-analytics, plugin-audit, the objectql engine) that does not depend
+// on this package. The data-door wrappers below (`storedMetadataBodyProjection`,
+// `redactStoredMetadataRow`'s `dropType`, `storedMetadataBodyGroupingRefusal`)
+// are this package's own, built ON that one definition — never a second one.
+import {
+    isStoredMetadataBodyObject,
+    redactStoredMetadataBody,
+    STORED_METADATA_BODY_COLUMN,
+    STORED_METADATA_TYPE_COLUMN,
+} from '@objectstack/spec/kernel';
 import { PLURAL_TO_SINGULAR } from '@objectstack/spec/shared';
 
+export { isStoredMetadataBodyObject };
+
 /**
  * Resolve the redactor for a request-shaped type name.
  *
@@ -589,19 +604,14 @@ function planCarryForward<T>(type: string, incoming: T, stored: unknown): { out:
  * no write verb; `sys_metadata_history` is append-only), so a redacted body
  * read here can never be PUT back — no carry-forward inverse is owed, which is
  * what makes a pure read projection a complete answer on this door.
+ *
+ * [#21120] The object set, the `isStoredMetadataBodyObject` predicate and the
+ * two column names are the family-wide definition in `@objectstack/spec/kernel`,
+ * imported above and re-exported — one set, consumed by every surface, so the
+ * audit, analytics and realtime exits cannot drift from this one.
  */
-const STORED_METADATA_BODY_OBJECTS: ReadonlySet<string> = new Set(['sys_metadata', 'sys_metadata_history']);
-
-/** The column holding the serialized body, on every {@link STORED_METADATA_BODY_OBJECTS} member. */
-const STORED_BODY_COLUMN = 'metadata';
-
-/** The column naming the body's metadata type — what selects its redactor. */
-const STORED_TYPE_COLUMN = 'type';
-
-/** Whether `object`'s rows carry a stored metadata body the generic data door must project. */
-export function isStoredMetadataBodyObject(object: string): boolean {
-    return STORED_METADATA_BODY_OBJECTS.has(object);
-}
+const STORED_BODY_COLUMN = STORED_METADATA_BODY_COLUMN;
+const STORED_TYPE_COLUMN = STORED_METADATA_TYPE_COLUMN;
 
 /**
  * The projection to hand the engine for a read of `object`, given the caller's
@@ -655,31 +665,20 @@ export function redactStoredMetadataRow<T>(object: string, row: T, opts?: { drop
         const { [STORED_TYPE_COLUMN]: _type, ...rest } = record;
         return rest;
     };
-    const withheld = (): T => {
+
+    // [#21120] The body decision is the ONE shared primitive — same object set,
+    // same per-type redactor, same fail-closed rules — so this door cannot
+    // disagree with the audit / analytics / realtime exits about what a
+    // credential is. This function adds only the door-local wrinkles on top: the
+    // `dropType` strip of the type column `storedMetadataBodyProjection` asked
+    // for, and omitting the body on a fail-closed outcome.
+    const outcome = redactStoredMetadataBody(row[STORED_TYPE_COLUMN], row[STORED_BODY_COLUMN]);
+    if (!outcome.ok) {
         const { [STORED_BODY_COLUMN]: _body, ...rest } = row;
         return strip(rest) as T;
-    };
-
-    const body = row[STORED_BODY_COLUMN];
-    if (body === undefined || body === null) return (dropType ? strip(row) : row) as T;
-    const type = row[STORED_TYPE_COLUMN];
-    if (typeof type !== 'string' || type === '') return withheld();
-    if (!hasMetadataRedactor(type)) return (dropType ? strip(row) : row) as T;
-
-    let parsed: unknown = body;
-    if (typeof body === 'string') {
-        try {
-            parsed = JSON.parse(body);
-        } catch {
-            return withheld();
-        }
     }
-    const served = redactMetadataItem(type, parsed);
-    if (served === parsed) return (dropType ? strip(row) : row) as T;
-    return strip({
-        ...row,
-        [STORED_BODY_COLUMN]: typeof body === 'string' ? JSON.stringify(served) : served,
-    }) as T;
+    if (outcome.body === row[STORED_BODY_COLUMN]) return (dropType ? strip(row) : row) as T;
+    return strip({ ...row, [STORED_BODY_COLUMN]: outcome.body }) as T;
 }
 
 /** {@link redactStoredMetadataRow} over the rows of one read. Non-array input passes through. */
@@ -727,3 +726,51 @@ export function storedMetadataBodyGroupingRefusal(object: string, groupBy: unkno
     }
     return undefined;
 }
+
+/**
+ * [#21120] The data door's FILTER / SORT refusal on the stored body column —
+ * maintainer ruling A, the further accept-set narrowing the grouping refusal
+ * (#21086) began.
+ *
+ * A filter on the body column EVALUATES the stored body row by row: a credential
+ * withheld from every served answer is still recoverable by prefix probing
+ * (`?filter={"metadata":{"$contains":"<guess>"}}` returns the row only when the
+ * guess is a prefix — a predicate oracle). A sort on it orders by the same
+ * stored bytes. Neither serves the body, so projecting it is no answer; the only
+ * answer is to refuse, the same posture the engine's own masked-field guard and
+ * the grouping refusal above take. Same family, shape and code: `INVALID_FIELD`
+ * / 400, naming the field, the object and the offending `param`.
+ *
+ * `filterFields` is the set of head field names the caller's `where` names
+ * (`collectFilterFieldKeys`), and `sortFields` the fields its `orderBy` names.
+ * Filter is judged before sort — a query that does both reads "the filter was
+ * not run" first. `undefined` when neither names the body column.
+ */
+export function storedMetadataBodyPredicateRefusal(
+    object: string,
+    opts: { filterFields?: readonly unknown[]; sortFields?: readonly unknown[] },
+): Error | undefined {
+    if (!isStoredMetadataBodyObject(object)) return undefined;
+    const namesBody = (fields: readonly unknown[] | undefined): boolean =>
+        Array.isArray(fields) && fields.some((f) => f === STORED_BODY_COLUMN);
+    const make = (param: 'filter' | 'sort', verb: string): Error => {
+        const err: any = new Error(
+            `Cannot ${verb} '${object}' by '${STORED_BODY_COLUMN}' (${param}): the query was not run. The `
+            + `${STORED_BODY_COLUMN} column holds a stored metadata body, served only as its type's read `
+            + `projection with stored credential material withheld. ${param === 'filter'
+                ? 'A filter on it evaluates the stored body row by row, which rebuilds a withheld credential by probing'
+                : 'A sort on it orders by the same stored bytes'}, so it is refused rather than evaluated. `
+            + `Filter or sort by '${STORED_TYPE_COLUMN}', 'name' or another scalar column instead.`,
+        );
+        err.code = 'INVALID_FIELD';
+        err.status = 400;
+        err.field = STORED_BODY_COLUMN;
+        err.fields = [STORED_BODY_COLUMN];
+        err.object = object;
+        err.param = param;
+        return err;
+    };
+    if (namesBody(opts.filterFields)) return make('filter', 'filter');
+    if (namesBody(opts.sortFields)) return make('sort', 'sort');
+    return undefined;
+}

packages/metadata-protocol/src/protocol.ts

@@ -184,6 +184,7 @@ import {
     redactStoredMetadataRow,
     redactStoredMetadataRows,
     storedMetadataBodyGroupingRefusal,
+    storedMetadataBodyPredicateRefusal,
     storedMetadataBodyProjection,
 } from './metadata-redaction.js';
 import type {
@@ -11646,6 +11647,25 @@ export class ObjectStackProtocolImplementation implements
         // unknown name keeps its own answer.
         const bodyGroupingRefusal = storedMetadataBodyGroupingRefusal(request.object, options.groupBy);
         if (bodyGroupingRefusal) throw bodyGroupingRefusal;
+        // [#21120] …and the FILTER / SORT half of the same family (maintainer
+        // ruling A): a predicate or an order key on the stored body column
+        // evaluates the body — a filter oracle that rebuilds a withheld
+        // credential by probing, or an order over the same bytes — so it is
+        // refused here, in the same shape as the grouping refusal, before the
+        // engine is asked. Field keys are collected the same way
+        // `assertFilterFieldsExist` reads them, so a nested-relation filter whose
+        // HEAD segment is the body column is caught too.
+        const aggregationFilterFields = Array.isArray(options.aggregations)
+            ? (options.aggregations as ReadonlyArray<{ filter?: unknown }>).flatMap((a) =>
+                  collectFilterFieldKeys(a?.filter))
+            : [];
+        const bodyPredicateRefusal = storedMetadataBodyPredicateRefusal(request.object, {
+            filterFields: [...collectFilterFieldKeys(options.where), ...aggregationFilterFields],
+            sortFields: Array.isArray(options.orderBy)
+                ? (options.orderBy as ReadonlyArray<{ field?: unknown }>).map((e) => e?.field)
+                : [],
+        });
+        if (bodyPredicateRefusal) throw bodyPredicateRefusal;
 
         // Route to engine.aggregate() when the query has GROUP BY / aggregations.
         // engine.find() does not do in-memory aggregation fallback, so without