Add field-reference-to-alias conversion: stored reference_to serves as reference

Protocol-18 ADR-0087 D2 entry canonicalizing the legacy objectql field-key
dialect reference_to -> reference on object and object-extension fields,
retiredFromLoadPath from day one (FieldSchema keeps its named rejection).
Registered in the step-18 migration chain so os migrate meta (and --stored)
rewrites old sources; serve-face end-to-end pins in metadata-protocol prove
a stored row seeded around the Zod gate is served with only the canonical
spelling across getMetaItems / getMetaItem / loadMetaFromDb.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PBjwYLS6BciTQW3c9xQiD2

Author: claude

.changeset/fluffy-clocks-reference.md

@@ -0,0 +1,15 @@
+---
+'@objectstack/spec': minor
+---
+
+Add the ADR-0087 D2 conversion `field-reference-to-alias` (#13700, ui#6837 half 1):
+the legacy objectql field-key dialect `reference_to` canonicalizes to `reference` on
+object and object-extension fields. The entry is `retiredFromLoadPath` from day one —
+`FieldSchema` has always refused the key by name, and that authoring-surface rejection
+is unchanged — so the widened accept surface is exactly the two paths that meet data
+written around the Zod gate: stored `sys_metadata` rehydration (every serve seam now
+emits only the canonical spelling, the guarantee objectui needs before deleting its
+`reference ?? reference_to` fallback arms) and `os migrate meta` (including
+`--stored`), which now rewrites the key mechanically. House #4923 precedence: an
+already-canonical `reference` wins — a redundant twin is dropped, a disagreeing pair
+is kept for the author to reconcile.

packages/metadata-protocol/src/protocol.stored-conversions.test.ts

@@ -147,6 +147,60 @@ describe('getMetaItems — stored rows are served canonical (#3903)', () => {
     });
 });
 
+// [#13700 — ui#6837 half 1] A stored object row carrying the legacy objectql
+// `reference_to` dialect. `FieldSchema` has always REFUSED the key (so this
+// row could only have been written by a seam that bypasses the parse — which
+// is exactly what the stub seeding reproduces), and objectui's
+// `reference ?? reference_to` fallback arms are being deleted on the strength
+// of this suite: the serve face must only ever emit `reference`.
+const legacyReferenceToRow = {
+    type: 'object',
+    name: 'crm_contact',
+    metadata: {
+        name: 'crm_contact',
+        label: 'Contact',
+        fields: {
+            company_id: { type: 'lookup', label: 'Company', reference_to: 'crm_company' },
+            title: { type: 'text', label: 'Title' },
+        },
+    },
+};
+
+describe('getMetaItems — stored reference_to serves as reference (#13700, ui#6837 half 1)', () => {
+    it('serves the lookup target under the canonical key ONLY', async () => {
+        const { engine } = makeStubEngine([legacyReferenceToRow]);
+        const protocol = new ObjectStackProtocolImplementation(engine);
+        const res = await protocol.getMetaItems({ type: 'object' });
+        const obj = (res.items as any[]).find((i) => i.name === 'crm_contact');
+        expect(obj.fields.company_id.reference).toBe('crm_company');
+        expect('reference_to' in obj.fields.company_id).toBe(false);
+        // The non-relationship sibling rides through untouched.
+        expect(obj.fields.title).toEqual({ type: 'text', label: 'Title' });
+    });
+
+    it('single read is canonical too, with clean _diagnostics (chain-owned history is not "invalid")', async () => {
+        const { engine } = makeStubEngine([legacyReferenceToRow]);
+        const protocol = new ObjectStackProtocolImplementation(engine);
+        const res: any = await protocol.getMetaItem({ type: 'object', name: 'crm_contact' });
+        expect(res.item.fields.company_id.reference).toBe('crm_company');
+        expect('reference_to' in res.item.fields.company_id).toBe(false);
+        // Unconverted, this row reads as invalid metadata (the dialect key is
+        // an `unrecognized_keys` rejection); converted first, it is valid —
+        // the serve face OWNS this history rather than reporting it broken.
+        expect(res.item._diagnostics?.valid).toBe(true);
+    });
+
+    it('boot hydration registers the CONVERTED body', async () => {
+        const { engine, registered } = makeStubEngine([legacyReferenceToRow]);
+        const protocol = new ObjectStackProtocolImplementation(engine);
+        const res = await protocol.loadMetaFromDb();
+        expect(res).toEqual({ loaded: 1, errors: 0, invalid: 0, storeUnavailable: false });
+        const obj = registered.find((r) => r.kind === 'object')!;
+        expect(obj.body.fields.company_id.reference).toBe('crm_company');
+        expect('reference_to' in obj.body.fields.company_id).toBe(false);
+    });
+});
+
 describe('getMetaItem — single stored read is canonical (#3903)', () => {
     it('returns the converted body with clean _diagnostics (chain-owned history is not "invalid")', async () => {
         const { engine } = makeStubEngine([legacyObjectRow]);

packages/spec/src/conversions/conversions.test.ts

@@ -1379,4 +1379,120 @@ describe('conversion layer (ADR-0087 D2)', () => {
       expect(entry!.toMajor).toBe(17);
     });
   });
+
+  /**
+   * `field-reference-to-alias` (#13700 — ui#6837 half 1).
+   *
+   * The fixture pair pins before → after; what needs its own cover is the
+   * SPLIT this entry lives on: the authoring surface keeps REJECTING
+   * `reference_to` by name (the schema pin below), while the stored/migrate
+   * paths — the only paths that meet data written around the Zod gate —
+   * canonicalize it. Every conversion case here therefore runs
+   * `includeRetired` (the stored posture); the one case without it pins that
+   * the load seam does NOT quietly accept the dialect.
+   */
+  describe('field-reference-to-alias (#13700 — the stored `reference_to` dialect canonicalizes)', () => {
+    const stackWith = (fields: Record<string, unknown>) => ({
+      objects: [{ name: 'crm_contact', label: 'Contact', fields }],
+    });
+    const fieldsOf = (stack: Record<string, unknown>) =>
+      (stack.objects as Array<{ fields: Record<string, Record<string, unknown>> }>)[0]!.fields;
+    const convert = (stack: Record<string, unknown>, notices?: ConversionNotice[]) =>
+      applyConversions(stack, { includeRetired: true, onNotice: notices ? (n) => notices.push(n) : undefined });
+
+    it('renames the dialect key on a lookup field (stored posture)', () => {
+      const notices: ConversionNotice[] = [];
+      const out = convert(
+        stackWith({ company_id: { type: 'lookup', label: 'Company', reference_to: 'crm_company' } }),
+        notices,
+      );
+      expect(fieldsOf(out).company_id).toEqual({ type: 'lookup', label: 'Company', reference: 'crm_company' });
+      expect(fieldsOf(out).company_id).not.toHaveProperty('reference_to');
+      expect(notices.filter((n) => n.conversionId === 'field-reference-to-alias')).toHaveLength(1);
+    });
+
+    it('emits a notice that names the surface, the site and the graduation', () => {
+      const notices: ConversionNotice[] = [];
+      convert(stackWith({ company_id: { type: 'lookup', reference_to: 'crm_company' } }), notices);
+      expect(notices).toHaveLength(1);
+      expect(notices[0]).toMatchObject({
+        conversionId: 'field-reference-to-alias',
+        surface: 'field.reference_to',
+        from: 'reference_to',
+        to: 'reference',
+        path: 'objects[0].fields.company_id.reference',
+        toMajor: 18,
+        retiresIn: 19,
+      });
+    });
+
+    it('keeps a `reference_to` that DISAGREES with an existing `reference` (both survive)', () => {
+      // The house precedence `renameKey` encodes since #4923: two different
+      // targets are the author's to reconcile — no rewrite, no notice.
+      const before = stackWith({
+        parent_id: { type: 'lookup', reference: 'crm_account', reference_to: 'crm_branch' },
+      });
+      const notices: ConversionNotice[] = [];
+      const out = convert(structuredClone(before), notices);
+      expect(out).toEqual(before);
+      expect(notices).toHaveLength(0);
+    });
+
+    it('is idempotent — the canonical shape is not a match', () => {
+      const before = stackWith({ company_id: { type: 'lookup', reference: 'crm_company' } });
+      const out = applyConversions(before, { includeRetired: true });
+      expect(out).toBe(before);
+    });
+
+    it('does NOT run on the plain load posture — the authoring surface keeps its rejection', () => {
+      // `retiredFromLoadPath`: without `includeRetired` (the
+      // `normalizeStackInput` posture) the dialect is untouched here, so the
+      // schema's named rejection stays the ONLY authoring-surface answer —
+      // accepting it quietly at load would widen the authoring surface, which
+      // is exactly what #11567/#13222 spent their refusal doors closing.
+      const before = stackWith({ company_id: { type: 'lookup', reference_to: 'crm_company' } });
+      const notices: ConversionNotice[] = [];
+      const out = applyConversions(structuredClone(before), { onNotice: (n) => notices.push(n) });
+      expect(out).toEqual(before);
+      expect(notices).toHaveLength(0);
+    });
+
+    it('reaches a STORED object row, so data at rest canonicalizes on rehydration', () => {
+      // `applyConversionsToStoredItem` wraps the row as `{ objects: [row] }`
+      // (#3903) and pins `includeRetired` — the serve-face guarantee ui#6837
+      // half 2 waits on lives on this seam.
+      const notices: ConversionNotice[] = [];
+      const row = {
+        name: 'crm_contact',
+        label: 'Contact',
+        fields: { company_id: { type: 'lookup', label: 'Company', reference_to: 'crm_company' } },
+      };
+      const out = applyConversionsToStoredItem('object', row, { onNotice: (n) => notices.push(n) });
+      expect(out.fields.company_id).toEqual({ type: 'lookup', label: 'Company', reference: 'crm_company' });
+      expect(notices.map((n) => n.conversionId)).toEqual(['field-reference-to-alias']);
+    });
+
+    /**
+     * The premise pin: this conversion is only correct while `FieldSchema`
+     * declares exactly one relationship spelling and refuses the dialect BY
+     * NAME. If the alias ever becomes a declared key, the entry above turns
+     * from history-replay into a silent overwrite of live data — this is the
+     * test that fails first.
+     */
+    it('the canonical field schema declares `reference` and rejects `reference_to` with the rename', async () => {
+      const { FieldSchema } = await import('../data/field.zod.js');
+      expect(
+        FieldSchema.safeParse({ name: 'company_id', type: 'lookup', reference: 'crm_company' }).success,
+      ).toBe(true);
+
+      const rejected = FieldSchema.safeParse({ name: 'company_id', type: 'lookup', reference_to: 'crm_company' });
+      expect(rejected.success).toBe(false);
+      const issue = rejected.error!.issues.find((i) => i.code === 'unrecognized_keys');
+      expect(issue).toBeDefined();
+      expect(issue!.message).toContain('`reference_to`');
+      // The rename the conversion performs, said out loud to the author who is
+      // typing the key right now.
+      expect(issue!.message).toContain('`reference`');
+    });
+  });
 });

packages/spec/src/conversions/registry.ts

@@ -8320,6 +8320,118 @@ const formViewOptionDefaultRemoved: MetadataConversion = {
   },
 };
 
+/**
+ * Field `reference_to` → `reference` (protocol 18, #13700 — ui#6837 half 1).
+ *
+ * One concept — the target object of a `lookup`/`master_detail` field — with
+ * two spellings, one per layer. `FieldSchema` declares `reference` and has
+ * always REFUSED `reference_to` by name (`unrecognized_keys`, carrying the
+ * rename: "Did you mean `reference_to` → `reference`?"); `reference_to` is the
+ * legacy objectql runtime dialect, which is exactly the spelling a row written
+ * by a seam that goes around Zod carries (`registerObject` deliberately skips
+ * the parse, #3896). The consumer-side reads of the dialect have been retired
+ * one by one — the SQL DDL door (#11567), the Mongo schema-sync door (#13222),
+ * the verify deriver (#13250) — each replaced by a loud refusal. What none of
+ * those doors reach is metadata AT REST.
+ *
+ * **Why a conversion when the schema already rejects** — the load-bearing
+ * half, and the reason this card blocks ui#6837. A stored `sys_metadata` row
+ * is runtime JSON: it is served without ever meeting the Zod gate, and the
+ * rejection surface does not reach back in time to rows already written. Today
+ * objectui still carries `reference ?? reference_to` fallback arms, so such a
+ * row still renders; ui#6837 half 2 deletes those arms ("the backend's
+ * metadata must be right; the frontend executes the protocol" — maintainer
+ * ruling on that card). Without this entry the deletion would silently degrade
+ * every at-rest `reference_to` lookup to a targetless picker — a raw-UUID
+ * text box, the #3405 failure shape, with no diagnostic anywhere. This entry
+ * makes the serve face own that history: `applyConversionsToStoredItem`
+ * replays the FULL chain (retired entries included) on every stored-row
+ * rehydration, so the wire only ever carries `reference`.
+ *
+ * **`retiredFromLoadPath` from day one** — the pre-launch one-step-rename
+ * shape ({@link MetadataConversion.retiredFromLoadPath}): the key never had a
+ * live authoring window to expire, because the schema never accepted it. The
+ * authoring surface keeps teaching with its named rejection; this entry covers
+ * the two paths that serve or rewrite EXISTING data — stored rehydration and
+ * `os migrate meta`. The DDL doors above keep guarding the third path
+ * (metadata handed straight to a driver, around both the gate and the stored
+ * pass); they are downstream of this entry, not replaced by it.
+ *
+ * Deliberately NOT converted here: `referenceTo` (camelCase). That spelling is
+ * objectui's *resolved action-param* dialect — a different surface with its
+ * own alias table (`ACTION_PARAM_KEY_ALIASES`, `ui/action.zod.ts`) — and it is
+ * not the spelling the objectql runtime wrote into stored object rows. Widening
+ * this entry to a spelling with no measured at-rest population would be scope
+ * invented at conversion time; the schema's rejection covers it either way.
+ *
+ * Precedence is the house rule {@link renameKey} encodes (#4923) and nothing
+ * new: an already-canonical `reference` WINS — a redundant twin is dropped, a
+ * DISAGREEING pair is left for the author to reconcile rather than the loader
+ * picking a target. Covers object fields and object-extension fields: the same
+ * `FieldSchema`, so the same dialect ({@link fieldConditionalRequiredToRequiredWhen}
+ * is the precedent, one protocol back).
+ */
+const fieldReferenceToAlias: MetadataConversion = {
+  id: 'field-reference-to-alias',
+  toMajor: 18,
+  retiredFromLoadPath: true,
+  surface: 'field.reference_to',
+  summary:
+    "field key 'reference_to' → 'reference' (the legacy objectql runtime dialect for a "
+    + "lookup/master_detail target; stored rows must serve the canonical spelling before "
+    + "objectui deletes its `reference ?? reference_to` fallback arms — ui#6837 half 1)",
+  apply(stack, emit) {
+    const withObjects = mapObjectFieldsKey(stack, 'objects', 'reference_to', 'reference', emit);
+    return mapObjectFieldsKey(withObjects, 'objectExtensions', 'reference_to', 'reference', emit);
+  },
+  fixture: {
+    before: {
+      objects: [{
+        name: 'crm_contact',
+        label: 'Contact',
+        fields: {
+          // The dialect spelling — the shape a legacy stored row carries.
+          company_id: { type: 'lookup', label: 'Company', reference_to: 'crm_company' },
+          // Canonical only: untouched.
+          owner_id: { type: 'lookup', label: 'Owner', reference: 'sys_user' },
+          // Both spellings, SAME target: the redundant twin goes (#4923).
+          account_id: { type: 'lookup', label: 'Account', reference: 'crm_account', reference_to: 'crm_account' },
+          // Both spellings, DIFFERENT targets: kept, so the author reconciles
+          // the two rather than the loader picking where the field points.
+          parent_id: { type: 'lookup', label: 'Parent', reference: 'crm_account', reference_to: 'crm_branch' },
+        },
+      }],
+      objectExtensions: [{
+        extend: 'crm_task',
+        fields: {
+          project_id: { type: 'master_detail', label: 'Project', reference_to: 'crm_project' },
+        },
+      }],
+    },
+    after: {
+      objects: [{
+        name: 'crm_contact',
+        label: 'Contact',
+        fields: {
+          company_id: { type: 'lookup', label: 'Company', reference: 'crm_company' },
+          owner_id: { type: 'lookup', label: 'Owner', reference: 'sys_user' },
+          account_id: { type: 'lookup', label: 'Account', reference: 'crm_account' },
+          parent_id: { type: 'lookup', label: 'Parent', reference: 'crm_account', reference_to: 'crm_branch' },
+        },
+      }],
+      objectExtensions: [{
+        extend: 'crm_task',
+        fields: {
+          project_id: { type: 'master_detail', label: 'Project', reference: 'crm_project' },
+        },
+      }],
+    },
+    // company_id (rename), account_id (redundant twin dropped), project_id
+    // (extension rename). owner_id and the disagreeing parent_id emit nothing.
+    expectedNotices: 3,
+  },
+};
+
 export const CONVERSIONS_BY_MAJOR: Readonly<Record<number, readonly MetadataConversion[]>> = {
   11: [flowNodeHttpRename, pageKindJsxToHtml, flowNodeFilterAlias, objectCompactLayoutRename],
   13: [stackRolesToPositions, owdLegacyReadAliases, sharingRecipientRoleToPosition],
@@ -8407,6 +8519,7 @@ export const CONVERSIONS_BY_MAJOR: Readonly<Record<number, readonly MetadataConv
     objectGridDefaultSortRemoved,
     permissionAllowRestorePurgeRemoved,
     formViewOptionDefaultRemoved,
+    fieldReferenceToAlias,
   ],
 };
 

packages/spec/src/migrations/registry.ts

@@ -5304,7 +5304,17 @@ const step18: MigrationStep = {
     'overlayWritable` are retiredKey tombstones (no D2 conversion — plugin/manager ' +
     'configs are not stack collection members, the additionalTypes reasoning). The ' +
     'customization that actually ships: ADR-0005\'s org overlay and ADR-0126\'s ' +
-    'packaged-metadata model.',
+    'packaged-metadata model. ' +
+    'Finally, it canonicalizes the legacy objectql field-key dialect `reference_to` → ' +
+    '`reference` on lookup/master_detail fields (#13700, ui#6837 half 1). `FieldSchema` ' +
+    'has always refused `reference_to` by name, but stored `sys_metadata` rows written by ' +
+    'seams that bypass the parse still carry it, held up today only by objectui\'s ' +
+    '`reference ?? reference_to` fallback arms — which ui#6837 half 2 deletes. The ' +
+    'mechanical conversion renames the key (house #4923 precedence: a canonical ' +
+    '`reference` wins, a disagreeing pair is kept for the author), replays on every ' +
+    'stored-row rehydration so the serve face only ever emits the canonical spelling, ' +
+    'and `os migrate meta` rewrites old sources; the authoring-surface rejection with ' +
+    'its rename prescription is unchanged.',
   conversionIds: [
     'field-malformed-scale-precision-removed',
     'record-chatter-position-vocabulary',
@@ -5320,6 +5330,7 @@ const step18: MigrationStep = {
     'object-grid-default-sort-removed',
     'permission-allow-restore-purge-removed',
     'form-view-option-default-removed',
+    'field-reference-to-alias',
   ],
   semantic: [
     // One file per entry under `entries/semantic/`, concatenated here sorted by