Skip to content

Commit f943f3b

Browse files
os-zhuangclaude
andauthored
test(dogfood): field-type capability-matrix round-trip (#2004 + AI-authoring) (#2022)
Extends the dogfood gate from "pin one example app" toward the platform's real risk: third parties have an AI author arbitrary metadata, so every authorable primitive needs a runtime proof — not a static shape check. `showcase_field_zoo` carries one field of (almost) every protocol FieldType but was only static-checked (the bundle registers it). This adds the first capability-matrix block: write one record covering many field types over the real REST data API, read it back, assert each round-trips with type fidelity. - Headliner coverage = the #2004 regressions: array-typed fields (multiselect/checkboxes/tags) and Field.time (time-of-day) — all proven to round-trip end-to-end on the in-memory (WASM) driver path `objectstack dev` uses for :memory:. - On first run it surfaced three real type-fidelity gaps — rating/slider/toggle persist but read back as strings ('4'/'25'/'1') while number/currency/percent/ boolean round-trip correctly. They are quarantined with `it.fails` (passes while broken, turns RED when fixed) rather than hidden by coercion, and filed as a separate task. Private package — no changeset. 21 passed + 3 expected-fail. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent ebc7ecb commit f943f3b

4 files changed

Lines changed: 164 additions & 1 deletion

File tree

‎packages/dogfood/README.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,4 +36,31 @@ sockets, CI-stable). Tests act as a browser client would: sign in, hit
3636
4. **Prove it catches the bug**: temporarily revert the relevant fix and confirm
3737
the test goes red. A green-on-the-bug test is not a gate.
3838

39+
## Capability matrix (the AI-authoring angle)
40+
41+
This is a **development platform**: third parties have an AI author arbitrary
42+
metadata. The risk is not "a platform change broke the CRM example" — it's "the
43+
AI used a valid primitive the examples don't exercise, and it silently breaks at
44+
runtime." So beyond pinning the example apps, the gate is growing into a
45+
**capability matrix**: every authorable primitive gets a runtime round-trip
46+
proof.
47+
48+
`test/field-zoo-roundtrip.dogfood.test.ts` is the first block — it writes one
49+
record covering many field types over the real REST API and asserts each reads
50+
back with type fidelity. On its first run it surfaced three real gaps
51+
(`rating`/`slider`/`toggle` read back as strings), which are **quarantined**, not
52+
hidden:
53+
54+
```ts
55+
// Known type-fidelity gap → it.fails passes while broken and turns RED the day
56+
// it's fixed, forcing the quarantine to be lifted instead of rotting.
57+
const runner = c.xfail ? it.fails : it;
58+
```
59+
60+
Roadmap (each its own slice, see the ADR): flow-node matrix, form-widget matrix,
61+
RLS-pattern matrix; then a generative pass that emits random valid metadata from
62+
the spec's zod surface; then run the AI **template corpus** through the harness.
63+
The binding policy — every authorable+live primitive must carry a runtime proof
64+
— is the subject of a dedicated ADR.
65+
3966
Runs in CI as the `Dogfood Regression Gate` job (and under `turbo run test`).

‎packages/dogfood/package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,8 @@
2020
"@objectstack/plugin-security": "workspace:*",
2121
"@objectstack/service-settings": "workspace:*",
2222
"@objectstack/service-analytics": "workspace:*",
23-
"@objectstack/example-crm": "workspace:*"
23+
"@objectstack/example-crm": "workspace:*",
24+
"@objectstack/example-showcase": "workspace:*"
2425
},
2526
"devDependencies": {
2627
"@types/node": "^25.9.3",
Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// CAPABILITY-MATRIX golden test — every authorable field type must survive a
4+
// real HTTP write → read round-trip.
5+
//
6+
// `showcase_field_zoo` carries one field of (almost) every protocol FieldType.
7+
// Until now it was only *static*-checked (the metadata bundle registers it);
8+
// nothing wrote a record and read it back. But the platform's value is that an
9+
// AI can author ANY field and it works at runtime — so each field type needs a
10+
// runtime proof, not a shape assertion. This is the first block of that matrix
11+
// and the direct guard for #2004 (array-typed fields silently failed to persist;
12+
// Field.time rejected time-of-day).
13+
14+
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
15+
import showcaseStack from '@objectstack/example-showcase';
16+
import { bootDogfoodStack, type DogfoodStack } from '../src/harness.js';
17+
18+
// A field-type coverage entry. `write` is the value POSTed; `expect` describes
19+
// how the value must come back. `equal` = exact (or set-equal for arrays);
20+
// `computed`/`present` cover server-owned fields you don't write.
21+
type Check =
22+
| { kind: 'equal'; write: unknown }
23+
| { kind: 'setEqual'; write: unknown[] }
24+
| { kind: 'present' } // server-assigned, just must be non-null
25+
| { kind: 'computed'; expected: unknown }; // derived, asserted not written
26+
27+
interface FieldCase {
28+
field: string;
29+
type: string;
30+
check: Check;
31+
// KNOWN GAP: the schema→SQL-column mapping / read coercion doesn't yet cover
32+
// this type, so it round-trips with the wrong JS type (e.g. '4' not 4). The
33+
// value persists (no data loss), but fidelity leaks. Quarantined via it.fails
34+
// so it stays visible AND auto-flags the day it's fixed. Tracked separately.
35+
xfail?: boolean;
36+
}
37+
38+
// The #2004 headliners are the three array types + f_time. The rest broaden the
39+
// matrix across scalars, temporals, structured JSON, and computed/system fields.
40+
const MATRIX: FieldCase[] = [
41+
// text-ish
42+
{ field: 'f_textarea', type: 'textarea', check: { kind: 'equal', write: 'line1\nline2' } },
43+
{ field: 'f_email', type: 'email', check: { kind: 'equal', write: 'zoo@example.com' } },
44+
{ field: 'f_url', type: 'url', check: { kind: 'equal', write: 'https://objectstack.ai' } },
45+
{ field: 'f_phone', type: 'phone', check: { kind: 'equal', write: '+14155550123' } },
46+
// numbers
47+
{ field: 'f_number', type: 'number', check: { kind: 'equal', write: 42 } },
48+
{ field: 'f_currency', type: 'currency', check: { kind: 'equal', write: 1234.56 } },
49+
{ field: 'f_percent', type: 'percent', check: { kind: 'equal', write: 75 } },
50+
{ field: 'f_rating', type: 'rating', check: { kind: 'equal', write: 4 }, xfail: true },
51+
{ field: 'f_slider', type: 'slider', check: { kind: 'equal', write: 25 }, xfail: true },
52+
// temporal — f_time is a #2004 fix (time-of-day)
53+
{ field: 'f_date', type: 'date', check: { kind: 'equal', write: '2024-03-15' } },
54+
{ field: 'f_time', type: 'time', check: { kind: 'equal', write: '14:30:00' } },
55+
// logic
56+
{ field: 'f_boolean', type: 'boolean', check: { kind: 'equal', write: true } },
57+
{ field: 'f_toggle', type: 'toggle', check: { kind: 'equal', write: true }, xfail: true },
58+
// scalar selection
59+
{ field: 'f_select', type: 'select', check: { kind: 'equal', write: 'high' } },
60+
{ field: 'f_radio', type: 'radio', check: { kind: 'equal', write: 'yes' } },
61+
// ── #2004 array headliners — these silently dropped before the fix ──
62+
{ field: 'f_multiselect', type: 'multiselect', check: { kind: 'setEqual', write: ['red', 'blue'] } },
63+
{ field: 'f_checkboxes', type: 'checkboxes', check: { kind: 'setEqual', write: ['email', 'push'] } },
64+
{ field: 'f_tags', type: 'tags', check: { kind: 'setEqual', write: ['alpha', 'beta', 'gamma'] } },
65+
// structured JSON
66+
{ field: 'f_json', type: 'json', check: { kind: 'equal', write: { a: 1, b: [2, 3] } } },
67+
{ field: 'f_color', type: 'color', check: { kind: 'equal', write: '#FF8800' } },
68+
// computed / system — not written, must materialize
69+
{ field: 'f_autonumber', type: 'autonumber', check: { kind: 'present' } },
70+
// f_number(42) * f_percent(75) / 100 = 31.5
71+
{ field: 'f_formula', type: 'formula', check: { kind: 'computed', expected: 31.5 } },
72+
];
73+
74+
describe('dogfood: field-type capability matrix round-trips over HTTP (#2004)', () => {
75+
let stack: DogfoodStack;
76+
let record: Record<string, unknown>;
77+
78+
beforeAll(async () => {
79+
stack = await bootDogfoodStack(showcaseStack);
80+
const token = await stack.signIn();
81+
82+
// Build the create body from every `equal`/`setEqual` entry (+ required name).
83+
const body: Record<string, unknown> = { name: 'zoo-roundtrip' };
84+
for (const c of MATRIX) {
85+
if (c.check.kind === 'equal') body[c.field] = c.check.write;
86+
else if (c.check.kind === 'setEqual') body[c.field] = c.check.write;
87+
}
88+
89+
const created = await stack.apiAs(token, 'POST', '/data/showcase_field_zoo', body);
90+
expect(created.status, `create failed: ${created.status} ${await created.clone().text()}`).toBeLessThan(300);
91+
const createdJson = (await created.json()) as { id?: string; record?: { id?: string } };
92+
const id = createdJson.id ?? createdJson.record?.id;
93+
expect(id, 'no id returned from create').toBeTruthy();
94+
95+
const got = await stack.apiAs(token, 'GET', `/data/showcase_field_zoo/${id}`);
96+
expect(got.status).toBe(200);
97+
const gotJson = (await got.json()) as { record?: Record<string, unknown> } & Record<string, unknown>;
98+
record = (gotJson.record ?? gotJson) as Record<string, unknown>;
99+
}, 60_000);
100+
101+
afterAll(async () => {
102+
await stack?.stop();
103+
});
104+
105+
for (const c of MATRIX) {
106+
// xfail entries are KNOWN type-fidelity gaps: `it.fails` passes while the
107+
// assertion throws and turns RED the moment the gap is fixed — forcing the
108+
// quarantine to be lifted rather than silently rotting.
109+
const runner = c.xfail ? it.fails : it;
110+
runner(`${c.type} (${c.field}) round-trips`, () => {
111+
const actual = record[c.field];
112+
switch (c.check.kind) {
113+
case 'equal':
114+
expect(actual).toEqual(c.check.write);
115+
break;
116+
case 'setEqual': {
117+
// Array-typed fields: persisted as a JSON array; order is not
118+
// guaranteed, so compare as sets (the #2004 break returned null/[]).
119+
expect(Array.isArray(actual), `${c.field} not an array: ${JSON.stringify(actual)}`).toBe(true);
120+
expect([...(actual as unknown[])].sort()).toEqual([...c.check.write].sort());
121+
break;
122+
}
123+
case 'present':
124+
expect(actual ?? null, `${c.field} should be server-assigned`).not.toBeNull();
125+
break;
126+
case 'computed':
127+
expect(Number(actual)).toBeCloseTo(Number(c.check.expected), 5);
128+
break;
129+
}
130+
});
131+
}
132+
});

‎pnpm-lock.yaml‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)