Skip to content

Commit 7cf283a

Browse files
xuyushun441-sysos-zhuangclaude
authored
feat(cli): make os validate the author-time gate; steer scaffolds/skills/examples/docs to it (#2201)
* feat(cli,create-objectstack): make `os validate` the author-time gate; steer scaffolds/skills/examples/docs to it ObjectStack metadata mistakes fail silently at runtime (bare-field predicates, dangling widget bindings). `os build` caught them; `os validate` did not — yet the scaffold's Next steps steered users to `validate`. Close the gap. - cli: `os validate` now runs the ADR-0032 CEL/predicate gate (CEL syntax + `record.<field>` existence) in addition to the protocol schema and widget bindings — a read-only superset of `os build`'s checks, no artifact emitted. Catches bare-field predicates (#2183/#2185). - create-objectstack: emit `AGENTS.md` + `.github/copilot-instructions.md` into every generated project telling coding agents to run `npm run validate` after editing metadata; align the blank template `dev`/`start` scripts with the example apps; sharpen the post-create "Next steps". Removes the previously dead copilot-instructions template (superseded by the emitted guide). - skills: add a "Verify your work" section to all 9 objectstack-* skills — authoritative in objectstack-platform, predicate/binding-focused in data/ui/automation/formula, tight pointers in api/query/i18n/ai. - examples: add a `validate` script to each app (folded into app-showcase's `verify`) + a single examples/AGENTS.md. - docs: new guide content/docs/guides/validating-metadata.mdx; correct the `os validate` reference in getting-started/cli.mdx. Verified: cli builds clean (tsc); `os validate` passes app-todo and fails a bare `visible: '!is_completed'` with a corrective message; scaffolder smoke test emits AGENTS.md with placeholders substituted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(create-objectstack): atomic exclusive-create for emitted guides (CodeQL js/file-system-race) CodeQL flagged the existsSync()+writeFileSync() check-then-write in writeAgentGuides as a TOCTOU file-system race (2 high alerts). Replace with an atomic `wx` (exclusive-create) write via a writeIfAbsent() helper that swallows EEXIST, and read the bundled template with try/catch ENOENT instead of a separate existence check. Same "don't clobber a template's own guide" semantics, no check-then-act gap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Jack Zhuang <277994282+os-zhuang@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 517dad9 commit 7cf283a

22 files changed

Lines changed: 499 additions & 13 deletions

File tree

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
"@objectstack/cli": minor
3+
"create-objectstack": minor
4+
---
5+
6+
Make `os validate` the author-time verification gate and steer scaffolds toward it.
7+
8+
- **`os validate`** now runs the same CEL/predicate gate as `os build`/`os compile`
9+
(ADR-0032): every `visible`/`disabled`/`requiredWhen`/validation/flow/sharing
10+
predicate is checked for CEL syntax and `record.<field>` existence on the target
11+
object. It already ran the protocol schema and widget-binding checks; the
12+
expression gate closes the gap so a bare field ref (`done` instead of
13+
`record.done`) — which silently hides an action on every record at runtime
14+
(#2183/#2185) — fails validation instead of shipping. `os validate` is now a
15+
read-only superset of the build's checks (no artifact emitted).
16+
- **`create-objectstack`** now emits an `AGENTS.md` (and `.github/copilot-instructions.md`)
17+
into every generated project instructing coding agents to run `npm run validate`
18+
after editing metadata, aligns the blank template's `dev`/`start` scripts with the
19+
example apps (`objectstack dev`/`objectstack start`), and sharpens the post-create
20+
"Next steps" output.

‎content/docs/getting-started/cli.mdx‎

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ Open [http://localhost:3000/_console/](http://localhost:3000/_console/) — you'
4747
### Validate & Build
4848

4949
```bash
50-
os validate # Check schema correctness
50+
os validate # Check schema + CEL predicates + widget bindings (no artifact)
5151
os compile # Build production artifact → dist/objectstack.json
5252
```
5353

@@ -303,7 +303,7 @@ shape they accept. See [Source vs Artifact](#source-vs-artifact) below.
303303
| Command | Description |
304304
|---------|-------------|
305305
| `os compile [config]` | Compile configuration to a JSON artifact (`dist/objectstack.json`) |
306-
| `os validate [config]` | Validate configuration against the ObjectStack Protocol schema |
306+
| `os validate [config]` | Validate schema, CEL predicates, and widget bindings — the same gates as `os compile`/`os build`, no artifact emitted |
307307
| `os info [config]` | Display metadata summary (objects, fields, apps, agents, etc.) |
308308

309309
#### `os compile`
@@ -346,7 +346,9 @@ over HTTP from another runtime. See [`os start`](#os-start) and
346346

347347
#### `os validate`
348348

349-
Standalone schema validation with rich error output. Use to check your configuration without compiling.
349+
The fast, artifact-free verification gate. It runs the **same structural and
350+
semantic checks as `os compile`/`os build`** but writes no `dist/`, so it is the
351+
command to run after every metadata edit. Use it before reporting a change done.
350352

351353
```bash
352354
os validate # Validate current directory
@@ -355,16 +357,37 @@ os validate --json # JSON output for CI
355357
os validate path/to/config # Validate specific file
356358
```
357359

360+
**Gates run (each exits non-zero with a located, corrective message):**
361+
1. **Protocol schema** — the stack conforms to `ObjectStackDefinitionSchema`
362+
(`@objectstack/spec`).
363+
2. **CEL / predicate validation (ADR-0032)** — every `visible` / `disabled` /
364+
`requiredWhen` / validation rule / flow condition / sharing rule is parsed
365+
for CEL syntax **and** checked that each `record.<field>` reference exists on
366+
the target object. This catches a **bare field ref** (`done` instead of
367+
`record.done`) that would otherwise evaluate to `null` and silently hide an
368+
action on every record (#2183/#2185).
369+
3. **Widget-binding integrity (ADR-0021)** — every dashboard widget's
370+
`dataset` / `dimensions` / `values` resolves to a declared dataset/field, so
371+
a dangling binding fails here instead of rendering an empty chart.
372+
358373
**Options:**
359374
- `--strict` — Treat warnings as errors (exit code 1)
360375
- `--json` — Output results as JSON
361376

362-
**Warnings checked:**
377+
**Warnings checked (advisory, non-blocking unless `--strict`):**
363378
- Missing `manifest.id` (required for deployment)
364379
- Missing `manifest.namespace` (required for multi-app hosting)
365380
- No objects defined
366381
- No apps or plugins defined
367382

383+
<Callout type="tip">
384+
`os validate` and `os build` share one validator, so a config that passes
385+
`os validate` will not fail the build on schema/predicate/binding grounds. In a
386+
scaffolded project these are wired as `npm run validate` and `npm run build`;
387+
your `AGENTS.md` tells coding agents to run `npm run validate` after editing
388+
metadata. See [Validating metadata](/docs/guides/validating-metadata).
389+
</Callout>
390+
368391
#### `os info`
369392

370393
Displays a summary of your metadata without compilation or validation:

‎content/docs/guides/meta.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
"seed-data",
1313
"common-patterns",
1414
"formula",
15+
"validating-metadata",
1516
"analytics-datasets",
1617
"airtable-dashboard-analysis",
1718
"---Building---",
Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
---
2+
title: Validating Metadata
3+
description: Why ObjectStack metadata mistakes fail silently at runtime, and the one command that catches them at author time — run it after every metadata edit.
4+
---
5+
6+
# Validating Metadata
7+
8+
ObjectStack metadata is data, not code paths — so most mistakes are **not** caught
9+
by the TypeScript compiler. They pass `tsc`, load fine, and then fail **silently
10+
at runtime**. The fix is one command you run after every metadata edit:
11+
12+
```bash
13+
os validate # schema + CEL predicates + widget bindings — no artifact
14+
```
15+
16+
In a scaffolded project this is wired as `npm run validate`. Your generated
17+
`AGENTS.md` instructs coding agents (Claude Code, Cursor, Copilot) to run it
18+
after editing metadata.
19+
20+
## Why typecheck isn't enough
21+
22+
Two classes of bug type-check cleanly but break at runtime:
23+
24+
### 1. Bare-field predicates
25+
26+
Predicates — an action's `visible`/`disabled`, a field's `requiredWhen`, a
27+
validation rule, a flow condition, a sharing rule — are **CEL expressions**, and
28+
they reference record fields through the `record.` scope:
29+
30+
```ts
31+
// ✗ Wrong — `done` is a bare reference. It type-checks (it's just a string),
32+
// but at runtime it resolves to null → the action is hidden on EVERY record.
33+
{ name: 'mark_done', visible: '!done' }
34+
35+
// ✓ Right
36+
{ name: 'mark_done', visible: '!record.done' }
37+
```
38+
39+
This is the trap behind the recurring "the button never shows / the rule never
40+
fires" bugs (#2183/#2185). `os validate` parses every predicate and checks that
41+
each `record.<field>` exists on the target object, so the bare ref fails the
42+
gate with a located, did-you-mean message instead of shipping.
43+
44+
### 2. Dangling widget bindings
45+
46+
A dashboard widget points at a `dataset` and reads `dimensions`/`values` from it.
47+
If a name doesn't resolve, the chart renders **empty** — no error (ADR-0021).
48+
`os validate` resolves every binding against the declared datasets and fails on a
49+
dangling one.
50+
51+
## The one gate, two entry points
52+
53+
`os validate` and `os build` (alias `os compile`) run the **same** validator:
54+
55+
| | `os validate` | `os build` |
56+
|---|---|---|
57+
| Protocol schema (Zod) | ✓ | ✓ |
58+
| CEL / predicate validation | ✓ | ✓ |
59+
| Widget-binding integrity | ✓ | ✓ |
60+
| Emits `dist/objectstack.json` | — | ✓ |
61+
62+
So `os validate` is the fast inner-loop check (no artifact); `os build` is what
63+
you run when you need the deployable artifact. A config that passes `os validate`
64+
will not fail `os build` on schema/predicate/binding grounds.
65+
66+
<Callout type="info">
67+
`os lint` is a **separate** pass — style and convention checks (snake_case
68+
naming, required labels, namespace prefixes, data-model patterns). Run it too,
69+
but it does not replace `os validate`, and `os validate` does not replace it.
70+
</Callout>
71+
72+
## The workflow
73+
74+
```bash
75+
# after editing any *.object.ts / *.view.ts / *.action.ts / *.flow.ts / *.dashboard.ts
76+
npm run validate # os validate — schema + predicates + bindings
77+
npm run typecheck # tsc --noEmit — types against @objectstack/spec
78+
```
79+
80+
**Rule of thumb: never report a metadata change as done until `npm run validate`
81+
passes.** In the example apps the equivalent is `pnpm --filter <pkg> validate`
82+
(`pnpm verify` in `app-showcase`, which chains validate + typecheck + test).
83+
84+
### Checking a single expression
85+
86+
To validate one CEL expression *before* you write it into a file — for example
87+
inside an AI build loop — call the `validate_expression` agent tool, which runs
88+
the same predicate validator inline. See the
89+
[objectstack-formula skill](/docs/guides/formula).
90+
91+
## In CI
92+
93+
Both commands support `--json` and exit non-zero on failure:
94+
95+
```yaml
96+
- name: Validate ObjectStack metadata
97+
run: npx objectstack validate --strict --json
98+
```
99+
100+
## See also
101+
102+
- [Command Line Interface](/docs/getting-started/cli) — full `os validate` / `os build` reference
103+
- [Formula & Predicates](/docs/guides/formula) — the CEL surface and `record.` scope
104+
- [Data Modeling](/docs/guides/data-modeling) — objects, fields, validations, sharing

‎examples/AGENTS.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Examples — Agent Instructions
2+
3+
These are reference ObjectStack apps (`app-crm`, `app-todo`, `app-showcase`).
4+
They double as the canonical shape of a user-scaffolded project, so the metadata
5+
here should always be correct.
6+
7+
## Verify after every metadata change
8+
9+
ObjectStack metadata mistakes fail **silently at runtime**, not at edit time:
10+
11+
- a bare field ref in a predicate (`done` instead of `record.done`) evaluates to
12+
`null` and silently hides the action/validation on every record (#2183/#2185);
13+
- a dangling dashboard widget binding renders an empty chart (ADR-0021).
14+
15+
`objectstack validate` catches both at author time. It runs the same gates as
16+
`objectstack build` — Zod protocol schema, CEL/predicate validation with
17+
`record.<field>` existence checks, and widget-binding integrity — but emits no
18+
`dist/`, so it is the fast inner-loop check. It exits non-zero with a located,
19+
corrective message.
20+
21+
After editing any `*.object.ts` / `*.view.ts` / `*.action.ts` / `*.flow.ts` /
22+
`*.dashboard.ts`, run the gate for that example:
23+
24+
```bash
25+
pnpm --filter @objectstack/example-crm validate # + typecheck, test
26+
pnpm --filter @objectstack/example-todo validate
27+
pnpm --filter @objectstack/example-showcase verify # validate + typecheck + test
28+
```
29+
30+
**Do not report a metadata change as done until `validate` passes.** When in
31+
doubt about how to express a predicate or binding, consult the relevant
32+
`objectstack-*` skill (e.g. `objectstack-formula` for CEL, `objectstack-ui` for
33+
views/actions/dashboards).

‎examples/app-crm/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
"dev": "objectstack dev",
1515
"start": "objectstack start",
1616
"build": "objectstack build",
17+
"validate": "objectstack validate",
1718
"typecheck": "tsc --noEmit",
1819
"test": "vitest run"
1920
},

‎examples/app-showcase/package.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,10 @@
1515
"dev": "objectstack dev --seed-admin",
1616
"start": "objectstack start",
1717
"build": "objectstack build",
18+
"validate": "objectstack validate",
1819
"typecheck": "tsc --noEmit",
1920
"test": "vitest run",
20-
"verify": "pnpm typecheck && pnpm test",
21+
"verify": "pnpm validate && pnpm typecheck && pnpm test",
2122
"test:smoke": "playwright test --config=playwright.config.ts"
2223
},
2324
"dependencies": {

‎examples/app-todo/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
"dev": "objectstack dev",
1515
"start": "objectstack start",
1616
"build": "objectstack build",
17+
"validate": "objectstack validate",
1718
"typecheck": "tsc --noEmit",
1819
"test": "objectstack test",
1920
"test:ai": "tsx test/ai.test.ts",

‎packages/cli/src/commands/validate.ts‎

Lines changed: 38 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import chalk from 'chalk';
55
import { ZodError } from 'zod';
66
import { ObjectStackDefinitionSchema, normalizeStackInput } from '@objectstack/spec';
77
import { loadConfig } from '../utils/config.js';
8+
import { validateStackExpressions } from '../utils/validate-expressions.js';
89
import { validateWidgetBindings } from '../utils/validate-widget-bindings.js';
910
import {
1011
printHeader,
@@ -19,7 +20,8 @@ import {
1920
} from '../utils/format.js';
2021

2122
export default class Validate extends Command {
22-
static override description = 'Validate ObjectStack configuration against the protocol schema';
23+
static override description =
24+
'Validate ObjectStack configuration against the protocol schema, CEL expressions, and widget bindings (no artifact emitted)';
2325

2426
static override args = {
2527
config: Args.string({ description: 'Configuration file path', required: false }),
@@ -70,6 +72,37 @@ export default class Validate extends Command {
7072
this.exit(1);
7173
}
7274

75+
// 2b. Expression validation (ADR-0032 §1a/1b) — the same gate `os build`
76+
// runs, brought to the read-only check so authors catch it without
77+
// emitting an artifact. CEL predicates in actions/validations/flows/
78+
// sharing/hooks are checked for syntax AND that `record.<field>`
79+
// references resolve on the target object. This is what catches a
80+
// BARE field ref (`done` instead of `record.done`) that would
81+
// otherwise silently hide an action on every record (#2183/#2185).
82+
if (!flags.json) printStep('Validating expressions (ADR-0032)...');
83+
const exprIssues = validateStackExpressions(result.data as Record<string, unknown>);
84+
const exprErrors = exprIssues.filter((i) => i.severity !== 'warning');
85+
const exprWarnings = exprIssues.filter((i) => i.severity === 'warning');
86+
87+
if (exprErrors.length > 0) {
88+
if (flags.json) {
89+
console.log(JSON.stringify({
90+
valid: false,
91+
errors: exprErrors,
92+
warnings: exprWarnings,
93+
duration: timer.elapsed(),
94+
}, null, 2));
95+
this.exit(1);
96+
}
97+
console.log('');
98+
printError(`Expression validation failed (${exprErrors.length} issue${exprErrors.length > 1 ? 's' : ''})`);
99+
for (const i of exprErrors.slice(0, 50)) {
100+
console.log(` • ${i.where}: ${i.message}`);
101+
console.log(chalk.dim(` source: \`${i.source}\``));
102+
}
103+
this.exit(1);
104+
}
105+
73106
// 3. Dashboard widget reference integrity (issue #1721) — a semantic
74107
// cross-reference pass the protocol schema cannot express: every
75108
// widget's `dataset`/`dimensions`/`values` and chartConfig
@@ -108,7 +141,7 @@ export default class Validate extends Command {
108141
valid: true,
109142
manifest: config.manifest,
110143
stats,
111-
warnings: widgetWarnings,
144+
warnings: [...exprWarnings, ...widgetWarnings],
112145
duration: timer.elapsed(),
113146
}, null, 2));
114147
return;
@@ -117,6 +150,9 @@ export default class Validate extends Command {
117150
// 5. Warnings (non-blocking)
118151
const warnings: string[] = [];
119152

153+
for (const i of exprWarnings) {
154+
warnings.push(`${i.where}: ${i.message}`);
155+
}
120156
for (const f of widgetWarnings) {
121157
warnings.push(`${f.where}: ${f.message}`);
122158
}

‎packages/create-objectstack/src/index.ts‎

Lines changed: 45 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -335,6 +335,49 @@ function rewriteProjectIdentity(
335335
md = md.replace(/^#\s+.*$/m, `# ${title}`);
336336
fs.writeFileSync(readmePath, md);
337337
}
338+
339+
writeAgentGuides(targetDir, title, projectName);
340+
}
341+
342+
// Emit the cross-agent guidance file (AGENTS.md) and the GitHub Copilot variant
343+
// (.github/copilot-instructions.md) from the shared template. This is what tells
344+
// the coding agent to run `npm run validate` after editing metadata — the gate
345+
// that catches bare-field predicates and dangling bindings that otherwise fail
346+
// silently at runtime. Skip either file if the template already shipped its own,
347+
// so a curated template can override the default.
348+
function writeAgentGuides(targetDir: string, title: string, projectName: string) {
349+
const templatePath = path.join(BUNDLED_TEMPLATES_DIR, 'AGENTS.md');
350+
let template: string;
351+
try {
352+
template = fs.readFileSync(templatePath, 'utf8');
353+
} catch (err: any) {
354+
if (err?.code === 'ENOENT') return; // bundled template absent — nothing to emit
355+
throw err;
356+
}
357+
358+
const rendered = template
359+
.replace(/\{\{PROJECT_TITLE\}\}/g, title)
360+
.replace(/\{\{PROJECT_NAME\}\}/g, projectName);
361+
362+
// Atomic exclusive-create (the `wx` flag) instead of existsSync()+writeFileSync():
363+
// it fails with EEXIST if the file already exists, so a curated template that
364+
// ships its own guide is preserved — without the check-then-write TOCTOU race a
365+
// separate existence check introduces.
366+
writeIfAbsent(path.join(targetDir, 'AGENTS.md'), rendered);
367+
368+
const copilotPath = path.join(targetDir, '.github', 'copilot-instructions.md');
369+
fs.mkdirSync(path.dirname(copilotPath), { recursive: true });
370+
writeIfAbsent(copilotPath, rendered);
371+
}
372+
373+
// Create a file only if it does not already exist, atomically — no time-of-check
374+
// to time-of-use gap between an existence test and the write.
375+
function writeIfAbsent(filePath: string, contents: string) {
376+
try {
377+
fs.writeFileSync(filePath, contents, { flag: 'wx' });
378+
} catch (err: any) {
379+
if (err?.code !== 'EEXIST') throw err;
380+
}
338381
}
339382

340383
// ─── CLI Program ────────────────────────────────────────────────────
@@ -450,7 +493,8 @@ const program = new Command()
450493
console.log(chalk.dim(' npm install'));
451494
}
452495
console.log(chalk.dim(' npm run dev # Start development server'));
453-
console.log(chalk.dim(' npm run validate # Check configuration'));
496+
console.log(chalk.dim(' npm run validate # Verify metadata: schema + predicates + bindings'));
497+
console.log(chalk.dim(' # (run after every metadata edit — see AGENTS.md)'));
454498
if (options.skipInstall || options.skipSkills) {
455499
console.log('');
456500
console.log(chalk.bold(' AI Skills (recommended):'));

0 commit comments

Comments
 (0)