FE-629: Compile Petrinaut user code through an HIR to buffer-native programs#8981
Open
kube wants to merge 12 commits into
Open
FE-629: Compile Petrinaut user code through an HIR to buffer-native programs#8981kube wants to merge 12 commits into
kube wants to merge 12 commits into
Conversation
Introduce a typed, source-spanned, JSON-serializable HIR as the single compiler for dynamics/lambda/kernel user code: - TS->HIR lowering (const destructuring, guard if/early-return, .map comprehensions, first-class distributions) with positioned diagnostics - Schema-checked typing against Color/parameter definitions; analyses for dependency sets, distribution DAGs (nodes/edges/sinks/shared draws), binding usage and constant folding - Buffer-ABI emitters: lambdas read token attributes at statically resolved offsets (tokenValues[slotBases[slot] + attr]); kernels write place-major staging floats with deferred distribution sampling ordered by output index (exact legacy RNG-stream parity); dynamics compile to allocation-free Float64Array loops - Engine hot loops (single-run + Monte Carlo) prefer buffer programs and fall back to object-convention HIR programs per item - HirArtifacts v2 compiled in the LSP worker (sdcpn/compileHirArtifacts request) and threaded through simulation/experiment configs; the engine no longer bundles any compiler (simulation workers: ~3MB -> ~40kB) - HIR semantic lints in the LSP checker (source "hir", codes 99001+); out-of-subset code is an error and Play gates on error severity only - Remove @babel/standalone and compile-user-code.ts; fix H-6519 violations in the supply-chain example; shipped examples are the buffer-ABI coverage gate Design docs: src/hir/README.md (architecture) and src/hir/dsl-sketch.md (planned OCaml-flavoured DSL lowering to the same HIR). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Interactive playground for the HIR pipeline: type dynamics/lambda/kernel TypeScript on the left, and inspect everything it compiles to on the right — the spanned HIR tree, diagnostics with exact ranges, the inferred return type, dependency sets, the distribution DAG (derivation edges, output sinks, shared draws), and both emitted programs (buffer-ABI fast path and object-convention fallback). The model schema (parameters, places, attribute types) is editable JSON and re-checks live. Exposes the compiler via new `@hashintel/petrinaut-core/hir` and `/hir-runtime` entries (typescript declared as an optional peer — only the ./hir entry needs it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… the HIR Rebase onto the FE-769 stack (token format v2, UUID + interned strings) and extend the compiler: - New HIR nodes: string literals, string predicates (startsWith/endsWith/includes), Uuid.generate()/Uuid.from(...) sentinels - Typecheck: uuid/string attribute types; kernel outputs accept strings for string attributes and uuid/string for uuid attributes (omitted uuid = auto-generated); string .length; uuid helpers are kernel-only - Buffer-ABI v2: lambdas and dynamics compile to packed-struct byte reads through the frame's shared f64/u64/u8 views (integers rounded, booleans u8, strings resolved via the per-run pool, uuids assembled from two u64 lanes); slotBases now carry byte offsets. Kernel buffer emission is deferred (needs an RNG-ordinal sink design to preserve the stream) — the object program runs per firing meanwhile - Object programs emit uuid sentinels/string literals; analyses track generatesUuids; string-equality constant folding - Replace the deleted Babel compiler in the dev token playground with a local evaluator Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Remove the object-convention path entirely: the engine no longer decodes token records for user code, and every dynamics/lambda/kernel runs a buffer program with all dimension offsets and strides resolved at compile time and baked into the emitted code as literal constants. - Kernel buffer emitter: static writes into per-transition staging bytes at compile-time-constant offsets; RNG-consuming slots (distributions, generated/converted UUIDs) defer through sink calls emitted in (arc, token, element-declaration) order, preserving the seeded RNG stream; strings intern through the per-run pool inline - Lambda programs now receive (placeBases, indices) with strides baked in — the per-slot offset loop is gone; token layouts are computed once per arc slot at emit time - CompiledTransition carries only buffer programs + reusable scratch (placeBases/indices/staging views); encode-kernel-token.ts and the record-decode adapters are deleted - Artifacts v3 are buffer-only; shapes that don't scalarize fail compilation with hir:not-compilable (all example models compile fully, kernels included — enforced by the coverage gate) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Metrics become the fourth HIR surface and lose their new Function path: - New arrayReduce/arrayConcat HIR nodes (.reduce/.concat lowering); metric code lowers as a function body with spans mapped back to the raw text - Metric buffer programs read frames directly with statically-resolved offsets: attribute offsets/strides baked as literals, place ordinals resolved once at instantiation, reductions emitted as loops over the packed token region (no per-frame MetricState decoding) - compileHirArtifacts covers sdcpn.metrics; expression metric specs carry the compiled artifact; single-run timeline and actual-mode metric paths migrated to the same evaluator via SimulationFrameReader.getRawView() - Delete compile-metric.ts and frames/metric-state.ts; LSP metric sessions get HIR lint incl. hir:not-compilable as blocking errors - All 18 example metrics compile (reduce/concat/guards/strings) — enforced by the coverage gate Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
PR SummaryHigh Risk Overview Engine & API: Dynamics, lambdas, kernels, and Monte-Carlo expression metrics run only from precompiled Supporting changes: Actual-mode timeline frames gain Compatibility: User code outside the supported subset does not run; it is rejected with error diagnostics at compile/LSP time. Reviewed by Cursor Bugbot for commit 328afb2. Bugbot is set up for automated code reviews on this repo. Configure here. |
github-actions
Bot
added
area/deps
Contributor
There was a problem hiding this comment.
Pull request overview
This PR replaces Petrinaut's Babel + eval execution of user code (dynamics, firing rates, transition kernels, metrics) with a new HIR (high-level intermediate representation) pipeline. User code is lowered from a TypeScript subset to a typed, spanned, JSON-serializable expression tree, typechecked against the token schema, analyzed (distribution DAGs, dependency sets, determinism), and emitted as buffer-native programs that read/write packed frame buffers directly with attribute offsets/strides baked in as constants. This removes @babel/standalone and per-call token decode/encode from the simulation hot loops (worker bundles shrink ~3 MB → ~40 kB), surfaces out-of-subset code as blocking LSP error diagnostics, and threads compiled artifacts through the simulation/Monte-Carlo start flows. It fits into the petrinaut-core/petrinaut packages as deliberately private API and lays groundwork for a future DSL frontend and WASM/GPU backends.
Changes:
- New HIR pipeline in
petrinaut-core/src/hir/(lowering, typecheck, analyses, lints, buffer emitters) plus engine execution of buffer programs and deletion of Babel/compile-user-code/compile-metric/record adapters. - Artifact compilation moved into the LSP worker (
sdcpn/compileHirArtifacts) and threaded through simulation/experiment configs; metric drawers and the timeline now rely on LSP diagnostics + async HIR artifacts instead of synchronouscompileMetric. - LSP/UX: HIR lints (codes
99001+,source: "hir"), error-only Play gating with an amber warnings-only indicator, and updated user docs.
Reviewed changes
Copilot reviewed 95 out of 96 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
yarn.lock |
Drops @babel/standalone/@types/babel__standalone; adds optional typescript peer dependency for petrinaut-core |
.../metrics/view-metric-drawer.tsx |
Removes synchronous compileMetric compile-check; validation now sourced from metric LSP diagnostics |
.../metrics/create-metric-drawer.tsx |
Same removal of live compileMetric preview compile in the create flow |
.../metrics/metric-form.tsx |
Updates comment to reflect HIR-based compilability rather than compileMetric |
.../experiments/create-experiment-drawer.tsx |
buildMetricSpecs returns ExperimentMetricSpecInput; expression metrics validated for non-empty code, compiled via HIR at experiment start |
.../simulation-timeline/use-streaming-data.ts |
New async useTimelineMetric hook compiling the selected metric via requestHirArtifacts + createHirMetricEvaluator |
.../BottomBar/diagnostics-indicator.tsx |
Adds amber warning status for warnings/hints-only state (Play still allowed) |
petrinaut-core/src/hir.ts (+ hir-runtime) |
New public entry exposing compileHirArtifacts and HIR types |
.../simulation/engine/buffer-transition.ts |
New buffer-ABI transition execution (kernel sinks, UUID lanes, staging) |
.../simulation/engine/build-simulation.ts |
Instantiates buffer programs, validates stale/missing artifacts, allocates reusable scratch |
.../simulation/frames/hir-metric.ts |
New HIR metric evaluator with cached place indices + per-run string-pool rebind |
.../simulation/frames/frame-reader.ts, simulation/api.ts |
getPlaceTokens gains optional color? param (currently ignored) |
.../lsp/lib/check-hir.ts |
HIR diagnostic-code registry and per-item HIR linting (subsetSeverity: "error") |
.../simulation-timeline/series-config/metric.ts |
Timeline metric config switches from places/buildMetricState to an injected evaluator |
react/experiments/context |
Adds ExperimentMetricSpecInput type for pre-artifact metric specs |
docs: petri-net-extensions.md, simulation.md |
Document supported subset (incl. metrics), error/warning split, and error-only Play gating |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Comment on lines
318
to
321
| return { | ||
| store, | ||
| metricError: compiledMetric.error, | ||
| metricError: timelineMetric.error, | ||
| }; |
Runs scenarios (dynamics, lambda combinations, kernel firings, Monte-Carlo experiment with expression metrics) through the public in-process createMonteCarloSimulator API. Feature-detects the HIR entry so the same script runs on pre-HIR baselines; emits per-scenario build/run medians, heap growth, and a work checksum that must match across variants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment on lines
+137
to
+146
| useEffect(() => { | ||
| if (!metric) { | ||
| // No compile needed; callers key results off the metric identity, so | ||
| // stale state is simply ignored. | ||
| return; | ||
| } | ||
|
|
||
| let cancelled = false; | ||
| const key = `${metric.id}\u0000${metric.code}`; | ||
| requestHirArtifacts( |
--no-warmup/--samples/--only for fresh-process cold runs; --compile isolates user-code compilation (HIR pipeline vs Babel per-item). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b4b537e. Configure here.
| const selected = ONLY === null ? SCENARIOS : [SCENARIOS[Number(ONLY)]]; | ||
| for (const scenario of selected) { | ||
| for (let index = 0; index < WARMUPS; index += 1) { | ||
| sample(scenario); |
There was a problem hiding this comment.
Bench --only invalid index crashes
Low Severity
When --only is passed a non-numeric or out-of-range index, SCENARIOS[Number(ONLY)] is undefined, yet the harness still calls sample(scenario) and dereferences scenario fields, which throws instead of reporting a clear usage error.
Reviewed by Cursor Bugbot for commit b4b537e. Configure here.
Comment on lines
+137
to
+142
| useEffect(() => { | ||
| if (!metric) { | ||
| // No compile needed; callers key results off the metric identity, so | ||
| // stale state is simply ignored. | ||
| return; | ||
| } |
Comment on lines
+145
to
+149
| const key = `${metric.id}\u0000${metric.code}`; | ||
| requestHirArtifacts( | ||
| { ...petriNetDefinition, metrics: [metric] }, | ||
| extensions, | ||
| ) |
Comment on lines
+46
to
49
| const hasErrors = !!formError || hasLspErrors; | ||
| const totalErrorCount = (formError ? 1 : 0) + lspErrorCount; | ||
| const firstError = formError ?? firstLspMessage ?? undefined; | ||
| const canSave = canSubmit && !hasErrors && !isSubmitting && !isDefaultValue; |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment on lines
+429
to
+437
| // Metric token counts are dynamic, so any scalar index is allowed; | ||
| // metric code is expected to guard with `.length` (out-of-range | ||
| // reads land in other places' token bytes or zeroed capacity). | ||
| const index = this.scalar(this.eval(expr.index, env)); | ||
| const stride = this.layoutOf(target.elements).strideBytes; | ||
| const strideCode = stride === 0 ? "0" : `(${index}) * ${stride}`; | ||
| return { | ||
| kind: "token", | ||
| baseCode: `placeOffsets[__places[${target.ordinal}]] + ${strideCode}`, |
Comment on lines
+114
to
+119
| export type HirArtifacts = { | ||
| version: 3; | ||
| dynamics: Record<string, HirDynamicsArtifact>; | ||
| lambdas: Record<string, HirLambdaArtifact>; | ||
| kernels: Record<string, HirKernelArtifact>; | ||
| metrics: Record<string, HirMetricArtifact>; |
Comment on lines
+138
to
+142
| if (!metric) { | ||
| // No compile needed; callers key results off the metric identity, so | ||
| // stale state is simply ignored. | ||
| return; | ||
| } |
Comment on lines
+144
to
+149
| let cancelled = false; | ||
| const key = `${metric.id}\u0000${metric.code}`; | ||
| requestHirArtifacts( | ||
| { ...petriNetDefinition, metrics: [metric] }, | ||
| extensions, | ||
| ) |
Comment on lines
+113
to
115
| // Live validation (TypeScript + HIR semantic/compilability checks) comes | ||
| // from the metric LSP session diagnostics. | ||
| const values = useStore(form.store, (state) => state.values); |
Comment on lines
+11
to
+18
| - `authoring/metric/` and `authoring/scenario/` compile metric/scenario | ||
| inputs (`new Function` with same-realm hardening from | ||
| `authoring/sandbox.ts`). Dynamics/lambda/kernel code is compiled solely | ||
| through the HIR pipeline (`src/hir/README.md`) into | ||
| `SimulationInput.hirArtifacts` — produced off-engine (LSP worker) and | ||
| instantiated dependency-free via `src/hir-runtime.ts`. Buffer-ABI programs | ||
| read/write packed token bytes directly (`engine/buffer-transition.ts`); | ||
| unsupported HIR shapes are compile errors. |
| @@ -0,0 +1,10 @@ | |||
| --- | |||
| "@hashintel/petrinaut": patch | |||
| "@hashintel/petrinaut-core": patch | |||
Comment on lines
+130
to
+132
| if (code.trim() === "") { | ||
| return []; | ||
| } |
Comment on lines
+379
to
+388
| // Compile the net's user code to HIR artifacts in the language worker — | ||
| // the simulation engine has no compiler of its own. | ||
| const { artifacts } = await requestHirArtifacts( | ||
| simulationSdcpn, | ||
| extensionsRef.current, | ||
| ); | ||
| sim = await createSimulation({ | ||
| sdcpn: simulationSdcpn, | ||
| extensions: extensionsRef.current, | ||
| hirArtifacts: artifacts, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🌟 What is the purpose of this PR?
Petrinaut user code (dynamics, firing rates, transition kernels, metrics) was executed by stripping TypeScript with Babel and
eval-ing it against per-call decoded token objects. That made the code impossible to analyze statically — e.g. knowing the probability-distribution structure of a kernel, or whether a rate depends on parameters — and slow, since every evaluation decoded packed frame bytes into records and back.This PR introduces an HIR (high-level intermediate representation): a typed, source-spanned, JSON-serializable expression tree that every user-code surface compiles through. Goals:
evalof raw user text. Simulation worker bundles shrank from ~3 MB to ~40 kB.And what it enables next: a Petrinaut DSL frontend lowering to the same HIR (TS ⇄ DSL migration becomes pretty-printing), WASM/GPU backends (the buffer ABI is already the right shape: scalars, static offsets, no GC), engine optimizations informed by analyses (e.g. constant-rate detection), and artifact caching.
Note
This HIR is a first version and should not be considered definitive. The node set, the accepted TypeScript subset, the artifact format, and the buffer ABI are all expected to evolve (they already went through three artifact revisions inside this PR). It is deliberately private API: nothing outside
petrinaut-core/petrinautdepends on its shape.How it fits together
flowchart LR subgraph authoring["Authoring"] TS["TypeScript user code<br/>dynamics · rates · kernels · metrics"] DSL["Petrinaut DSL<br/>(future)"] end subgraph hir["HIR pipeline (this PR — runs in the language worker)"] LOWER["Lower + typecheck<br/>against the token schema"] IR[("HIR<br/>typed · spanned · JSON")] ANALYZE["Analyses<br/>distribution DAG · dependencies · lints"] EMIT["Emit buffer programs<br/>offsets/strides baked as constants"] end subgraph runtime["Simulation workers"] ENGINE["Engine hot loops<br/>packed frame buffers (format v2)"] FUTURE["WASM / GPU backends<br/>(future)"] end LSPD["Editor diagnostics<br/>exact source ranges"] TS --> LOWER DSL -.-> LOWER LOWER --> IR IR --> ANALYZE --> LSPD IR --> EMIT -->|"artifacts (plain JS sources)"| ENGINE EMIT -.-> FUTURE style DSL stroke-dasharray: 5 5 style FUTURE stroke-dasharray: 5 5Touched by the HIR: the four code surfaces above, the LSP checker/worker (new lints + compile request), the engine hot loops (buffer-ABI execution), and the simulation/experiment start flows (artifact threading). Not touched: the frame format itself, the net editor, scenarios (still sandboxed
new Function, run once at start) and visualizers (JSX in the UI) — those are candidates for later HIR/DSL surfaces.🔗 Related links
🔍 What does this change?
petrinaut-core/src/hir/, entries./hir+ dependency-free./hir-runtime): lowering (incl. destructuring, guardifs,.map/.reduce/.concat,Distribution.*,Uuid.*, string predicates), schema typechecking, analyses, lints, and buffer-program emitters for all four surfaces. Design doc:src/hir/README.md; DSL sketch:src/hir/dsl-sketch.md.f64/u64/u8views at literal offsets (placeBases+ baked strides); kernels write per-transition staging with RNG-consuming slots deferred through declaration-order sinks (exact RNG-stream preservation); dynamics compile to allocation-free loops; metrics emit reduction loops over the token region (the per-frameMetricStatefull decode is gone). Strings resolve via the per-run intern pool; uuids as bigints from two u64 lanes.CompiledTransitioncarries only buffer programs + reusable scratch; Babel (@babel/standalone),compile-user-code.ts,compile-metric.ts, and all record decode/encode adapters are deleted.sdcpn/compileHirArtifactsrequest), threaded through simulation/Monte-Carlo configs; items with no artifact fail per-item at build.99001+andsource: "hir"; out-of-subset or non-compilable code is a blocking error; Play now gates on error-severity diagnostics only (warnings/hints show an amber indicator).📊 Benchmarks
Measured with
libs/@hashintel/petrinaut-core/benches/bench.mjs(yarn build && node --expose-gc benches/bench.mjs), which runs identical scenarios through the public in-processcreateMonteCarloSimulatorAPI on this branch and onmain(bfb99e7570, Babel path). The script feature-detects the HIR entry, so the same file runs on both; a per-run work checksum (RNG state + frames + token bytes) matched on every scenario, i.e. both sides did bit-identical simulation work.Warm (median of 5 samples after 2 warmups) and cold (fresh process, no warmup — the first-click experience):
(The 1,000-run row is a single sample per branch — at this duration the process is self-warming, so warm/cold no longer differ. The speedup holds steady at experiment scale: ~6 min → ~40 s.)
Memory (1,000-run experiment, via
/usr/bin/time -l+ forced-GC retained heap):Footprint is unchanged: peak RSS is dominated by the recorded frame buffers, which are byte-identical on both branches (the HIR changes how user code reads/writes frames, not how they are stored). The HIR allocates far less transient garbage per frame (no per-token record objects) — that shows up as the speedup, not as a smaller footprint.
Compilation (10 programs: 1 dynamics + 4 lambdas + 2 kernels + 3 metrics), isolated from simulation:
new Function)Experiments are the headline (the old path decoded every place's tokens into records per metric per frame); dynamics/lambdas gain ~1.8× from removing the record decode/encode; kernels are cold-path (once per firing) so they barely move. There is no compile-time trade-off — the HIR pipeline compiles faster than Babel despite doing typechecking and analysis, on top of the ~3 MB → ~40 kB simulation-worker bundle reduction. Flags:
--no-warmup,--samples N,--only <scenario>,--runs N(experiment run count),--compile.Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
petri-net-extensions.mddocuments the supported code subset (now incl. metrics) and the error/warning split;simulation.mdupdated for error-only Play gating. Screenshots showing the diagnostics indicator may want refreshing (new amber warnings-only state).🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
constdestructuring, guard clauses,.map/.reduce/.concat,Math.*, distributions,Uuid.*).new Functionpath (cold, once per run); visualizers keep JSX/Babel in the UI package..reduceinside a conditional branch is evaluated eagerly (pure, so semantics hold — just possible wasted work).storybook buildoutput fails to render for all stories in this package (pre-existingreact/jsx-runtimeissue); useyarn dev.🐾 Next steps
materializeEngineFramein the single-run engine (buffer path needs only offsets/counts + token bytes)src/hir/dsl-sketch.md)🛡 What tests cover this?
❓ How to test this?
cd libs/@hashintel/petrinaut && yarn dev→ open HIR → Playground: type code across the four surfaces, break it (add aforloop), edit the schema (make an attribute"integer"and feed it a Distribution)yarn workspace @hashintel/petrinaut-core test:unit run🤖 Generated with Claude Code