feat(flow-graph): port FlowGraph runtime + glTF KHR_interactivity (Calculator slice)#293
Draft
RaananW wants to merge 13 commits into
Draft
feat(flow-graph): port FlowGraph runtime + glTF KHR_interactivity (Calculator slice)#293RaananW wants to merge 13 commits into
RaananW wants to merge 13 commits into
Conversation
Add a pure-state architecture/port design for the FlowGraph system (docs/lite/architecture/42-flow-graph.md), a reusable block-porting skill (.github/copilot/skills/port-flow-graph-block.md), and wire both into the architecture TOC. The glTF KHR_interactivity spec is unratified (tracking BJS PR #18455), so all spec-dependent code is quarantined under flow-graph/gltf/ and the runtime core stays spec-agnostic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implements the spec-agnostic FlowGraph core for Babylon Lite as plain data + standalone functions (no classes), per docs/lite/architecture/42-flow-graph.md: - types/block-def/block-type/context: FgValue, FgType, FgBlock, FgGraph, sockets, FgBlockDef, FgContext/FgEnv, FgPendingTask, custom FgInteger/FgMatrix2D/3D. - runtime: pull data edges (recompute every read, cycle guard), push signal cascade, cancellation-safe async pending loop (dedupe + cancel mid-tick), init-priority event listener ordering, start/tick/dispose lifecycle. - event-bus: pure-data bus + standalone subscribe/pump/clear. - rich-type: defaultForType/coerceValue (incl Vector4|Matrix->Quaternion) / animationTypeForFgType — the RichType replacement. - block-registry: side-effect-free getBlockDef switch (no blocks yet; Phase 2+). - scene-flow-graph: attach/detach via onBeforeRender/onSceneDispose seams so scene-core gains only an optional type-only _flowGraphs field. Scene wiring is byte-neutral: the per-scene bundle manifest is unchanged, so non-interactivity scenes pay zero bytes. 24 new unit tests (608 total) pass; full lint clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…blocks (Phase 2) Implements the Phase 2 vertical slice of the FlowGraph port: a minimal runnable set of blocks plus the glTF KHR_interactivity load path that turns an interactive glb into a running flow graph. Runtime / blocks: - 12 blocks across events (sceneStart/sceneTick), control-flow (branch, sequence), data (get/set property + variable), math (add), debug (consoleLog) and animation (play/stop) under flow-graph/blocks/. - block-registry, sockets, fg-math and runtime helpers to support them. - runFlowGraphs + sceneAnimationCaps in scene-flow-graph.ts as the explicit public API to bind and drive loaded graphs (not auto-wired into addToScene, keeping scene-core byte-identical). - LoadedFlowGraph type in context.ts; AssetContainer.flowGraphs field. glTF loader (spec-volatile, quarantined under flow-graph/gltf/): - interactivity-parser, declaration-mapper, object-model-mapping and path-converter: parse IKHRInteractivity_Graph JSON into an FgGraph and resolve json pointers to accessors. Loud-fails on unknown ops. - gltf-feature-interactivity loader feature, registered as a lazy ["KHR_interactivity", () => import(...)] tuple in gltf-feature-registry. The tuple is never imported unless a glb declares KHR_interactivity, so all existing scenes render byte-identically (verified via content-hash invariance of every non-registry chunk). The KHR_interactivity spec is unratified; all spec-dependent code is isolated under flow-graph/gltf/ and tagged for re-sync against BJS PR #18455. Tests: 23 new unit tests (blocks, gltf parser, gltf feature); full unit suite 633 green; lint + typecheck clean. Bundle manifest regenerated for the 48 affected glTF scenes (chunk-hash renames + ~0.1KB raw bumps); bundle-size ceilings unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…lculator demo Phase 2d of the FlowGraph → Babylon Lite port. Adds the block coverage, pointer-template handling and glTF path-converter extensions needed to load and run the Khronos Calculator KHR_interactivity sample, plus a runnable demo. Runtime / blocks: - fg-math: refactor to shared binary/unary helpers; add sub, mul, div, rem, abs, floor, lt, clamp, combine2, extract2 dispatchers. - New math block defs (subtract, multiply, divide, modulo, less-than, abs, floor, combine2, extract2, clamp) + OnSelect event block (reuses the existing Pointer event channel, filtered by node index). - block-type / block-registry: enum entries + lazy registry cases. glTF bridge (spec-volatile, re-sync to BJS PR #18455): - declaration-mapper: all 11 math ops in NATIVE_OPS; EXTENSION_OPS table routes BABYLON + KHR_node_selectability; nodeConfigKey support. - interactivity-parser: ref-index extraction + relative-prefix in resolvePointerTemplate; widened value type; nodeConfigKey config handling. - path-converter: new resolve-context signature; material UV-transform get/set, node visibility set, selectability no-op (reuses the animation-pointer writer for per-texture isolation + visibility-epoch bumps). - gltf-feature-interactivity: build glTF-material → runtime-material map. API / demo: - Export runFlowGraphs from the package entry. - Add the Calculator demo (lab/lite/src/demos/calculator.ts + demo HTML + demos-config entry + asset-copy step). On load the onStart graph resets the display to "00" (texture-transform offsets), hides the minus sign, and seeds scene variables — exercising the new math/pointer/event blocks. Verified in a WebGPU browser probe. KHR_interactivity remains an unratified draft; all spec-dependent code stays quarantined under flow-graph/gltf/. Tests: 657 unit pass; lint + all typechecks clean; 198 bundle-size ceilings pass; bundle manifest regenerated. No rendering chunk changed (only the inert gltf-feature-registry chunk hashes cascade). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| * glTF `useSlerp` targets force this so interpolation runs as slerp. */ | ||
| function toQuaternion(value: FgValue): FgValue { | ||
| // A Vec4 already has x,y,z,w — reinterpret as a quaternion directly. | ||
| if (typeof value === "object" && value !== null && "w" in value && "x" in value && "y" in value && "z" in value) { |
Spell out the planned BJS FlowGraph serialization (editor / coordinator) JSON support as a concrete Phase 5 deliverable: a second parser front-end producing the same FgGraph, a className→FgBlockType map, rich-type de/serialization, and a scene-object-reference binding layer. Records that it is gated on BJS-editor interop need and on the format settling after BJS PR #18455. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… conversion blocks Broaden the Lite flow-graph block library with the non-matrix math surface and type conversions, wired through the registry and glTF declaration mapper: - fg-math.ts: ~50 type-generic dispatchers (neg/sign/rounding/exp/log/roots, full trig, min/max/pow/atan2, eq/le/gt/ge, isNaN/isInf, mix, random, bitwise and/or/xor/not + shifts + clz/ctz/popcnt, length/normalize/dot/cross, rotate2D/3D, combine3/4, extract3/4, conjugate). - blocks/math/*: 55 new data blocks (scalar, trig, comparison, constants E/PI/Inf/NaN, bitwise, vector, combine/extract, select, data-switch). - blocks/conversion/*: 6 type-conversion blocks (bool/int/float). - block-registry.ts + gltf/declaration-mapper.ts: lazy registry cases and NATIVE_OPS entries (pass-through a/b/c; explicit mapping for extract3/4, rotate2D, select). - tests: Phase 3 unit coverage (729 unit tests green). - Regenerated lab/public/bundle/manifest.json (hash-only chunk cascade from the inert gltf-feature-registry; no bundle-size change). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the matrix and quaternion math surface, using Lite's column-major storage convention (flow-graph 4x4 == core Mat4; FgMatrix2D/3D column-major), which is mathematically equivalent to BJS's row-major implementation: - fg-math.ts: column-major dispatchers for transform, matMul, transpose, determinant, inverse, compose/decompose (4x4 reuses core mat4Multiply/ mat4Invert/mat4Compose/mat4Decompose; 2x2/3x3 + transpose/determinant self-contained), plus quaternion conjugate/angleBetween/fromAxisAngle/ toAxisAngle/fromDirections. - blocks/math/*: 18 new block defs (transform-vector, combine/extract matrix 2d/3d/4x4, transpose, determinant, invert-matrix, matrix-multiplication, matrix-compose, matrix-decompose, quat-conjugate, angle-between, quaternion-from-axis-angle, axis-angle-from-quaternion, quaternion-from-directions). - interactivity-parser.ts: load float2x2/3x3/4x4 literals (column-major direct). - block-registry.ts + gltf/declaration-mapper.ts: lazy cases + NATIVE_OPS entries (glTF op strings verified against pinned BJS declarationMapper). - tests: matrix/quaternion unit coverage with hand-computed assertions (749 unit tests green). Quaternion multiply is intentionally left out of the generic Multiply block: Vec4 and Quaternion are shape-identical at runtime in Lite, so Hamilton-product dispatch would break component-wise Vec4 math. A dedicated block can be added later if needed. Note: bundle manifest regeneration deferred to the end of Phase 3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the control-flow block surface (sync + async) on Lite's pure-data runtime:
- blocks/control-flow/*: switch (flow), for-loop, while-loop, do-n, multi-gate,
wait-all, throttle, set-delay, cancel-delay.
- blocks/data/constant.ts: emits config.value (KHR_interactivity inlines value
literals, so no glTF op).
- Async blocks (throttle/set-delay) use the addPending/onTick countdown pattern;
set-delay/cancel-delay coordinate via a per-context delay registry in
ctx.executionVariables (no module-level state).
- gltf/declaration-mapper.ts: added configKeys/configArrayKeys/switchOutputs to
FgOpMapping; NATIVE_OPS entries for flow/{switch,for,while,doN,multiGate,
waitAll,throttle,setDelay,cancelDelay} + the deferred math/switch (DataSwitch).
- gltf/interactivity-parser.ts: generic config-copy mechanism (configKeys/
configArrayKeys) in pass 1, and switch flow-output naming.
- tests: control-flow Phase 3h coverage (768 unit tests green).
BJS semantics confirmed from source: ForLoop end-exclusive; DoN reset re-arms;
MultiGate sequential round-robin with isLoop wrap; WaitAll fires after all
inputs; Throttle/SetDelay durations in seconds. Throttle/SetDelay are
tick-driven in Lite (vs BJS wall-clock) so they advance deterministically with
tickFlowGraph.
Note: bundle manifest regeneration deferred to the end of Phase 3.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- blocks/events/send-custom-event.ts: pumps a CustomEvent on the shared bus (eventName + named values) and fires out. - blocks/events/receive-custom-event.ts: event block subscribed to the CustomEvent channel; filters by config.eventId, writes named values to data outputs, fires out/done. Non-matching events are ignored. - blocks/animation/value-interpolation.ts: self-contained async block (addPending/onTick); type-aware lerp (scalar, Vec2/3/4, Color3/4) with quaternion slerp; duration in seconds; a new in cancels the prior run. - block-registry.ts + gltf/declaration-mapper.ts: lazy cases + NATIVE_OPS (event/send, event/receive, variable/interpolate, pointer/interpolate) using the configKeys mechanism for eventId. - tests: send→receive round-trip (incl. non-match rejection) + numeric/vector interpolation over ticks (777 unit tests green). No runtime.ts/parser changes needed: startFlowGraph already subscribes CustomEvent receivers before Start fires. Deferred (documented): parsing the glTF events table for per-event value sockets, bezier easing control points, and pointer/interpolate accessor write-back — follow-ups aligned with BJS PR #18455 re-sync. Note: bundle manifest regeneration deferred to the end of Phase 3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Single manifest regeneration covering all Phase 3 src changes (3a-3i). Most interactivity-eligible scenes show hash-only gltf-feature-registry chunk renames. Animation/skeleton scenes (scene5/7/34/35/39, etc.) gain shared mat4-invert / mat4-multiply chunks: Phase 3f's matrix blocks reuse core mat4Invert/mat4Multiply, so Rollup's global chunk analysis hoists those core functions into shared chunks rather than duplicating them. Net effect is code sharing (no duplication); per-scene deltas are sub-KB chunk-boundary overhead and stay within size ceilings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ings fg-math previously imported core mat4Invert/mat4Multiply for the Mat4 matrix dispatchers. Those core modules are also used by the skeleton/animation runtime, and flow-graph block chunks are emitted into every glTF scene's bundle via the lazy gltf-feature-interactivity -> getBlockDef dynamic-chunk graph (even when the asset declares no interactivity). Rollup therefore hoisted mat4-invert/ mat4-multiply into shared chunks that the live skeleton chunk loads at runtime, pushing scene7 (ChibiRex) and scene247 (TeapotsGalore) over their raw-size ceilings. Give fg-math fully self-contained column-major invert/multiply (math identical to core, verified by the existing matrix unit tests) so those bytes stay inside the lazily-loaded flow-graph block chunks only. Existing scenes return to their master chunk layout; all 198 bundle-size ceilings pass. mat4Compose/mat4Decompose stay on core (they are not shared with any live scene chunk, so no extraction). Also regenerates the bundle manifest to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…/log
Phase 4a closes the declaration-mapper gap against BJS commit 8f728b23ea:
- math/quatMul: dedicated QuaternionMultiplication block (Hamilton product
via self-contained fgQuatMul); Lite's generic Multiply is component-wise.
- animation/stopAt: StopAnimation gains an optional stopAtFrame input that
defers the stop via an onTick monitor + new stopAnimationAt capability
(poses the target at the frame instead of resetting to 0).
- debug/log: ConsoleLog now expands {name} placeholders from a messageTemplate
config (faithful to BJS), reading named inputs or message-object keys.
Adds an allMappedBlockTypes() coverage guard test asserting every mapped op
resolves to a registered, type-matching block def. 784 unit tests green;
lint + typecheck clean. Manifest regen follows.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Only the inert gltf-feature-registry chunk hash changes across 48 glTF scenes (it lazily references the flow-graph block graph, which gained the QuaternionMultiplication block + ConsoleLog/StopAnimation edits). No rendering chunk changed; all 198 bundle-size ceilings pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| } | ||
|
|
||
| function serializeValue(value: FgValue): string { | ||
| if (value === null || value === undefined) { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports Babylon.js's FlowGraph (the runtime behind glTF
KHR_interactivity) to Babylon Lite, re-architected to Lite's pure-data model: plain-stateFgBlock/FgBlockDef+ standalone functions, a tree-shakablegetBlockDefregistry, and zero module-level side effects. The slice is sized to load and run the KhronosCalculator.glbsample end-to-end and ships a runnable demo.This is a draft — it's a self-contained milestone (Phases 0–2d). Phases 3–5 (broaden to ~60 blocks → full 168-op mapper → editor serialization) continue in follow-ups.
Why this is safe to land now
KHR_interactivity. Existing scenes render byte-identically; the only manifest change is the inertgltf-feature-registrychunk-hash cascade. No rendering chunk changed.KHR_interactivityis an unratified glTF draft. All spec-dependent code (parser / declaration-mapper / path-converter) lives underflow-graph/gltf/, version-tagged and mirrored against a recorded BJS commit. The runtime core never changes when the spec churns.What's included
Runtime core (spec-agnostic)
FgBlock/FgBlockDef,FgContext/FgEnv, event bus, rich types (Vec2 / integer / matrix).onBeforeRender/onSceneDisposeseams (scene._flowGraphs) — same model as animation groups; blocks touch only loader-wired accessors, never the scene.runFlowGraphs,attachFlowGraph/detachFlowGraph,createFgRuntime,AssetContainer.flowGraphs.Blocks (Calculator subset)
onStart,onSelect(KHR_node_selectability)branch,sequenceadd, sub, mul, div, rem, abs, floor, lt, clamp, combine2, extract2get/set, pointerget/setplay/stopglTF bridge (
flow-graph/gltf/, re-sync target: BJS PR #18455)EXTENSION_OPStable (BABYLON +KHR_node_selectability).Demo + docs
lab/lite/src/demos/calculator.ts+ page + gallery entry. On load, theonStartgraph resets the display to "00" (texture-transform offsets), hides the minus sign (KHR_node_visibility), and seeds scene variables — exercising the new math/pointer/event blocks. Verified in a headless WebGPU browser probe.docs/lite/architecture/42-flow-graph.md(design/port spec) and.github/copilot/skills/port-flow-graph-block.md(mechanical recipe to port a single block from BJS FlowGraph to Lite).Validation
manifest.jsonregenerated + validates (201 scenes)Visual parity was not re-run: no rendering chunk changed, so it cannot move (consistent with the Phase 2 precedent where the 4 environmental MAD failures are proven unrelated to inert-registry changes).
Known limitations / follow-ups
flow-graph/gltf/layer is the single quarantine point.OnSelectfor clickable digits is a separate enhancement.🤖 Generated with Copilot CLI