Bundle-manifest gate is a CI-GPU-render measurement, not a build artifact — not reproducible locally (blocks engine PRs)

[RaananW]

Bundle-manifest gate is a CI-GPU-render measurement, not a build artifact — not reproducible locally (blocks engine PRs)

Summary

The per-scene bundle-size manifests (lab/public/bundle/manifest/<scene>.json) that the Bundle Size CI job validates as "committed and in sync" are not deterministic build artifacts. Their runtimeChunks list is produced by rendering each scene in a headless browser and recording which JS chunks get fetched over the network. That measurement only works in the CI Linux environment (which enables SwiftShader software-WebGPU), so the manifests cannot be regenerated correctly on a developer machine (macOS confirmed; likely any non-CI environment).

This makes the mandatory "regenerate and commit the manifest on every PR" workflow (GUIDANCE.md) effectively impossible to satisfy locally for any PR that changes which chunks a GPU scene fetches. It is a significant, recurring source of red CI on engine PRs (surfaced while working on #302).

Important: this is NOT a build-determinism bug

The compiled bundles themselves are byte-for-byte deterministic across OSes (pinned Rollup/esbuild; tree-shaking is pure JS). Verified: a local build emitted all 86 scene5-*.js chunk files, identical to what CI builds. Only the measured runtimeChunks subset differs. Developers' shipped output is safe — the problem is isolated to the manifest measurement step.

Mechanism (root cause)

1. The Bundle Size job (azure-pipelines.yml, job BundleSize, vmImage: ubuntu-latest) runs pnpm build:bundle-scenes then pnpm validate:bundle-manifest, which compares committed manifests against a fresh rebuild.
2. runtimeChunks is written by measurePage() in scripts/bundle-scenes-core.ts (~line 1247): it launches headless Chrome, loads bundle-<scene>.html, waits up to 50s for canvas.dataset.ready === "true", and records every /bundle/*.js response as a fetched chunk. If the scene never reaches ready (no working WebGPU), it silently measures whatever partial set was fetched (catch block ~line 1279: "BJS pages may not reach ready state without GPU — just measure fetched JS").
3. measurementBrowserArgs() (~line 944) only adds the SwiftShader software-WebGPU flags (--enable-features=Vulkan, --use-vulkan=swiftshader, --use-angle=swiftshader, --ignore-gpu-blocklist) when process.env.CI is set. Locally you get only --enable-unsafe-webgpu.

Net effect: whether a scene's skeleton/morph/PBR dynamic-import chunks are counted depends entirely on how completely the measuring browser renders the scene.

Evidence

Same commit, same scene5 (Alien skinned model), same bundles on disk — three different manifests purely from the render environment:

Environment	canvas.ready	scene5 runtimeChunks	rawKB
CI (ubuntu + SwiftShader)	reaches ready	16	91.6
Local macOS, no CI env	partial	7	53.8
Local macOS, CI=1 (SwiftShader)	fails (Vulkan/SwiftShader unsupported on macOS)	1	47.4

The CI value of 16 is canonical and reproducible (confirmed: PR #343's build 55557 passed its own validate:bundle-manifest against the committed 16-chunk manifest). The local values are wrong because headless Chrome on macOS can't drive the same WebGPU path.

Impact

• Any engine PR that changes a GPU scene's fetched-chunk set cannot produce a correct manifest locally -> Bundle Size drift failure until regenerated in the CI env.
• Developers "fight" the gate: local pnpm build:bundle-scenes produces divergent manifests that must not be committed, but committing nothing also fails the drift check.
• GUIDANCE.md instructs devs to run pnpm build:bundle-scenes and commit the regenerated manifests as part of every PR — which is not achievable off the CI Linux+SwiftShader environment.
• Concretely blocking now: the tree-shaking fix in perf(ci): cut cloud parity time — public static site + sharded BrowserStack (CDP) #302 (commit fb5011fb) legitimately adds the primitive-resolver chunk to scene257/scene260 (+ related primitive/sampler scenes); their manifests can only be regenerated on CI.

Options to fix the gate (for discussion)

1. CI regenerates + commits manifests. Add a pipeline step (or bot) that rebuilds manifests on the ubuntu agent and commits them back to the PR branch, removing the local-regen burden entirely.
2. Make local regen reproducible. Provide a pinned linux/amd64 container (or documented flags) with working SwiftShader-WebGPU so any dev on any OS can regenerate byte-identical manifests. (Note: SwiftShader-Vulkan does not work on macOS host directly — a Linux container is the realistic path.)
3. Gate on stable/static data instead of a render measurement. Validate the statically-derived built chunk graph and/or the existing size ceilings (scene-config.json maxRawKB) rather than the GPU-render-measured runtimeChunks fetch set. Keeps the size signal without the environment dependency.

Suggested acceptance criteria

• A developer on macOS/Windows/Linux can produce a manifest that matches CI's validate:bundle-manifest byte-for-byte, OR the gate no longer depends on a GPU-render measurement, OR CI owns manifest regeneration so contributors never hand-generate them.
• The chosen approach is documented in GUIDANCE.md.

References

• scripts/bundle-scenes-core.ts — measurePage() (~1247), measurementBrowserArgs() (~944), catch block (~1279)
• azure-pipelines.yml — BundleSize job (build:bundle-scenes + validate:bundle-manifest)
• scripts/validate-bundle-manifest.ts
• GUIDANCE.md — "The per-scene bundle manifest is MANDATORY on every PR"
• Surfaced during PR perf(ci): cut cloud parity time — public static site + sharded BrowserStack (CDP) #302; real engine fix landed in fb5011f

[RaananW]

Correction: the local shortfall is NOT "no WebGPU" — the render throws mid-composition

The original write-up attributed the divergent local manifest to "headless Chrome on macOS can't drive the same WebGPU path / no working WebGPU." Direct instrumentation shows that framing is inaccurate and should be corrected.

Ground truth (instrumented scene5 render, local macOS, non-CI args --enable-unsafe-webgpu)

• navigator.gpu.requestAdapter() → adapter OK. WebGPU is available locally.
• All remote glTF assets load 200 (playground.babylonjs.com/scenes/Alien/Alien.gltf + .bin + 3 textures).
• canvas.dataset.ready never becomes "true" — but not because of a missing adapter. The render throws partway through shader composition:

Error: morph _postCompose: expected to rewrite 2 binding declarations, rewrote 0
  at scene5-morph-fragment-*.js   (patchMorphStorage)
  <- scene5-pbr-renderable-*.js
  <- scene5.js

• The abort point is non-deterministic across runs: one run stopped at 7 chunks (before morph-fragment/pbr-renderable even loaded), another run got further (morph-fragment + pbr-renderable fetched) and then threw in patchMorphStorage. So the "partial fetched set" that measurePage silently records varies run-to-run.

Why this matters for the gate

The measurement mechanism conclusion is unchanged and still correct: runtimeChunks is a render-completeness measurement, not a build artifact, so it can't be reproduced off the CI env. But the reason the local render is incomplete is:

• NOT "no WebGPU adapter."
• It is a render-time throw in packages/babylon-lite/src/shader/fragments/morph-fragment-core.ts patchMorphStorage() (line ~52): the CPU-side WGSL rewrite expects exactly two @group(1)@binding(N) var<uniform> (morphDeltas|morph): decls and throws when it finds 0. Under CI's SwiftShader path the scene renders to completion (16 chunks, canonical); under the local Metal-backed WebGPU path the morph/PBR shader-composition path hits this throw and aborts.

Follow-up spawned

The patchMorphStorage throw is itself a candidate latent engine bug (a CPU-side WGSL string rewrite that finds 0 decls where it asserts 2 — possibly a double-invocation / shader-cache idempotency issue, since the rewrite flips var<uniform>→var<storage, read> and a second pass over the same WGSL would find 0). It surfaces here only as measurement noise, but it should be investigated separately to confirm it cannot also manifest on the CI/parity render path. Not blocking the gate-design discussion above.

Correcting the record so the fix direction isn't chosen based on a "macOS lacks WebGPU" premise, which is false.