ForceFreeStates - PERF! - 🚨 Precompile the solve path and pin the harness to one build mode

[d-burg]

Release note

• Audience: users
• Numerical impact: none measured on x86_64: on feynman the precompiled build agrees with one compiled on first use on all 17 tracked quantities of the Riccati Δ′ case, in each of three runs (items 2 to 4 below). Earlier runs on Apple Silicon showed the two builds rounding differently in the last bit, which the Δ′ BVP amplified to about 1 % in the real parts of the Δ′ diagonal; at a near-marginal surface the imaginary part changed sign and magnitude, and the stored order of eigenmode_energies differed between the two builds. None of this has been reproduced on x86_64. As a precaution the regression harness still runs every case with the workload off, so existing values stay comparable, and a new diiid_n1_riccati_precompiled case tracks the precompiled build. (harness @ 6b92b2c)
• Migration: the first build after updating takes longer, because it now runs the workload. To keep the previous build, disable it with the precompile_workload preference (docs/src/set_up.md).

A PrecompileTools workload runs a small inline Solovev case through the Riccati and forward paths at build time, so a fresh process skips most first-call compilation: the first main() on the DIII-D-like example drops from 108 s to 25 s, for a one-time build of about 160 s.

The workload runs only when Julia rebuilds the package image, not at the start of each run: after any change under src/ (a pull, a branch switch or a local edit) and after a change of dependency versions, Julia version or the preference. Repeated runs on an unchanged checkout reuse the cached image. Developers who edit src/ and restart Julia often will usually want it off locally; docs/src/set_up.md says how.

Left: wall time of one run of the DIII-D-like Riccati Δ′ case from a fresh Julia process on feynman, and the one-time cost of building the image. Right: the precompiled build against one compiled on first use, for every tracked numeric quantity of that case: bit-identical on x86_64.

Regression report

Runs on feynman (SLURM, 8 threads, manifest pinned). The first three are against develop cb18e386e, the last against current develop 0e68a0553, which includes #480:

1. Every case, at 43ce87228: regress --refs cb18e386e,43ce87228 --force. All workload-off cases are unchanged apart from the known SLAYER γ run-to-run noise:

case	result
ggj_ray_q500i	4 unchanged
solovev_kinetic_nuzero	14 unchanged
diiid_error_field	8 unchanged
diiid_slayer_n1	1 changed, 16 unchanged
gal_resistive_pe	8 missing/N/A
solovev_kinetic_calculated	14 unchanged
gal_resistive_diiid	10 unchanged
solovev_n1	22 unchanged
ggj_reference	4 unchanged
solovev_multi_n	15 unchanged
diiid_n1	53 unchanged
solovev_kinetic_ntv	6 unchanged
diiid_n1_riccati	17 unchanged
solovev_kinetic_multiion	6 unchanged
diiid_multi_n	19 unchanged
efit_fixedbdy_separatrix	5 unchanged

2. The two Riccati cases, at d5246bfd4: regress --cases diiid_n1_riccati,diiid_n1_riccati_precompiled --refs cb18e386e,d5246bfd4 --force.
   • Run 1 had found that the precompiled build failed: Vacuum.reset_caches! still cleared two caches that develop had removed. d5246bfd4 fixes that.
   • That fix runs only inside the workload, so it cannot change the workload-off cases in run 1.
   • diiid_n1_riccati: 17 unchanged.
   • diiid_n1_riccati_precompiled: 17 unchanged. On this run the precompiled (aot) build matched develop's jit numbers on every quantity.

diiid_n1_riccati_precompiled

Regression Report: diiid_n1_riccati_precompiled
============================================================================
Ref 1: cb18e386e  @ cb18e386e (2026-10-02)
       env: julia 1.11.9, x86_64-linux-gnu, manifest 4bdd795f (pinned), 8 threads/8 BLAS, jit build
Ref 2: d5246bfd4  @ d5246bfd4 (2026-10-02)
       env: julia 1.11.9, x86_64-linux-gnu, manifest 4bdd795f (pinned), 8 threads/8 BLAS, aot build

!! ENVIRONMENTS DIFFER — source code is not the only variable in this comparison:
     - jit vs aot build (precompile workload); results differ between build modes
   Differences below may be environment artifacts. Re-run with --force to rebuild
   both refs in the current environment.
----------------------------------------------------------------------------
Quantity                       cb18e386e      d5246bfd4      Diff     Status
----------------------------------------------------------------------------
delta prime (BVP diagonal)     [5 elem]       [5 elem]       0.0e+00  OK    
delta prime (raw side-major)   [10 elem]      [10 elem]      0.0e+00  OK    
edge coil response delta_coil  [10 elem]      [10 elem]      0.0e+00  OK    
total energy Re(et[1])         8.037196e-01   8.037196e-01   0.0e+00  OK    
plasma energy Re(ep[1])        -1.344751e+00  -1.344751e+00  0.0e+00  OK    
vacuum energy Re(ev[1])        2.148470e+00   2.148470e+00   0.0e+00  OK    
total energy (all)             [35 elem]      [35 elem]      0.0e+00  OK    
# singular surfaces            5              5              0.0e+00  OK    
singular psi locations         [5 elem]       [5 elem]       0.0e+00  OK    
singular q values              [5 elem]       [5 elem]       0.0e+00  OK    
ODE steps (saved)              51             51             0.0e+00  OK    
ODE steps (total)              1647           1647           0.0e+00  OK    
mpert                          35             35             0.0e+00  OK    
npert                          1              1              0.0e+00  OK    
q0                             1.204212e+00   1.204212e+00   0.0e+00  OK    
q95                            4.781723e+00   4.781723e+00   0.0e+00  OK    
beta_n                         1.372511e+00   1.372511e+00   0.0e+00  OK    
Runtime (s)                    242.7s         53.1s                   --    
============================================================================
Summary: 17 unchanged

3. After the rename to src/Precompile.jl, at 6c6c04266: regress --cases diiid_n1,diiid_n1_riccati,diiid_n1_riccati_precompiled --refs cb18e386e,6c6c04266 --force. diiid_n1: 53 unchanged. Both Riccati cases: 17 unchanged. The rename moves no code, and the harness still detects the workload under the new file name (the precompiled case reports an aot build).

4. After merging develop, at 6b92b2c31: regress --cases diiid_n1,diiid_n1_riccati,diiid_n1_riccati_precompiled --refs 0e68a0553,6b92b2c31 --force. diiid_n1: 53 unchanged. Both Riccati cases: 17 unchanged, including the precompiled case, where develop runs compiled on first use and this branch runs from the precompiled image. The workload builds on the merged code in 203 s.

5. After the delta-review fixes, at cd4389948 (harness, test and docs only; no file under src/ changed): regress --cases diiid_n1_riccati,diiid_n1_riccati_precompiled --refs 0e68a0553,cd4389948 --force. Both cases: 17 unchanged, and the precompiled case still reports an aot build.

The Vacuum test file passes 392/392, including a test that fills all three run-filled caches and checks that both modules' reset_caches! empty them.

Notes for reviewers

• Where the last-bit difference came from on Apple Silicon: the FastInterpolations spline kernels inlined into direct_get_bfield!. It is not seen on x86_64 (see the release note).
• Harness changes:
  • each run records its build mode (aot or jit) in the environment fingerprint, and the report flags two runs compared across modes;
  • the harness's build-mode preference outranks a developer's LocalPreferences.toml;
  • a case file can inherit another with [case] base = "<case>", which is how the precompiled case is derived without a copy;
  • the workload file is recognised under either spelling, so refs from before the rename to Precompile.jl are not mislabelled on a case-sensitive file system.
• Case-only rename: src/precompile.jl became src/Precompile.jl. On a case-insensitive file system an existing checkout may keep the old spelling on disk after a pull, and the naming hook then reports precompile as missing from the Area table. A fresh checkout, or a two-step git mv, clears it.
• Cache resets: Vacuum and KineticForces each gain a reset_caches!, which the workload calls before the image is serialized.
• CI pins: develop's pins with only project_hash updated for the two new dependencies (PrecompileTools, Logging), both already in the resolved set. No package versions move.

---

🤖 Generated with Claude Code

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 2

Open (2)

• The harness hard-codes GPEC’s UUID as a literal string. This creates an easy-to-miss drift risk if… · New
• In Vacuum.reset_caches!, _PN_CACHE is cleared under _PN_CACHE_LOCK but… · New

What changed in this PR

This PR adds a PrecompileTools workload to reduce first-run JIT latency and updates the regression harness to explicitly model/record “jit vs aot” build mode so numerical comparisons remain meaningful.

Changes:

• Introduces a build-time precompile workload that exercises Riccati/forward solve paths and resets run-filled caches before image serialization.
• Adds reset_caches! APIs for Vacuum/KineticForces module caches and tests that those caches are cleared.
• Updates the regression harness to control the precompile_workload preference per-case, and to persist/report build mode in fingerprints and the DB schema.

File	Description
test/​runtests_vacuum.jl	Switches to explicit keyword calls and adds a test ensuring caches emptied by reset_caches!.
src/​Vacuum/​PnQuadCache.jl	Adds Vacuum.reset_caches!() for module-level caches.
src/​KineticForces/​BounceAveraging.jl	Adds KineticForces.reset_caches!() and minor formatting/docstring fixes.
src/​Precompile.jl	Adds PrecompileTools workload that runs a small Solovev case during package image build.
src/​GeneralizedPerturbedEquilibrium.jl	Includes the new Precompile.jl file into the module.
regression-harness/​src/​types.jl	Adds precompile_workload::Bool to case specs and documents build-mode meaning.
regression-harness/​src/​config.jl	Adds case inheritance ([case] base) and parses precompile_workload.
regression-harness/​cases/​diiid_n1_riccati_precompiled.toml	New case that enables the workload to track the precompiled build.
regression-harness/​src/​runner.jl	Applies per-case workload preference via a stacked env; records/validates build mode.
regression-harness/​src/​env.jl	Extends environment fingerprint with build_mode and updates formatting/docs.
regression-harness/​src/​reporter.jl	Flags build-mode mismatches in environment comparisons.
regression-harness/​src/​database.jl	Adds build_mode column and persists it in run records.
docs/​src/​set_up.md	Documents how to disable the workload via Preferences/LocalPreferences.toml.
docs/​development/​regression-harness.md	Documents harness build-mode behavior and case inheritance semantics.
docs/​development/​naming.md	Adds Precompile to the naming table.
Project.toml	Adds PrecompileTools (and Logging) deps/compat entries.
ci/​manifests/​*	Updates CI manifest project hashes/checksum after Project.toml changes.
.gitignore	Ignores LocalPreferences.toml.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[adrianaghiozzi]

Reviewed with Claude Code, going through each piece independently rather than just reading the PR description. Summary below; happy to expand on any item.

What I checked and verified directly (not just read)

• reset_caches! completeness. Grepped the whole src/ tree for every module-level mutable const (Dict/Ref/etc.), not just the three this PR touches. Found 6 total: the 3 genuine run-filled caches (Vacuum._PN_CACHE, Vacuum.SINGULAR_QUAD_CACHE, KineticForces._QUAD_WEIGHTS) are now correctly emptied by the two new reset_caches! functions; the other 3 (_COIL_KEY_DEFAULTS, ANALYTIC_EQ, ITPA_THRESHOLD_SCALINGS) are static lookup tables — confirmed every reference to them anywhere in src/ is a read, never a mutation, so they genuinely don't need resetting. This is a complete fix, not just "the two bugs they happened to find."
• The load-path preference-override mechanism (with_workload_preference in runner.jl), which the whole build-mode-pinning feature depends on. Reproduced it directly: planted a LocalPreferences.toml in the worktree set to false (simulating a developer's own setting), then confirmed a synthetic environment prepended to JULIA_LOAD_PATH with precompile_workload = true correctly overrides it (Base.get_preferences resolves to true). Reversed the experiment — appending instead of prepending the same synthetic env — and confirmed the developer's false wins instead, proving it's genuinely the load-path order doing the work, not an artifact of the test. The "outranks a developer's LocalPreferences.toml" claim is correct and now empirically tested, which matters since every regression-database row's build-mode label depends on it being right.
• The CI manifest pin. Confirmed PrecompileTools's manifest stanza (git-tree-sha1) is byte-identical between develop and this branch — it was already a transitive dependency, just not a direct one, so promoting it changes nothing resolved. Logging is a stdlib, needs no stanza. The one-line project_hash diff in each pinned manifest is exactly what you'd expect from a Project.toml deps-list change with zero actual version movement.
• test/runtests_vacuum.jl's 115/97-line diff, which looks unrelated to a precompile PR at first glance. It isn't: the only substantive addition is one new @testset ("reset_caches! empties the run-filled caches") that fills all three caches, asserts they're populated, resets, and asserts empty — exactly the right regression guard against a future rename silently breaking the reset. Everything else in the diff is JuliaFormatter normalizing this file's pre-existing kwarg-call style to the repo's semicolon convention, triggered as a side effect of touching the file — expected per this repo's own documented formatter-churn policy, not something to push back on.
• Ran the full kinetic/vacuum reasoning through by hand; did not find anything in Precompile.jl itself, the CaseSpec/case-inheritance mechanism (resolve_case_base, one level deep, chained bases correctly rejected), or the _warn_build_mode safety net that looks wrong.

Independently reproduced the Apple-Silicon AOT-vs-JIT claim — and it's larger than "about 1%"

This machine is arm64, so I built the branch both ways (AOT via the real with_workload_preference load-path trick, JIT with the workload off) and ran the actual DIIID-like_riccati_deltaprime_example through both, same commit, same machine. Speedup matched the PR's feynman numbers proportionally (here: 35.7s AOT vs 169.3s JIT, ~4.7x; PR: 53.1s vs 242.7s, ~4.6x).

To rule out generic multithreading nondeterminism before attributing anything to build mode, I ran the JIT build twice as a control: bit-for-bit identical on every quantity I checked, including Delta_prime_matrix's diagonal and eigenmode_energies' stored order (0.0 diff both). So this build is fully deterministic on its own, and the AOT-vs-JIT divergence below is a genuine build-mode effect, not run-to-run noise.

The divergence itself is real but non-uniform, and bigger in the worst case than "about 1%" suggests:

• 3 of 5 singular surfaces: real parts of the Δ′ diagonal agree to ~0.1–1%, roughly matching the PR's framing.
• The surface closest to marginal (ψ≈0.968): -2201.14+276.67i (AOT) vs -2197.22-1035.19i (JIT) — real parts close, but the imaginary part flips sign and changes magnitude by ~4x.
• eigenmode_energies' stored order also differs between AOT and JIT (confirmed real via the same JIT-vs-JIT control, which reproduced the order exactly) — so build mode can reshuffle which eigenvalue lands at which index, not just perturb values. Anyone consuming that array positionally across build modes would see what looks like total divergence rather than a reordering of an otherwise mostly-consistent set.

This is consistent with, not contradicting, the PR's physical explanation (Δ′ BVP amplifying near-marginal conditioning) — if anything it strengthens the case for pinning the harness to one build mode. Suggested amendment, not a blocker: the release note should say the divergence can include sign flips and eigenvalue reordering at near-marginal surfaces on Apple Silicon, not just "about 1%," so nobody reading it later underestimates how different two builds can look.

One non-blocking design question

The workload gets expensive specifically for people in a Revise-based edit/restart loop (every src/ change that forces a Julia restart re-pays the ~160s Solovev rebuild). The PR's answer is a manual opt-out preference, which is simple and predictable. Worth asking whether a Revise-presence check (auto-skip the workload when Revise is loaded) was considered instead of, or in addition to, the manual toggle — not requesting a change, just curious about the tradeoff that was weighed.

Overall

No correctness issues found anywhere in the diff after going through it piece by piece and independently verifying the three parts most likely to hide a subtle bug (cache-reset completeness, the preference-override ordering, and the Apple-Silicon build-mode divergence itself). The regression report in the PR body is thorough and the one new case is well-designed to track the precompiled build without polluting comparability of the other 16. The one suggested change is tightening the release note's description of how large the AOT/JIT divergence can get — everything else here is ready as-is.

[adrianaghiozzi]

Claude review provided above, but just wanted to highlight the one part that I (human Adriana!) thought was interesting. You do add a slight cost with this method for someone who is working in a development loop and potentially invoking a fresh "using GeneralizedPerturbedEquilibrium" several times in a row. I see that to get around that you currently have the ability to have a toml file with instructions to skip the small bit of extra precompile work. Since the codebase is still in such active development and I imagine many people rely on the Revise package when developing, one optional update would be to automatically check whether someone has loaded Revise and skip the extra precompile if so. Up to you if that sounds worthwhile or not but otherwise this should be good to go!

[d-burg]

Thanks @adrianaghiozzi!

Regarding the release note: you’re right that “about 1 %” undersold it, I’ve amended the note. From claude: "on Apple Silicon the real parts of the Δ′ diagonal differ by about 1 %, but at a near-marginal surface the imaginary part changed sign and magnitude, and the stored order of eigenmode_energies differed. That matches what I saw in September on a near-marginal Solovev case, where the order change was two nearly degenerate modes swapping in the sort by Re(energy). I’ll re-measure the arm64 pair once OpenFUSIONToolkit/GPEC#491 is in, since it tightens the equilibrium tolerance this difference is amplified through."

Regarding skipping the workload under Revise – I looked into it and I don't think it can be detected at build time. Julia builds the package image in a separate process that loads only GPEC’s dependencies, so it cannot see whether the session that triggered the build has Revise loaded (checked on 1.11). The image is also cached per preference setting, not per session, so whichever session built first would set the build mode for the next one, which is exactly what the harness now records. So the opt out stays the precompile_workload preference in docs/src/set_up.md: this is one line, per each checkout, and gitignored. Within a single Revise session nothing is rebuilt, and the cost only appears on a restart after editing src/.