Regression - FEATURE! - ⚠️ Golden values: committed pins with provisional tolerances

.github/workflows/regression-harness-unit.yaml

@@ -0,0 +1,46 @@
+name: Regression Harness Unit Tests
+
+# Unit tests for the regression harness's own logic (golden values, case loading), on
+# synthetic data only. They load the harness project, not GPEC, so they run in minutes.
+on:
+  pull_request:
+    branches:
+      - main
+      - develop
+    paths:
+      - 'regression-harness/**'
+      - '.github/workflows/regression-harness-unit.yaml'
+  push:
+    branches:
+      - main
+      - develop
+    paths:
+      - 'regression-harness/**'
+      - '.github/workflows/regression-harness-unit.yaml'
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+jobs:
+  unit:
+    name: regression-harness unit tests
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v6
+
+      - name: Set up Julia
+        uses: julia-actions/setup-julia@v2
+        with:
+          version: '1.11'
+
+      - name: Instantiate the harness environment
+        run: julia --project=regression-harness -e 'using Pkg; Pkg.instantiate()'
+
+      - name: Run the harness unit tests
+        run: julia --project=regression-harness regression-harness/test/runtests.jl

docs/development/regression-harness.md

@@ -140,7 +140,23 @@ re-run once — those are exactly the entries whose provenance cannot be establi
 
 If the two refs in a comparison ran under different thread counts, the report flags it.
 
+## Golden values
+
+`--refs` comparisons say whether a number changed; golden values say whether it still matches a committed, reviewed reference. Each case may have a golden file at `regression-harness/golden/<case>.toml` holding every pinned quantity with its `rtol`, `atol`, tolerance class and the evidence behind the tolerance.
+
+```bash
+regress --check --cases slayer_dels_fitzpatrick                          # working tree vs the committed goldens
+regress --update-golden --cases slayer_dels_fitzpatrick --reason "..."   # re-pin from a fresh run
+```
+
+- **Classes** come only from the `class` key that every `[quantities.*]` block in a case file must declare, never from the golden file or the quantity's name; a missing or unknown class makes the case fail to load. `topological` (exact), `equilibrium_scalar`, `physics_converged` gate; `diagnostic` and `unconverged` are reported only.
+- **Tolerances** are either the provisional class default (`tolerance_basis = "class-default (provisional)"`) or `"measured"`, which requires a recorded `plateau_drift` or `platform_spread` no larger than `rtol`. No harness tool measures these yet, so every committed golden currently gates on provisional defaults, and `--check` labels those rows `provisional`.
+- **`atol`** must be 0 on a gating entry unless its golden value contains an exact zero (a zero scalar, or an array element that is zero throughout); the relative test is the gate everywhere else. Loading and saving both enforce this.
+- **Re-pinning** requires `--cases` and `--reason`, refuses a working tree with uncommitted tracked changes or with untracked files under `examples/`, `src/` or `regression-harness/cases/`, and prints an old → new line per moved quantity, flagging any move the old tolerance would have failed.
+- **A move beyond the old tolerance** makes `--update-golden` refuse the whole re-pin unless `--accept-exceeding` is given. When it is, the entry loses its recorded evidence and becomes provisional, with a tolerance no looser than before, and the golden file's `[meta.exceeded.<name>]` table records the previous version's value, the deviation and the old `rtol`/`atol`, so the reviewer sees each accepted move in the git diff.
+- A failing `--check` is fixed by explaining the physics or fixing the regression, never by widening `rtol` or `atol`. The unit tests for this logic run with `julia --project=regression-harness regression-harness/test/runtests.jl` and need no GPEC build.
+
 ## Exit status
 
 - `0` — every run completed (and, with `--fail-on-change`, nothing changed)
-- `1` — a run failed, or a quantity changed under `--fail-on-change`
+- `1` — a run failed, a quantity changed under `--fail-on-change`, or a `--check` golden comparison failed or gated nothing

docs/development/toml-conventions.md

@@ -16,7 +16,8 @@ modify them. **Do not invent a new convention** — match what the existing exam
    files** — `examples/DIIID-like_ideal_example/gpec.toml` is the canonical reference for the
    common sections. Exception: in `regression-harness/cases/*.toml` the repeated
    `[quantities.*]` schema keys (`h5path`, `type`, `extract`, `label`, `noise_threshold`,
-   `order`) are developer metadata and do **not** need inline comments — a case file needs
+   `order`, and the required golden tolerance `class`) are developer metadata and do **not**
+   need inline comments (a `class` that is not the obvious one for its quantity should say why) — a case file needs
    the header plus an informative block comment per quantity group instead.
 3. **Section comments only when informative.** No comment is required above a section. Keep a
    block comment only when it carries real information (e.g. the Solovev `[Wall]` note on why a

regression-harness/cases/diiid_error_field.toml

@@ -16,6 +16,7 @@ extract = "all_real"
 label = "EFC overlap per kAt"
 noise_threshold = 1e-10
 order = 10
+class = "physics_converged"
 
 [quantities.efc_torque_full]
 h5path = "ErrorFields/NTV/torque_full_per_kat2"
@@ -24,6 +25,7 @@ extract = "all_real"
 label = "EFC whole-field NTV torque per kAt² [N·m]"
 noise_threshold = 1e-8
 order = 11
+class = "physics_converged"
 
 [quantities.efc_torque_residual]
 h5path = "ErrorFields/NTV/torque_residual_per_kat2"
@@ -32,6 +34,7 @@ extract = "all_real"
 label = "EFC residual-field NTV torque per kAt² [N·m]"
 noise_threshold = 1e-8
 order = 12
+class = "physics_converged"
 
 # The torque-versus-rotation table: reference rotation, the adaptive shift grid and the residual torque on it.
 [quantities.efc_omega_reference]
@@ -41,6 +44,7 @@ extract = "all_real"
 label = "EFC reference rotation ω_ref [rad/s]"
 noise_threshold = 1e-6
 order = 13
+class = "physics_converged"
 
 [quantities.efc_rotation_shift]
 h5path = "ErrorFields/NTV/RotationScan/rotation_shift"
@@ -49,6 +53,7 @@ extract = "all_real"
 label = "EFC rotation-scan shifts [rad/s]"
 noise_threshold = 1e-6
 order = 14
+class = "physics_converged"
 
 [quantities.efc_torque_residual_scan]
 h5path = "ErrorFields/NTV/RotationScan/torque_residual"
@@ -57,6 +62,7 @@ extract = "all_real"
 label = "EFC residual torque against rotation [N·m/kAt²]"
 noise_threshold = 1e-8
 order = 15
+class = "physics_converged"
 
 # Anchors of the stages feeding the couplings.
 [quantities.dominant_singular_values]
@@ -66,6 +72,7 @@ extract = "all_real"
 label = "dominant-coupling singular values"
 noise_threshold = 1e-8
 order = 20
+class = "physics_converged"
 
 [quantities.ef_delta_nominal]
 h5path = "ErrorFields/CoilSensitivities/DominantMode/delta_nominal"
@@ -74,10 +81,12 @@ extract = "abs_first"
 label = "|delta_nominal| of coil set 1"
 noise_threshold = 1e-10
 order = 21
+class = "physics_converged"
 
 [quantities.runtime]
 h5path = ""
 type = "runtime"
 extract = "value"
 label = "Runtime (s)"
 order = 99
+class = "diagnostic"

regression-harness/cases/diiid_multi_n.toml

@@ -16,6 +16,7 @@ extract = "real_first"
 label = "total energy Re(et[1])"
 noise_threshold = 1e-10
 order = 10
+class = "physics_converged"
 
 [quantities.et_all]
 h5path = "ForceFreeStates/FreeBoundaryStability/eigenmode_energies"
@@ -24,6 +25,7 @@ extract = "all_complex"
 label = "total energy (all)"
 noise_threshold = 1e-10
 order = 20
+class = "physics_converged"
 
 # Integration
 [quantities.nstep]
@@ -33,6 +35,7 @@ extract = "value"
 label = "ODE steps (saved)"
 noise_threshold = 0
 order = 30
+class = "diagnostic"
 
 # Mode space
 [quantities.mpert]
@@ -42,6 +45,7 @@ extract = "value"
 label = "mpert"
 noise_threshold = 0
 order = 40
+class = "topological"
 
 [quantities.npert]
 h5path = "Info/npert"
@@ -50,6 +54,7 @@ extract = "value"
 label = "npert"
 noise_threshold = 0
 order = 41
+class = "topological"
 
 [quantities.mode_n]
 h5path = "PerturbedEquilibrium/mode_n"
@@ -58,6 +63,7 @@ extract = "checksum"
 label = "toroidal mode label of each mode index (checksum)"
 noise_threshold = 0
 order = 42
+class = "physics_converged"
 
 # Control-surface response, one block per n (b~ space)
 [quantities.permeability_diag]
@@ -67,6 +73,7 @@ extract = "diagonal_complex"
 label = "permeability diagonal P~_ii (all n blocks)"
 noise_threshold = 1e-8
 order = 50
+class = "physics_converged"
 
 [quantities.forcing_b_rootarea]
 h5path = "PerturbedEquilibrium/forcing_b_root_area"
@@ -75,6 +82,7 @@ extract = "all_complex"
 label = "forcing b~ (root-area-weighted), n = 1, 2"
 noise_threshold = 1e-10
 order = 51
+class = "physics_converged"
 
 [quantities.response_b_rootarea]
 h5path = "PerturbedEquilibrium/response_b_root_area"
@@ -83,6 +91,7 @@ extract = "all_complex"
 label = "response b~ (root-area-weighted), n = 1, 2"
 noise_threshold = 1e-8
 order = 52
+class = "physics_converged"
 
 # Perturbed equilibrium energies and the Σ_n torque
 [quantities.pe_plasma_energy]
@@ -92,6 +101,7 @@ extract = "value"
 label = "PE plasma energy (sum over n)"
 noise_threshold = 1e-10
 order = 60
+class = "physics_converged"
 
 [quantities.pe_vacuum_energy]
 h5path = "PerturbedEquilibrium/Energies/vacuum_energy"
@@ -100,6 +110,7 @@ extract = "value"
 label = "PE vacuum energy (sum over n)"
 noise_threshold = 1e-10
 order = 61
+class = "physics_converged"
 
 [quantities.pe_surface_energy]
 h5path = "PerturbedEquilibrium/Energies/surface_energy"
@@ -108,6 +119,7 @@ extract = "value"
 label = "PE surface energy (sum over n)"
 noise_threshold = 1e-10
 order = 62
+class = "physics_converged"
 
 [quantities.pe_toroidal_torque]
 h5path = "PerturbedEquilibrium/Energies/toroidal_torque"
@@ -116,6 +128,7 @@ extract = "value"
 label = "PE toroidal torque sum_n -2n Im(Phi_n' Lambda_nn^-1 Phi_n)/4"
 noise_threshold = 1e-12
 order = 63
+class = "physics_converged"
 
 # Singular coupling: one row per resonant (surface, n); integer-q surfaces appear once per n
 [quantities.rational_n]
@@ -125,6 +138,7 @@ extract = "all_real"
 label = "resonant n per row (rows sorted by psi across n)"
 noise_threshold = 0
 order = 70
+class = "physics_converged"
 
 [quantities.rational_m]
 h5path = "PerturbedEquilibrium/SingularCoupling/rational_m"
@@ -133,6 +147,7 @@ extract = "all_real"
 label = "resonant m per row"
 noise_threshold = 0
 order = 71
+class = "physics_converged"
 
 [quantities.resonant_area_weighted_field_all]
 h5path = "PerturbedEquilibrium/SingularCoupling/resonant_area_weighted_field"
@@ -141,6 +156,7 @@ extract = "all_complex"
 label = "resonant area-weighted field b^r per row"
 noise_threshold = 1e-8
 order = 72
+class = "physics_converged"
 
 [quantities.island_half_width]
 h5path = "PerturbedEquilibrium/SingularCoupling/island_half_width"
@@ -149,6 +165,7 @@ extract = "all_real"
 label = "island half-widths per row"
 noise_threshold = 1e-8
 order = 73
+class = "physics_converged"
 
 [quantities.chirikov_parameter]
 h5path = "PerturbedEquilibrium/SingularCoupling/chirikov_parameter"
@@ -157,6 +174,7 @@ extract = "all_real"
 label = "Chirikov parameter per row"
 noise_threshold = 1e-8
 order = 74
+class = "physics_converged"
 
 [quantities.delta_prime_all]
 h5path = "PerturbedEquilibrium/SingularCoupling/Delta_prime"
@@ -165,6 +183,7 @@ extract = "all_complex"
 label = "forcing-driven Delta' per row"
 noise_threshold = 1e-8
 order = 75
+class = "physics_converged"
 
 # Runtime
 [quantities.runtime]
@@ -174,3 +193,4 @@ extract = "value"
 label = "Runtime (s)"
 noise_threshold = 0.0
 order = 999
+class = "diagnostic"