Repository navigation
C-LEARN: residual Ref.vdf numeric divergence after element-cycle-resolution (SAMPLE UNTIL / INIT-:NA: / numeric tail / NaN-vs-NA) #591
Description
Activity
Full residual investigation — measurement methodology + histogram progression
Recording the complete quantitative investigation behind this residual (so it is fully tracked), spanning the diagnostic, the two high-leverage fixes, and the post-comparator re-measure.
Methodology
A diagnostic harness runs C-LEARN through the incremental path (
open_vensim→compile_vm→Vm::new→run_to_end→into_results), parsestest/xmutil_test_models/Ref.vdf(VdfFile::parse(...).to_results_via_records()), and categorizes every matched(ident, step)cell — matched set =Ref.vdf.offsets ∩ results.offsets— into buckets. Scale: 873,982 matched cells = 3,482 matched series × 251 steps.Comparator (post-comparator measurement;
ensure_vdf_results, commit8775ae97)A cell matches if
|e − a| ≤ atol + rtol·max(|e|,|a|)withrtol = 0.01(the cross-simulator 1%) and a per-series absolute flooratol = K_ATOL · max_step|e[ident]|,K_ATOL = 1e-4. Plus: a matched-variable floormax(10, 10%·|vdf vars|); a NaN-skip guard (fail on an entirely-NaN core series, or a global NaN-skipped fraction > 10%); and the:NA:-sentinel reconciliation (Simlincrate::float::NA = -2^109≡ Vensim0; a Simlin:NA:vs a genuinely-nonzero Vensim value is a real mismatch, never silently passed). The engine keeps the:NA:sentinel — only the comparator interprets it.Histogram progression (three measurements)
Bucket Baseline (pre-GF-fix) Post-GF-fix (#589 / 61573545)Post-comparator (current, 8775ae97)match 58.8% 59.7% 83.7% (731,777) :NA:-sentinel / reconciled12.5% 12.5% 12.5% (108,934) near-zero (reconciled) 0.1% 24.0% 0.1% (1,129) genuine divergence 28.0% (1,314 idents) 3.27% (39 vars) 3.12% (27,251 cells / 31 vars) NA-mismatch (spurious :NA:)0.08% (14) 0.56% 0.08% (665 / 2 vars) other-NaN 0.48% (25) 0% 0.48% (4,226 / 9 vars) The near-zero bucket jumped post-GF-fix because correcting the per-element-GF mapping moved many wrong-value cells to ≈0-where-Vensim-is-≈0; the near-zero-robust comparator then folds those into
match(hence near-zero drops back to 0.1% whilematchrises to 83.7%).The two high-leverage fixes (28% → 3.12% genuine divergence)
- Per-element-GF element→dimension-index mapping (engine: per-element graphical functions are mapped by Arrayed-Vec (sorted) position instead of element-name -> dimension index #589, commit
61573545): per-element graphical-function tables were laid out byEquation::ArrayedVec position (alphabetically sorted by the MDL importer) instead of by element-name → declared dimension index, so for C-LEARN's non-sortedCOPdimension every arrayed GF fed the wrong element's table. Fixing it collapsed thepopulation → gdp → per-capita/emissions/HFC/CO2eqcascade (1,314 → 39 diverging vars). :NA:-aware + near-zero-robust comparator (8775ae97): mechanically reconciled the 12.5%:NA:-sentinel bucket (Simlin-2^109≡ Vensim0) and the near-zero bucket, with no numeric change.
Current genuine residual (~3.7%), banked + tracked
- Dominant
0+0data/graph-lookup-import cluster → engine: C-LEARN data/graph-lookup variables import as 0+0 (zeroed) instead of their lookup/data values #590 (the bulk of the 31 genuine-divergence vars / 27,251 cells; rel = 1.0, whole vars at 0 vs real Vensim values). - SAMPLE UNTIL / INIT-
:NA:/ target-policy cluster, the ~25%im_3_emissionsnumeric tail, and the other-NaN (NaN-vs-:NA:) cluster → this issue (breakdown in the issue body above).
The exact exhaustive list of excluded variables will live in
simulates_clearn'sEXPECTED_VDF_RESIDUALconstant (the Phase 7 un-stub commit). Lineage: element-cycle-resolution AC8.1; depends on the:NA:sentinel fix #586 and the GF-mapping fix #589.- Per-element-GF element→dimension-index mapping (engine: per-element graphical functions are mapped by Arrayed-Vec (sorted) position instead of element-name -> dimension index #589, commit
Exact
EXPECTED_VDF_RESIDUALmembership (measured onclearn-hero-model, un-stubbedsimulates_clearn)The exact exhaustive exclusion set is now baked into
EXPECTED_VDF_RESIDUALinsrc/simlin-engine/tests/simulate.rs, enumerated by running C-LEARN through the exactensure_vdf_resultsper-cell comparator logic (classify_vdf_ident) and collecting every base with >=1 failing cell. Through the hardened (:NA:-aware + near-zero-robust) comparator the residual is 32 failing bases over 154 failing idents (3298 of 3482 matched idents reconcile, ~95.6%).This issue's clusters cover 15 of those 32 bases. Two were discovered by the exhaustive measurement beyond the original representative lists; recording them here so every excluded base is attributable:
- Cluster 1 (NA-arithmetic):
historical_gdp(the bare twin of the engine: C-LEARN data/graph-lookup variables import as 0+0 (zeroed) instead of their lookup/data values #590historical_gdp_lookup) belongs HERE, not engine: C-LEARN data/graph-lookup variables import as 0+0 (zeroed) instead of their lookup/data values #590. Its non-:NA:cells match Vensim exactly; only its-2*NAcells diverge -- VDF-1.298e33(-2*NA) vs Simlin's bare:NA:sentinel-6.49e32(-2^109). Same NA-arithmetic pattern aslast_set_target_yearet al. - Cluster 2 (numeric tail):
diffusion_flux-- the ocean-diffusion physical variable ([sensitivity, layer]) feeding the SLR sub-model. It diverges on small magnitudes (VDF 0 vs sim ~1e-8 where the per-series abs floor doesn't cover, and ~0.5 rel on ~1e-5 cells). Same meaningful-but-small numeric-tail character as theim_3_*chain.
Cluster 3 (other-NaN, e.g.
slr_inches_from_2000) needs NO exclusion entry: the hardened comparator NaN-skips cells where either side is IEEE NaN (Vensim's early-step NaN vs Simlin's:NA:), and the late finite steps match within 1%, so no cluster-3 base reaches the failure set. The representation gap is still tracked here; it is just not gate-excluded.Cluster-1+2 bases excluded (this issue):
last_set_target_year,last_active_target_year,time_from_target_to_ultimate_target,target_emissions_for_rate,ultimate_target_value_from_rate,depth_at_bottom,emissions_with_stopped_growth,historical_gdp,im_3_emissions,im_3_emissions_vs_rs,im_3_ff_co2,relative_emissions_to_equity,relative_emissions_to_equity_target,co2eq_gap_closing_percentage,diffusion_flux.- Cluster 1 (NA-arithmetic):
Final per-cluster disposition after C-LEARN-residual Phases 1-3 + the runlist-determinism fix
The bulk of the residual this umbrella tracked is now reconciled. After Phases 1-3 of the C-LEARN-residual plan plus the init-runlist-determinism fix (
e24b0080),simulates_clearnpasses the hard 1% gate on 3360 matched variables (vs a 348 floor, ~9.7x) and the live failing set shrank from 32 base variables to 21, all fully attributed under a five-category taxonomy intests/simulate.rs::EXPECTED_VDF_RESIDUAL, pinned exact byclearn_residual_exactness(re-run deterministically, twice).Cluster 1 -- SAMPLE UNTIL / INIT-of-
:NA:/ target-policyMostly reconciled.
last_active_target_year,time_from_target_to_ultimate_target,target_emissions_for_rate,ultimate_target_value_from_rate,depth_at_bottom, andemissions_with_stopped_growthall dropped out of the failure set (passthrough-INIT handling in Phase 3, plus the init-runlist-determinism fixe24b0080that stoppeddepth_at_bottom's per-layer init values from permuting).Remaining (boundary category, genuine documented remainder):
last_set_target_year-- every cell is-2*NA: Vensim computes-1.298e33, Simlin keeps the bare:NA:sentinel-6.49e32(crate::float::NA).historical_gdp-- non-:NA:cells match exactly (e.g.[oecd_us]6.667e4); only its-2*NAcells diverge, same as above.
:NA:-arithmetic is confirmed correct and explicitly out of scope (seefloat::NA): the sentinel is finite by design so theIF THEN ELSE(x = :NA:, ...)existence idiom works. These two stay carved out asboundarywith a sourced reason.Cluster 2 -- numeric tail
The substantive
im_3_*/relative_emissions_*chain (rel ~0.247 on meaningful values) is reconciled --im_3_emissions,im_3_emissions_vs_rs,im_3_ff_co2,relative_emissions_to_equity,relative_emissions_to_equity_targetall dropped out of the failure set after the Phase 1-3 engine fixes. There is no remaining engine-genuine numeric divergence.Remaining (benign-near-zero category):
co2eq_gap_closing_percentage-- a ratio of near-equal small values, peak ~1.1e-2 (vdf-6.8e-4vs sim6.9e-4); divergence only on near-zero magnitudes.diffusion_flux-- ~2% on a small early transient (~7% of cells, vdf0vs sim ~5e-10), matching late; cross-simulator near-zero noise.
Cluster 3 -- NaN-vs-
:NA:representation gapNot in the failure set, by construction. Where Vensim writes literal IEEE NaN and Simlin writes the finite
:NA:sentinel (e.g.slr_inches_from_2000), the comparator NaN-skips those cells and the finite tail matches within tolerance, so the series is neither all-NaN nor has failing cells. This is pinned independently of C-LEARN by the new unit testclassify_vdf_ident_nan_vs_na_skips_without_failing. The representation gap itself remains tracked here (not excluded).The dominant former cluster (the lookup/data
0+0import) moved to a reader-decode artifactThe 17 graphical-function / lookup-only bases are now classified as VDF-decode-artifact (engine proven correct; the
Ref.vdfreader mis-decodes the standalone descriptor columns). The engine0+0bug is fixed (#590, Phase 1); the scalar reader half is fixed (d69754bc); the arrayed half + 4 scalar no-distinct-column cases are tracked in #597.Net
This umbrella's genuine engine remainder is now just the
:NA:-arithmetic boundary (2 bases, out of scope) and the benign-near-zero noise (2 bases). Everything else either reconciled (Phases 1-3 /e24b0080) or moved to the reference-reader-decode bucket (#597). Suggest narrowing this issue's scope to the:NA:-arithmetic representation note (or closing if that is deemed working-as-intended), with #597 carrying the reader work.- added 5 commits that reference this issue
on May 20, 2026 - added 2 commits that reference this issue
on Sep 4, 2026
Summary
Umbrella tracking item for the smaller residual clusters of C-LEARN's numeric divergence from the genuine-Vensim reference output (
test/xmutil_test_models/Ref.vdf) that remain after the element-cycle-resolution work. C-LEARN now compiles, runs to FINAL TIME, and matchesRef.vdfwithin 1% on 96.3% of matched cells (after the per-element-GF mapping fix #589 / commit61573545and a:NA:-aware + near-zero-robust comparator inensure_vdf_results). The decision was made to bank the 96.3% and track the residual: every variable below is excluded from thesimulates_clearngate via the documentedEXPECTED_VDF_RESIDUALexclusion constant, so the test passes on the reconciled set.The dominant residual cluster (whole variables importing as a literal
0+0stub) is tracked in a separate issue. This umbrella collects everything else.Discovered on branch
clearn-hero-model, HEAD8775ae97; element-cycle-resolution AC8.1.Residual clusters
1. SAMPLE UNTIL / INIT-of-:NA: / target-policy (COP dimension, often
cop_developing_b)A
:NA:-arithmetic edge and/orSAMPLE UNTILsemantics. Vensim performs NA-arithmetic yielding e.g.-2*NAwhere Simlin yields a bare:NA:sentinel or0:last_set_target_year-- Vensim =-1.298e33(i.e.-2*NA) vs Simlin =0; equation isINIT(VECTOR SELECT(...))last_active_target_year--INITwith a-6.49e32(:NA:) guardtime_from_target_to_ultimate_target-- rel ~ 0.96target_emissions_for_rate/ultimate_target_value_from_rate--SAMPLE UNTILdepth_at_bottom--INITrecurrenceemissions_with_stopped_growthThis subsumes the 2-variable "NA-mismatch" bucket (spurious
:NA:, e.g.last_set_target_year).2. Genuine ~25% numeric tail (meaningful-value divergence, not near-zero)
A connected chain that diverges by rel ~ 0.247 on substantial values -- needs a numeric root-cause, not comparator tuning:
im_3_emissions/im_3_ff_co2/im_3_emissions_vs_rs/relative_emissions_to_equity/relative_emissions_to_equity_target-- e.g.oecd_us@ step 187co2eq_gap_closing_percentage-- large rel but tiny absolute (~1.4e-3); a near-zero-ratio percentage, mostly comparator-ish rather than a true divergence.3. other-NaN (NaN-vs-:NA: representation mismatch)
~9 variables, led by
slr_inches_from_2000[deterministic](sea-level-rise), where Vensim writes literal IEEE NaN for all 251 steps while Simlin writes the:NA:sentinel (-6.49e32). A NaN-vs-:NA:representation mismatch on a deterministic-scenario series.Why it matters
:NA:-arithmetic (e.g.-2*NA),SAMPLE UNTILsemantics, and NaN-vs-:NA:representation are each places where Simlin's runtime diverges from genuine Vensim on real model logic.im_3_emissionschain is a meaningful-magnitude divergence (not near-zero noise) and is the most likely to indicate a substantive engine bug rather than a representation/comparator nuance.Components affected
src/simlin-engine/src/-- builtins /SAMPLE UNTIL,:NA:-arithmetic, VDF NaN handlingtest/xmutil_test_models/C-LEARN v77 for Vensim.mdl,test/xmutil_test_models/Ref.vdfsrc/simlin-engine/tests/simulate.rs--simulates_clearn; the exact exhaustive list of excluded vars lives in itsEXPECTED_VDF_RESIDUALexclusion constant.Discovery context
Identified during the C-LEARN hero-model element-cycle-resolution work (branch
clearn-hero-model, HEAD8775ae97), element-cycle-resolution AC8.1. All variables above are excluded from thesimulates_clearngate and tracked here.Relationship to existing issues (all DISTINCT)
:NA:mishandled as IEEE NaN -> finite sentinel-2^109) -- FIXED; established the runtime:NA:sentinel. The clusters here are the remaining numeric residual after that fix::NA:-arithmetic results (-2*NA),SAMPLE UNTIL, and the VDF NaN-vs-:NA:representation gap. engine: Vensim :NA: was mishandled as IEEE NaN (should be the finite sentinel -2^109) #586's own "Remaining follow-on" (test comparator reconciling sentinel-vs-VDF-0.0) is the comparator now inensure_vdf_results; this issue is the genuine divergence that comparator does not paper over.0+0-stub lookup/data import cluster is tracked in a separate issue.