Repository navigation
test: no macro-using metasd model is compared against Vensim output #561
Description
Activity
Broader follow-up: no standing numeric-fidelity oracle/policy for Vensim-imported models
Filing this as a comment on the natural umbrella issue (per a macro-epic retrospective) rather than a duplicate, since #561 already tracks the specific metasd simulation-tier prerequisite. The broader, recurring problem:
There is no standing policy or CI-available oracle for numeric-fidelity validation of Vensim-imported models. Because no licensed Vensim DSS is available to generate ground-truth
.vdf/output.tabseries, every Vensim-compat epic re-litigates this and ends up down-scoping acceptance criteria. Concretely, in the just-completed macro epic:- The metasd simulation-tier eligible-set is empty (this issue, test: no macro-using metasd model is compared against Vensim output #561).
- macros.AC6.3 was gated against formula-derived
output.tabarithmetic (hand-computed from the verbatim macro body + the engine's own STEP/RAMP/forward-Euler semantics) rather than DSS-validated series — see the C-LEARN-derived SAMPLE UNTIL fixture rationale in commit7cba4ad2.
This is a policy decision, not a code defect, and it is broader than #561 (which scopes the metasd corpus specifically). Options to decide between (enumerated here so a future session need not re-derive them):
- (a) A one-time licensed-Vensim-DSS run to bank a reference
.vdfcorpus that is then checked in and reused indefinitely (Theil_2011 is the cheapest first target per test: no macro-using metasd model is compared against Vensim output #561 — already green end-to-end, one-lineCORPUSedit once a reference exists). - (b) A CI-available alternative oracle (e.g. PySD) with documented fidelity caveats (integration-method and builtin-coverage differences vs. Vensim DSS must be stated).
- (c) A formal written policy that accepts formula-derived references with a stated precision bound, so future epics cite the policy instead of silently down-scoping ACs per-epic.
Recommend resolving with an explicit decision among (a)/(b)/(c) and recording it in the repo (e.g.
docs/dev/or the test-models README) so it is durable across epics.Context: surfaced during the Vensim macro epic retrospective (PR #564, branch
macros). Not requesting any change now — tracking the policy gap so it stops being re-litigated.- added 2 commits that reference this issue
on May 15, 2026 - changed the title
[-]engine: metasd macro corpus SIMULATION tier is empty -- enable it (Vensim DSS reference outputs starting with Theil_2011 + per-model non-macro blocker triage)[/-][+]test: no macro-using metasd model is compared against Vensim output[/+]on Oct 12, 2026
None of the 17 macro-using metasd models is checked against Vensim output.
The harness in
tests/integration/metasd_macros.rsonly checks that their macros expand; its simulate-and-compare tier lists no models.Each model either has no Vensim reference output checked in, or fails to compile for a reason unrelated to macros.
Example
test/metasd/theil-statistics/Theil_2011.mdlcompiles and simulates with no errors today (corpus_theil_multi_output_materializes_and_simulatesinsimulate.rs).It is left out only because no Vensim output for it is checked in.
Why it matters
These are real published models, and they are the only end-to-end numeric check of macro support against Vensim.
The underlying problem is wider than this corpus: there is no Vensim DSS run available to make reference outputs, so Vensim-compat work keeps falling back to hand-computed expected values.
Fix
Theil_2011.mdland check it in beside the model; changing itsCORPUSentry toSimTier::Eligibleturns on the comparison.docs/dev/, where reference outputs come from: a one-time Vensim DSS run banked in the repo, PySD (with its known differences from Vensim stated), or hand-computed values with a stated tolerance.The other models are blocked by engine gaps tracked separately:
RANDOM NORMAL(#560),DELAY Norder (#562),DELAY FIXED(#346),GET XLS DATA(#555), plus MDL-parse and dimension errors infree 6.mdl,groupon 1-3.mdl,scirev7/8.mdl,Wonderland3.mdl,IDch15d.mdlandInterpolatingArrays.mdlthat are not filed.Each one goes into the comparison as its blocker is fixed and an output is added.
Alternative: if no reference source is going to exist, delete the unused simulation tier from the harness rather than keep dead code waiting for it.
Pointers
src/simlin-engine/tests/integration/metasd_macros.rs:CORPUS,SimTier,metasd_simulation_tiertest/metasd/theil-statistics/