Skip to content

ltm: residual ~30k synthesized PREVIOUS helper-aux vars on C-LEARN threaten the VM 65k-slot ceiling #654

Description

@bpowers

Summary

LTM instrumentation on C-LEARN v77 (test/xmutil_test_models/C-LEARN v77 for Vensim.mdl) with LTM discovery mode still synthesizes ~30k helper-aux variables, down from ~145k after the work in commits 7e6048e7, 82d20304, and 51392894 on main. C-LEARN now fits under the bytecode VM's 65,536-slot limit (56,891 slots) and works end-to-end, but the remaining helpers cost compile time, memory, and runtime, and a moderately larger model would hit the slot limit again.

This is the residual scalability/cost half of the PREVIOUS-helper-synthesis problem: it is distinct from #587 (a subscripted-PREVIOUS LTM fragment silently dropping its score to 0 -- correctness/accuracy) and from #647 (the discovery algorithm being intractable at element-expanded scale). This issue is specifically about the volume of helper-aux variables the augmentation emits, the slot/compile-time/memory pressure that creates, and how close it sits to the hard VM slot ceiling.

Remaining helper sources (measured on C-LEARN v77, LTM discovery mode)

  1. ~9.7k PREVIOUS calls with bare-element subscripts that cannot be qualified. Two sub-cases: (a) elements declared by multiple dimensions at different positions (genuinely ambiguous), and (b) subscripted references of variables whose dims are not in scope at the wrap site. Root cause: wrap_index_non_matching_in_previous / qualify_element_index in src/simlin-engine/src/ltm_augment.rs (~:737-:766) only have the project DimensionsContext, not the subscripted variable's declared dims, so they cannot position-match the element and fall back to synthesizing a helper.
  2. ~2k module-output refs (PREVIOUS(module·output)). These have fixed slots too and could compile to a direct LoadPrev with submodel-offset resolution instead of a synthesized helper.
  3. ~2k index-wrapped refs (x[PREVIOUS(elem)]) from paths where qualification falls back due to ambiguous element membership -- same root cause as (1).
  4. The per-target-element generators (~9.4k helpers) plus remaining lowercase previous(...) calls embedded from original equations.

Why it matters

  • Headroom against the hard slot ceiling. C-LEARN now compiles at 56,891 slots vs. the 65,536-slot VM limit. The ~30k residual helpers are a large fraction of that budget; a moderately larger arrayed model that auto-flips to discovery would blow the ceiling and fail to compile. This is a structural scalability limit in the LTM-augmentation path, not just a polish item.
  • Compile time, memory, runtime. Every helper is an extra synthetic variable to compile, lay out, and evaluate each step. Cutting the residual helpers directly reduces LTM compile time and per-step cost on large models.
  • Maintainability. Several of these helper sources exist only because qualification lacks the per-variable declared dims at the wrap site -- threading that context through removes whole categories of fallback synthesis rather than papering over them.

Components affected

  • src/simlin-engine/src/ltm_augment.rs -- wrap_index_non_matching_in_previous, qualify_element_index (the wrap path that lacks the subscripted variable's declared dims and so falls back to helper synthesis).
  • src/simlin-engine/src/db/ltm/link_scores.rs -- the call sites that already thread per-variable declared dims into qualification (the pattern to extend to the wrap path).
  • src/simlin-engine/src/builtins_visitor.rs -- index_is_static (:322) and the static-subscript check (:767) that gate direct LoadPrev/INIT compilation vs. helper synthesis.

Possible directions

  • Thread per-variable declared dims into the wrap path so qualify_element_index can be position-matched everywhere -- the same approach already used at the link_scores.rs call sites. This removes the bulk of (1) and (3).
  • Extend the builtins-visitor static check (index_is_static) to accept unambiguous bare element names, so they compile to a direct static-subscript PREVIOUS instead of a helper.
  • Resolve module-output refs to static slot offsets (submodel-offset resolution -> direct LoadPrev) to eliminate (2).

How it was discovered

Follow-up work from the LTM compile-time/correctness work on 2026-05-31 (commits 7e6048e7, 82d20304, 51392894), profiling residual helper-aux synthesis on C-LEARN v77 with LTM discovery enabled after the model came under the slot limit.

Linkage

Activity

  1. added
    ltmLoops that Matter (LTM) analysis subsystem
    on Jun 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ltmLoops that Matter (LTM) analysis subsystem

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions