engine: surface polarity confidence for LTM loop classification

docs/README.md

@@ -22,6 +22,7 @@
 - [tech-debt.md](tech-debt.md) -- Known technical debt items with measurement commands
 - [design-plans/](design-plans/) -- Design plans (architecture and phasing for major efforts)
   - [design-plans/2026-04-05-server-rewrite.md](design-plans/2026-04-05-server-rewrite.md) -- Local-first `simlin-serve` binary: filesystem-backed editor + in-process MCP
+  - [design-plans/2026-05-06-ltm-482-variable-level-loop-enumeration.md](design-plans/2026-05-06-ltm-482-variable-level-loop-enumeration.md) -- Tiered LTM loop enumeration: variable-level Johnson first, expand only the cross-element subgraph
 - [plans/](plans/README.md) -- Implementation plans (active and completed)
 - [test-plans/](test-plans/) -- Human verification plans for completed features
 - [implementation-plans/](implementation-plans/) -- Detailed phase-by-phase implementation plans

docs/design/ltm--loops-that-matter.md

@@ -90,8 +90,12 @@ and average absolute score for ranking.
 ### Exhaustive Mode (`ltm_discovery_mode = false`)
 
 1. `model_causal_edges` (salsa tracked, `db_analysis.rs`) builds the causal graph
-2. `model_loop_circuits` uses Johnson's algorithm to enumerate all
-   elementary circuits via DFS (`find_circuits_from` / `dfs_circuits`)
+2. `model_loop_circuits_tiered` runs Johnson's algorithm on the variable
+   graph and partitions cycles by `RefShape` composition:
+   - Pure-scalar / pure-A2A cycles emit a single `Loop` directly (fast path).
+   - Cross-element / mixed cycles drive an element-level Johnson run on
+     the slow-path subgraph (the element graph restricted to the
+     variables in those cycles).
 3. Module nodes appear as regular vertices in the parent graph; loops through
    modules are found naturally by the same algorithm
 4. After circuit detection, `enrich_with_module_stocks()` post-processes each
@@ -613,9 +617,27 @@ element's contribution.
 
 ### Loop Scores
 
-Element-level circuits from `model_element_loop_circuits` are grouped by their
-variable-level node sequence (strip subscripts, join) to distinguish A2A loops
-from mixed loops:
+Tiered loop enumeration (`model_loop_circuits_tiered`) classifies each
+variable-level cycle into one of three categories before deciding whether
+element-level enumeration is needed:
+
+- **PureScalar / PureSameElementA2A**: every traversed edge has only `Bare`
+  references and every variable in the cycle is either uniformly scalar or
+  uniformly arrayed over the same dimension list. The cycle materializes
+  directly into a single `Loop` (with `dimensions` populated for the A2A
+  case) without entering the element-level enumerator. This is the fast
+  path; cost is O(K) per cycle of size K rather than O(K * N) on N
+  elements.
+- **CrossElementOrMixed**: any edge has a `Wildcard`, `FixedIndex`, or
+  `DynamicIndex` reference, or the cycle mixes scalar and arrayed nodes,
+  or the arrayed nodes don't share a dimension list. These cycles drive
+  the slow-path subgraph: the element graph restricted to the variables
+  participating in such cycles. Johnson runs on this restricted subgraph,
+  and the results flow through the same per-circuit grouping logic the
+  legacy `build_element_level_loops` uses.
+
+Slow-path element-level circuits are grouped by their variable-level node
+sequence (strip subscripts, join) to distinguish A2A loops from mixed loops:
 
 **A2A loops**: All circuits in a group have the same variable-level structure
 and every node carries a subscript. These are collapsed into a single `Loop`
@@ -679,12 +701,16 @@ computational interval" strategy.
    does not build that composite pre-reduction: `ltm_enabled` runs exhaustive
    enumeration and `ltm_discovery_mode` runs `discover_loops()`. However,
    `model_ltm_variables` in `src/simlin-engine/src/db_ltm.rs` does automatically
-   switch from exhaustive to discovery when the largest SCC of the element-level
-   causal graph exceeds `MAX_LTM_SCC_NODES` (currently 50, defined in
-   `src/simlin-engine/src/ltm.rs`). Above this size, Johnson circuit enumeration
-   blows past reasonable memory and time budgets on its own; see
-   `docs/design-plans/2026-04-18-ltm-cap-lift-diagnosis.md` for the
-   measurements. (The legacy per-loop relative-score equation synthesis
+   switch from exhaustive to discovery in two phases. The early gate fires on
+   the variable-level causal graph's largest SCC (cheap Tarjan, no Johnson
+   yet). The late gate fires on the slow-path element-level subgraph's largest
+   SCC, computed inside `model_loop_circuits_tiered` after variable-level
+   cycles are classified. Both gates use `MAX_LTM_SCC_NODES` (currently 50,
+   defined in `src/simlin-engine/src/ltm.rs`). Above either size, Johnson
+   circuit enumeration blows past reasonable memory and time budgets on its
+   own; see `docs/design-plans/2026-04-18-ltm-cap-lift-diagnosis.md` and
+   `docs/design-plans/2026-05-06-ltm-482-variable-level-loop-enumeration.md`
+   for the measurements. (The legacy per-loop relative-score equation synthesis
    compounded this with an O(P^2) text blowup; moving the normalization
    post-simulation -- divergence 5 below -- removed that factor from
    augmentation cost.) Auto-flip emits a `CompilationDiagnostic` at

docs/tech-debt.md

@@ -224,13 +224,13 @@ Known debt items consolidated from CLAUDE.md files and codebase analysis. Each e
 
 ### 25. LTM Element-Level Loop Enumeration Runs at Wrong Granularity
 
-- **Component**: simlin-engine (src/simlin-engine/src/db_analysis.rs `model_element_loop_circuits`)
-- **Severity**: medium
-- **Description**: `model_element_loop_circuits` enumerates circuits on the element graph. For a pure-A2A model with 20 variables over a 100-element dimension, every variable-level circuit produces 100 element-level circuits that `build_element_level_loops` then collapses into one A2A loop. The circuit count is no longer artificially capped (the old `MAX_LTM_CIRCUITS = 100_000` gate was retired on 2026-04-18 once auto-flip made it vestigial), but the `MAX_LTM_SCC_NODES = 50` gate in `model_ltm_variables` still flips dense feedback subgraphs to discovery mode far sooner than variable-level enumeration would. Fix: enumerate at the variable level first, tag each loop's edges by same-element / cross-element / scalar, and only element-level-enumerate the cross-element subgraph. Cost becomes additive in N for pure-A2A loops instead of multiplicative, and SCC width stays below the auto-flip threshold.
-- **Note (2026-04-25):** The per-reference element graph refactor (`docs/design-plans/2026-04-25-ltm-per-ref-elem-graph.md`) eliminated the spurious NxN edge density that previously inflated element-graph SCCs on `FixedIndex`-using models. The Phase 5 measurement postscript records before/after numbers on cross_element_ltm, arrayed_population_ltm, hero_culture_ltm, and WRLD3-03: edge counts and circuit counts dropped on cross_element_ltm (20 -> 18 edges, 12 -> 8 circuits) without changing the largest element-SCC, and the other three fixtures were unchanged because they do not exercise the FixedIndex path. `MAX_LTM_SCC_NODES = 50` was retained because WRLD3 still trips the gate from variable-level cycle structure (population, capital, agriculture, persistent-pollution, non-renewable resources) rather than element-graph artifacts. The "enumerate at the variable level first, then expand only the cross-element subgraph" approach remains the right structural fix for pure-A2A models, but its pressure is materially lower now that FixedIndex no longer inflates SCC width.
+- **Component**: simlin-engine (src/simlin-engine/src/db_analysis.rs `model_loop_circuits_tiered`)
+- **Severity**: RESOLVED (pending #496 merge)
+- **Description**: (**Resolved** by the variable-level loop enumeration refactor; see `docs/design-plans/2026-05-06-ltm-482-variable-level-loop-enumeration.md`. Marked RESOLVED contingent on PR #496 landing on main.) `model_element_loop_circuits` enumerated circuits on the full element graph; for a pure-A2A model with V variables over an N-element dimension, every variable-level cycle inflated to N element-level circuits that `build_element_level_loops` collapsed back to one A2A loop. The fix is `model_loop_circuits_tiered`: it runs Johnson on the variable graph first, classifies each cycle by its `RefShape` composition via `model_edge_shapes`, and emits pure-scalar / pure-A2A cycles as Loops directly. Only cross-element / mixed cycles flow through element-level Johnson, and only on the subgraph induced by their nodes. The auto-flip gate now keys on the variable-level SCC (cheap pre-Johnson check) and the slow-path subgraph SCC (computed inside the tiered enumerator) rather than the full N-times-inflated element-graph SCC.
+- **Note (2026-04-25):** The per-reference element graph refactor (`docs/design-plans/2026-04-25-ltm-per-ref-elem-graph.md`) had earlier eliminated the spurious NxN edge density on `FixedIndex`-using models. After that refactor pure-A2A models already had small element-graph SCCs; the remaining win for tiered enumeration is the elimination of redundant per-element circuit enumeration (O(N) circuits per pure-A2A cycle that were collapsed back to one Loop). `MAX_LTM_SCC_NODES = 50` is retained: WRLD3 still trips it because its variable-level SCC is 166 (population, capital, agriculture, persistent-pollution, non-renewable resources) and the tiered enumerator gates on that.
 - **Tracked in**: #482 (LTM tracking epic: #488)
 - **Owner**: unassigned
-- **Last reviewed**: 2026-04-29
+- **Last reviewed**: 2026-05-06
 
 ### 26. LTM A2A Partial Equation Is Wrong When Target Mixes Same-Element and Cross-Element References
 
@@ -272,10 +272,10 @@ Known debt items consolidated from CLAUDE.md files and codebase analysis. Each e
 
 - **Component**: simlin-engine (src/simlin-engine/src/ltm.rs `LoopPolarity::from_runtime_scores`)
 - **Severity**: low
-- **Description**: The paper's polarity-confidence metric `|r - |b|| / (r + |b|)` classifies loops as Rux/Bux when mostly one polarity. Simlin collapses any sign change to Undetermined, losing information on mostly-reinforcing loops that briefly dip balancing (or vice versa). Fix: retain the ratio alongside the categorical polarity and surface it in `DetectedLoopsResult`.
+- **Description**: The paper's polarity-confidence metric `|r - |b|| / (r + |b|)` classifies loops as Rux/Bux when mostly one polarity. Simlin collapses any sign change to Undetermined, losing information on mostly-reinforcing loops that briefly dip balancing (or vice versa). Fix: retain the ratio alongside the categorical polarity and surface it in `DetectedLoopsResult`. Implementation in flight on branch `ltm/485-polarity-confidence` (PR #490, in draft); mark RESOLVED with the merge commit hash once the PR lands.
 - **Tracked in**: #485 (LTM tracking epic: #488)
 - **Owner**: unassigned
-- **Last reviewed**: 2026-04-29
+- **Last reviewed**: 2026-05-06
 
 ### 31. RK4 + LTM Combination Has No Hard-Error Guard
 

src/libsimlin/src/analysis.rs

@@ -121,9 +121,19 @@ pub unsafe extern "C" fn simlin_analyze_get_loops(
         } else {
             ptr::null_mut()
         };
+        // The C ABI exposes only the legacy three-way polarity surface
+        // (Reinforcing / Balancing / Undetermined), so MostlyReinforcing /
+        // MostlyBalancing fold into their dominant cousin here.  The
+        // polarity_confidence ratio is dropped at this boundary because
+        // the FFI struct has no field for it; native Rust callers that
+        // need confidence go through `engine::db::DetectedLoop` directly.
         let polarity = match loop_item.polarity {
-            engine::db::DetectedLoopPolarity::Reinforcing => SimlinLoopPolarity::Reinforcing,
-            engine::db::DetectedLoopPolarity::Balancing => SimlinLoopPolarity::Balancing,
+            engine::db::DetectedLoopPolarity::Reinforcing
+            | engine::db::DetectedLoopPolarity::MostlyReinforcing => {
+                SimlinLoopPolarity::Reinforcing
+            }
+            engine::db::DetectedLoopPolarity::Balancing
+            | engine::db::DetectedLoopPolarity::MostlyBalancing => SimlinLoopPolarity::Balancing,
             engine::db::DetectedLoopPolarity::Undetermined => SimlinLoopPolarity::Undetermined,
         };
         c_loops.push(SimlinLoop {

src/pysimlin/README.md

@@ -488,10 +488,11 @@ Loops are classified by polarity, which indicates how they affect the system:
 
 Loop IDs use the polarity as a prefix (e.g., "R1", "B2", "U3").
 
-When you run a simulation, pysimlin computes actual loop scores at each timestep. The polarity is classified based on these runtime values:
-- If loop scores are consistently positive throughout: Reinforcing
-- If loop scores are consistently negative throughout: Balancing
-- If loop scores change sign during simulation: Undetermined (occurs in nonlinear models where link effects depend on variable values)
+When you run a simulation, pysimlin computes actual loop scores at each timestep. The polarity is classified based on these runtime values using the Schoenberg & Eberlein (2020) polarity-confidence ratio `|r - |b|| / (r + |b|)`:
+- All loop scores positive: Reinforcing
+- All loop scores negative: Balancing
+- Mixed-sign with one polarity dominating at confidence >= 0.99: Mostly Reinforcing (Rux) or Mostly Balancing (Bux)
+- Mixed-sign with no clear dominance: Undetermined
 
 ```python
 from simlin import LoopPolarity
@@ -502,6 +503,8 @@ run = model.run()
 reinforcing = [l for l in run.loops if l.polarity == LoopPolarity.REINFORCING]
 balancing = [l for l in run.loops if l.polarity == LoopPolarity.BALANCING]
 undetermined = [l for l in run.loops if l.polarity == LoopPolarity.UNDETERMINED]
+mostly_r = [l for l in run.loops if l.polarity == LoopPolarity.MOSTLY_REINFORCING]
+mostly_b = [l for l in run.loops if l.polarity == LoopPolarity.MOSTLY_BALANCING]
 ```
 
 ### Loops That Matter (LTM)

src/pysimlin/simlin/analysis.py

@@ -24,36 +24,68 @@ def __str__(self) -> str:
             return "?"
 
 
+# Threshold above which a loop with mixed-sign runtime scores is classified
+# as MOSTLY_REINFORCING / MOSTLY_BALANCING (Rux / Bux) instead of UNDETERMINED.
+# Mirrors `POLARITY_CONFIDENCE_THRESHOLD` in `src/simlin-engine/src/ltm.rs`;
+# both implementations use the `>= 0.99` form of the cutoff described in
+# Schoenberg & Eberlein (2020) "Seamlessly Integrating LTM" / Schoenberg
+# (2020) thesis section 7.6 (the cited papers describe it as "above 0.99").
+POLARITY_CONFIDENCE_THRESHOLD: float = 0.99
+
+
 class LoopPolarity(IntEnum):
     """Polarity of a feedback loop.
 
     The polarity indicates how the loop affects the system:
     - REINFORCING (R): Loop amplifies changes (positive loop score)
     - BALANCING (B): Loop counteracts changes (negative loop score)
-    - UNDETERMINED (U): Loop polarity cannot be determined or changes during simulation
+    - MOSTLY_REINFORCING (Rux): Mixed-sign runtime scores but predominantly
+      reinforcing (confidence at or above POLARITY_CONFIDENCE_THRESHOLD)
+    - MOSTLY_BALANCING (Bux): Mixed-sign runtime scores but predominantly
+      balancing (confidence at or above POLARITY_CONFIDENCE_THRESHOLD)
+    - UNDETERMINED (U): Loop polarity cannot be determined; mixed-sign
+      runtime scores with neither polarity dominant
+
+    Integer values 0-2 mirror the C FFI; values 3 and 4 are Python-only
+    classifications produced by `from_runtime_scores` (the FFI does not
+    surface a polarity-confidence ratio yet, so structural loops never
+    arrive as MOSTLY_*).
     """
 
     REINFORCING = 0
     BALANCING = 1
     UNDETERMINED = 2
+    MOSTLY_REINFORCING = 3
+    MOSTLY_BALANCING = 4
 
     def __str__(self) -> str:
         if self == LoopPolarity.REINFORCING:
             return "R"
         elif self == LoopPolarity.BALANCING:
             return "B"
+        elif self == LoopPolarity.MOSTLY_REINFORCING:
+            return "Rux"
+        elif self == LoopPolarity.MOSTLY_BALANCING:
+            return "Bux"
         else:
             return "U"
 
     @classmethod
     def from_runtime_scores(cls, scores: "NDArray[np.float64]") -> Optional["LoopPolarity"]:
         """Classify loop polarity based on actual runtime loop score values.
 
-        This function examines the loop score values from a simulation run
-        and determines the appropriate polarity:
-        - All valid (non-NaN, non-zero) scores positive -> Reinforcing
-        - All valid scores negative -> Balancing
-        - Mix of positive and negative -> Undetermined
+        Mirrors `LoopPolarity::from_runtime_scores` in
+        `src/simlin-engine/src/ltm.rs`.  The polarity confidence
+        ``|r - |b|| / (r + |b|)`` (Schoenberg & Eberlein, 2020) drives
+        the classification:
+
+        - All valid (non-NaN, non-zero) scores positive -> REINFORCING
+        - All valid scores negative -> BALANCING
+        - Mixed signs and confidence at or above
+          POLARITY_CONFIDENCE_THRESHOLD -> MOSTLY_REINFORCING /
+          MOSTLY_BALANCING based on which side dominates the magnitude
+          tally
+        - Mixed signs and confidence below the threshold -> UNDETERMINED
         - No valid scores -> returns None
 
         Args:
@@ -68,17 +100,34 @@ def from_runtime_scores(cls, scores: "NDArray[np.float64]") -> Optional["LoopPol
         if len(valid_scores) == 0:
             return None
 
-        has_positive = np.any(valid_scores > 0)
-        has_negative = np.any(valid_scores < 0)
+        positive_sum = float(valid_scores[valid_scores > 0].sum())
+        negative_sum_abs = float(-valid_scores[valid_scores < 0].sum())
+
+        denom = positive_sum + negative_sum_abs
+        if denom <= 0.0:
+            # Mathematically unreachable: at least one filtered-in score
+            # is non-zero, so the magnitude sum is strictly positive.
+            # Guard anyway so a hostile array of subnormals can't surface
+            # a divide-by-zero NaN.
+            return None
+
+        confidence = abs(positive_sum - negative_sum_abs) / denom
+
+        has_positive = positive_sum > 0.0
+        has_negative = negative_sum_abs > 0.0
 
         if has_positive and not has_negative:
             return cls.REINFORCING
-        elif has_negative and not has_positive:
+        if has_negative and not has_positive:
             return cls.BALANCING
-        elif has_positive and has_negative:
-            return cls.UNDETERMINED
-        else:
-            return None  # All zeros after filtering
+        if confidence >= POLARITY_CONFIDENCE_THRESHOLD:
+            # Equal-magnitude r and |b| would yield confidence 0, which
+            # cannot pass the threshold check for any threshold > 0, so
+            # the dominant side is always strictly larger here.
+            if positive_sum > negative_sum_abs:
+                return cls.MOSTLY_REINFORCING
+            return cls.MOSTLY_BALANCING
+        return cls.UNDETERMINED
 
 
 @dataclass
@@ -131,7 +180,10 @@ class Loop:
     """Variables in this loop"""
 
     polarity: LoopPolarity
-    """Loop polarity: REINFORCING (R), BALANCING (B), or UNDETERMINED (U)"""
+    """Loop polarity: REINFORCING (R), BALANCING (B), MOSTLY_REINFORCING (Rux),
+    MOSTLY_BALANCING (Bux), or UNDETERMINED (U). MOSTLY_* values only arise
+    from `LoopPolarity.from_runtime_scores`; the C FFI surface coalesces them
+    onto REINFORCING/BALANCING because it has no polarity-confidence field."""
 
     behavior_time_series: NDArray[np.float64] | None = None
     """