Skip to content

ltm: de-hoisted array-valued RANK gets diagonal-only element edges, so rank-mediated cross-element loops are not enumerated #776

Description

@bpowers

Summary

The #771 fix (de-hoist: reducer_is_hoistable now requires reducer_collapses_to_scalar, invariant I5 of docs/design-plans/2026-06-11-ltm-shape-expressiveness.md, commit 0b14a9e3 on branch ltm-shape-phase1) routes array-valued RANK references onto the Direct conservative paths. That kills #771's defect (the cannot-compile scalar synthetic agg and its zeroed agg-routed loop scores), but the resulting element-edge treatment is spelling-dependent, and for the bare-arg spelling it under-represents RANK's true dataflow: RANK(pop, 1) read at element e is an order statistic that depends on every element of pop, not just pop[e].

Mechanism: two spellings, two arms

The de-hoisted classification sends the two legal spellings of the same RANK read down different conservative arms (established by the T1 adversarial review of commit 0b14a9e3):

  • Bare-arg RANK(pop, 1) -- classifies as a Direct/Bare site and takes the diagonal arm, emitting only pop[e] -> grow[e] element edges. Off-diagonal rank-mediated influence is missing, so cross-element loops through the rank reordering are not enumerated. This is the under-enumeration this issue tracks.
  • Wildcard-arg RANK(pop[*], 1) -- yields a Direct Wildcard site with in_reducer=false (the ltm: sliced reducer sub-expressions are not hoisted into aggregate nodes (stay on the wildcard-link-score path) #514 reclassification doesn't fire), which takes the conservative cross-product arm. Rank-mediated cross-element loops are enumerated there -- reviewer-demonstrated: pop[north] -> {grow[north], grow[south]}, loops enumerated, scores ~0 under constant ranks, zero warnings. This spelling is safe but noisy (conservative over-enumeration).

So the residual is spelling-dependent: bare-arg = diagonal under-enumeration (the filed concern); wildcard-arg = conservative over-enumeration (safe, noisy). Two spellings of the same dataflow get inconsistent loop sets.

What is lost (bare-arg spelling)

Cross-element feedback loops mediated by rank reordering are not enumerated. Concretely: if pop[b] overtaking pop[a] changes the rank assigned at a and therefore what flows to a's consumer, that b -> a influence has no off-diagonal element edge, so the loop through it never appears in the element graph and is never scored.

Current residual behavior (loud-ish, conservative)

  • Bare-arg: diagonal element edges only; same-element loops through RANK are enumerated and scored.
  • Wildcard-arg: full cross-product edges; rank-mediated cross-element loops are enumerated, with zero warnings.
  • For genuinely-constant rank orderings the rank-mediated link contributes ~0 score on either arm (RANK output is piecewise-constant; its delta-ratio stand-in is ~0 between reorderings -- cf. docs/tech-debt.md entry 27's RANK delta-ratio history).
  • Pinned by rank_frozen_subtree_link_score_scores_correctly (src/simlin-engine/tests/integration/ltm_array_agg.rs), whose doc comment describes this residual; also documented in the reducer_collapses_to_scalar rustdoc (src/simlin-engine/src/ltm_agg.rs).

So this is a documented under-enumeration (missing loops) on the bare-arg spelling, not silent wrong numbers on enumerated loops -- consistent with the epic's "no silent wrong numbers" invariant, but short of full expressiveness, and inconsistent across spellings.

Why it matters

Low. RANK-mediated feedback is rare (no corpus model exercises it), and RANK's partial is non-differentiable at reordering points, so even an enumerated rank-mediated loop would score via a conservative stand-in. But a model whose dynamics genuinely hinge on rank reordering inside a loop would have that loop invisible to dominance analysis if spelled bare-arg -- and the same model gains or loses loops by rewriting RANK(pop, 1) as RANK(pop[*], 1).

Components affected

Possible approaches

  1. Arrayed agg node for array-valued reducers (the shape the de-hoist deliberately deferred, and ltm: enumerate_agg_nodes hoists whole-extent array-valued RANK into a scalar agg node that cannot compile (agg-routed loop scores zeroed) #771's noted alternative): mint a synthetic agg node whose result_dims are the argument's dims, with conservative all-to-all source->agg element edges. Enumerates the cross-element loops AND unifies both spellings' treatment (one arm regardless of bare vs wildcard arg); scoring stays conservative (delta-ratio).
  2. Accept as documented limitation: RANK in feedback loops is rare, its partial is non-differentiable, and the current behavior is conservative and pinned. Keep this issue as the record and close as wontfix-by-design if the cost/benefit doesn't change -- though the spelling-dependent inconsistency weakens this option relative to the original filing.

Discovery context

Identified during T1 of the LTM shape-expressiveness Phase 1 work (branch ltm-shape-phase1, commit 0b14a9e3), as the deliberate residual of the #771 de-hoist. The spelling-dependence of the residual (bare-arg diagonal vs wildcard-arg cross-product) was established by the T1 adversarial review of the same commit. Part of LTM tracking epic #488 (cluster A). Related: #771 (the de-hoisted defect), #742 (closed capture sibling), docs/tech-debt.md entry 27 (RANK delta-ratio partial, by-design).

Activity

  1. added
    ltmLoops that Matter (LTM) analysis subsystem
    on Jun 11, 2026
  2. added a commit that references this issue on Jun 18, 2026
    871022c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ltmLoops that Matter (LTM) analysis subsystem

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions