fix(optimization): trim_to_length must retain the newest positions, not the oldest (#13033)

[mrveiss]

Thinking Path

LayerKVCache.trim_to_length() did the opposite of what it exists for. Storage is linear and append-only — update() writes at [filled_len : filled_len + n] and get() returns k[:, :filled_len] — so index 0 is the first token of the sequence and filled_len - 1 is the most recent. The method only moved the pointer back:

if entry.filled_len > max_len:
    entry.filled_len = max_len

That retains [0, max_len) — the oldest window — and silently drops everything recent. A sliding window needs the newest positions, so every caller got a context anchored at the start of the sequence that never advances. The shape was right and the content was the wrong end, which is why it went unnoticed.

What Changed

trim_to_length now relocates the newest max_len positions down to offset 0 before moving filled_len, via a small _retain_tail helper.

The copy goes through an explicit .clone() of the source. Source and destination overlap whenever max_len > filled_len / 2, and copying a tensor onto an overlapping view of itself is undefined — that is a real case, not a hypothetical, and it has its own test.

The existing test was asserting the bug

test_trim_preserves_content_up_to_limit read:

cache.trim_to_length(6)
assert torch.allclose(r[0], k[:, :6, :, :])   # the OLDEST six

Implementation and test were wrong in the same direction, so the suite confirmed the defect rather than catching it. It is now test_trim_retains_the_newest_positions_not_the_oldest, asserting k[:, 6:] and explicitly asserting the head does not match, so a symmetric fixture cannot pass by luck.

Two further tests cover what the fix could plausibly get wrong:

• test_trim_then_update_appends_after_the_retained_window — the write pointer must follow the relocated window. Moving data without the pointer (or the reverse) would either overwrite live positions or leave a hole of stale tokens.
• test_trim_with_overlapping_source_and_destination — seq=10, max_len=7, so [3:10] and [0:7] overlap across four positions.

Verification

The tests fail against the old implementation. Source reverted, tests kept:

FAILED ...::test_trim_retains_the_newest_positions_not_the_oldest
FAILED ...::test_trim_then_update_appends_after_the_retained_window
FAILED ...::test_trim_with_overlapping_source_and_destination
3 failed, 4 passed

With the fix restored:

53 passed          # kv_cache_test.py
341 passed         # autobot-backend/llm_shared/optimization/

These tests are @requires_torch and skip without it. Under the default interpreter they report 28 passed, 25 skipped and every trim test is in the skipped set — running them there proves nothing. The runs above use an interpreter with torch 2.13.0+cpu so the assertions actually execute.

Scope

This is Step 1 of #13033 (correctness). Step 2 — replacing linear storage with a fixed-capacity circular ring so trimming is a pointer move with no copy — is a separate performance change and is not in this PR, so the issue stays open. The _write_to_entry overflow behaviour noted in the issue (raises rather than evicting) is also untouched.

Refs #13033

Model Used

Claude Opus 5

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No hardcoded values detected that have SSOT config equivalents!