Scan full responses for repetition metrics

[Shi-Dong]

Summary

• scan the full response for repetition using overlapping 10,000-character windows with a 5,000-character stride
• preserve coverage of the exact final 10,000-character suffix when it is not stride-aligned
• add focused tests for middle-response repetition, boundary lengths, suffix coverage, and the existing strict compression threshold

Why

rollout/repetition_frac currently checks only the final 10,000 characters of each response. A long response can contain severe repetition earlier and end with non-repetitive text, causing the metric to miss the behavior. Scanning bounded overlapping windows makes the metric represent repetition anywhere in the response while retaining short-circuit evaluation once repetition is found.

Testing

• uv run --active --no-sync pytest -q --confcutdir=tests/fast/utils tests/fast/utils/test_metric_utils.py — 5 passed
• uv run --active --no-sync pre-commit run --files miles/utils/metric_utils.py tests/fast/utils/test_metric_utils.py — passed

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[Shi-Dong]

@claude review always

[claude]

Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.

[claude]

Looks good — the follow-up commit (e6ea865) directly addresses the unbounded-cost concern I raised on the previous version by capping the scan at REPETITION_MAX_WINDOWS=32 regardless of response length, keeping has_repetition cheap on the executor's synchronous event loop.

What was reviewed: the _repetition_windows/has_repetition windowing and stride-widening math, and the new test file (stride-aligned/unaligned window detection, boundary length, short-text short-circuit, strict threshold, bounded window count for 2M-char text, first/last coverage, unmocked end-to-end detection). Also checked configured --rollout-max-response-len values across the repo's run scripts (max ~32768 tokens) against the documented ~320k-char gap-free coverage limit — production response lengths stay well under that threshold, so the stride-widening gap for extremely long text is not practically reachable.

Extended reasoning...

This is a follow-up to a PR I already reviewed twice. My prior inline comment flagged that has_repetition's cost scaled with full response length (O(len/stride) zlib compressions) and ran synchronously inside the rollout executor's event loop with no bound. The author's latest commit (e6ea865) directly fixes this by introducing REPETITION_MAX_WINDOWS=32 and widening the stride so the total window count — and thus the compute cost — never exceeds a small constant regardless of text length, which is exactly one of the two fixes I suggested (cap total scanned length/window count).

Overview

The change is confined to miles/utils/metric_utils.py (a small windowing/compression-ratio helper) and its new test file. The core logic (_repetition_windows, has_repetition) is self-contained, has no I/O, no external state, and no security-sensitive surface — it only reads a string and computes a compression ratio.

Security risks

None. No injection, auth, or data-exposure surface; it's a pure text-processing metric used for observability.

Level of scrutiny

Moderate is warranted given this runs on a hot path (rollout executor) previously flagged for a performance concern, but the fix is narrow, well-tested (7 tests covering boundary conditions, mocked and unmocked paths, and the bounded-window invariant), and the remaining theoretical gap (windows can have unscanned gaps between them once stride > window size, i.e. for texts beyond ~320k chars) is explicitly documented in a code comment and not reachable given the repo's configured --rollout-max-response-len values (max observed ~32768 tokens, well under the 320k-char threshold).

Other factors

No third-party CHANGES_REQUESTED or unresolved objections are present in the timeline — the only reviewer was this bot's own prior pass, and its concern was directly addressed by the follow-up commit. Test coverage is solid for the new logic.