fix(models): gemma4 sliding-window single-pass prefill degenerates to <pad> beyond the window #401

Problem / Background

For Gemma 4 sliding-window models (gemma4_unified and gemma4_vl), a single-pass prefill whose length exceeds the model's sliding window (1024 tokens for gemma-4-12b-it-4bit) degenerates. The rotating KV cache evicts keys that the earliest query rows still need, leaving an all-masked attention row. That row propagates NaN and the model emits <pad> output.

The failure is modality-independent. It reproduces with:

• a coherent text-only prompt of ~1700 tokens,
• the existing gemma4 VLM video path (gemma4_vl), and
• the new gemma4_unified video path added in feat: add video input support for Gemma 4 Unified (gemma4_unified) #400.

This was surfaced while implementing #164 (video input for gemma4_unified, merged as PR #400).

Concrete trigger from #164

The literal acceptance command from #164:

mlxcel generate -m mlx-community/gemma-4-12b-it-4bit --video clip.mp4 -p "Describe this video."

At the default 2.0 fps, a multi-second clip expands into ~20 frames (~1377 tokens), which exceeds the 1024-token sliding window, so the prefill degenerates and the output is <pad>.

The current user-side mitigation is to lower --fps (for example --fps 0.6) so the expanded prompt fits inside the window. This is documented in docs/supported-models.md as a known limitation.

Relationship to PR #400 (distinct bug, already fixed)

PR #400 already fixed a related but distinct bug. overlay_block_bidirectional in src/models/gemma4.rs used to panic ("Cannot reshape array of size N into shape (1, window)") when the prefill exceeded the window. It now aligns block ids to the trailing windowed keys, so the panic is gone.

This issue is about the remaining numerical degeneration (NaN -> <pad>) caused by rotating-cache eviction during a single-pass prefill. The panic fix did not address the eviction-of-needed-keys problem.

Proposed Solution

Make long-prompt prefill numerically valid for sliding-window Gemma 4 models. Likely direction:

• Chunked prefill for sliding-window Gemma 4: process the prompt in window-sized chunks so the rotating cache never drops keys that an in-flight query row still needs. Reuse the existing chunked-prefill machinery in the codebase where applicable rather than adding a parallel path.
• Or an equivalent approach that keeps a >window single-pass prefill from leaving all-masked attention rows.

Acceptance Criteria

• A prompt longer than the sliding window (text, image, or video) on mlx-community/gemma-4-12b-it-4bit produces coherent output instead of <pad>.
• The default-fps video command from feat: video input support for Gemma 4 Unified (gemma4_unified) #164 (mlxcel generate -m mlx-community/gemma-4-12b-it-4bit --video clip.mp4 -p \"Describe this video.\") works without manually lowering --fps.
• No regression for prefills at or below the window length.
• Update / remove the known-limitation note in docs/supported-models.md once the fix lands.

Technical Considerations

• Affects gemma4_unified and gemma4_vl; the root cause is in the sliding-window rotating KV cache path (src/models/gemma4.rs), so the fix should be shared across both.
• The bug is independent of input modality; a text-only repro is the simplest regression case and avoids needing a video fixture.
• Origin: follow-up to feat: video input support for Gemma 4 Unified (gemma4_unified) #164 / PR feat: add video input support for Gemma 4 Unified (gemma4_unified) #400.