PN59 streaming-GDN orchestrator never engages on chunked-prefill — Cliff 2b still fires on 1× RTX 3090

[noonghunna]

PN59 streaming-GDN orchestrator never engages on chunked-prefill — Cliff 2b still fires on 1× RTX 3090

Summary

PN59's eligibility gate rejects single-seq calls when chunk_indices or chunk_offsets is non-None, falling back to _vanilla_path — which then OOMs at the exact chunk_o.py:161 o = torch.empty_like(v) site PN59 was advertised to eliminate. On a 24 GB single-card config, vLLM's chunked-prefill is mandatory to fit anything, so chunk_indices/chunk_offsets are always populated → PN59's protection envelope excludes the path that actually needs it.

Cliff 2b is unchanged on club-3090's long-text.yml config vs v7.69 baseline.

Repro

• Hardware: 1× RTX 3090, 24 GB VRAM, sm_86
• Pin: Genesis 7b9fd319 (v7.72.2), vLLM 0.20.2rc1.dev9+g01d4d1ad3 (allowlist entry patch_genesis_unified.py #2)
• Compose: noonghunna/club-3090 models/qwen3.6-27b/vllm/compose/docker-compose.long-text.yml (TP=1, qwen3.6-27b-AutoRound INT4, MTP K=3, KV turboquant_3bit_nc, max_model_len=180000, gpu-memory-utilization=0.93, max-num-batched-tokens=4128)
• Trigger: single-shot 60K-token prompt (Python webhook code filler)
• GENESIS_ENABLE_PN59_STREAMING_GDN=1 in env

HTTP 500 within 18s
torch.OutOfMemoryError: tried to allocate 50 MiB, 56 MiB free
File ".../fla/ops/chunk_o.py", line 161, in chunk_fwd_o
    o = torch.empty_like(v)

Root cause (PN59 IS firing, but bypasses to vanilla then OOMs)

PN59's text-patch is correctly applied to chunk.py at boot — verified via in-container head of chunk_gated_delta_rule_fwd. The streaming_gdn_driver module imports cleanly. GdnScratchPool.is_production_eligible() returns True. threshold_T = 4 × 64 × 4 = 1024; our T=4128 (per chunked-prefill chunk) and T=60000 (full prefill) both exceed the threshold.

After injecting an exception logger into PN59's silent except Exception: pass, the actual flow surfaced:

chunk.py:47               → return _genesis_pn59_streaming(...)        [dispatcher fires]
streaming_gdn_driver.py:118 → return _vanilla_path(...)                [eligibility REJECTED]
streaming_gdn_driver.py:195 → o = chunk_fwd_o(...)                     [internal vanilla path]
chunk_o.py:161             → o = torch.empty_like(v)                  [OOM]
                             OutOfMemoryError propagates → outer
                             chunk.py except: pass catches it
                             → falls through to on-disk vanilla
                             → OOMs again at chunk_o.py:161
                             → user-facing 500

Single-seq is true, production_eligible is true, T>threshold is true — the only remaining condition is has_no_chunk_metadata. v7.72.2's streaming_gdn_driver.py comment confirms this gate was tightened in the v7.72 audit:

# Audit P2 fix 2026-05-05 (genesis_deep_cross_audit, P2.4): tightened
# eligibility to also reject calls with non-trivial chunk metadata.
# The streaming path passes `cu_seqlens=None, chunk_indices=None,
# chunk_offsets=None` to inner kernels; if the caller supplied
# significant chunk_offsets/chunk_indices we'd silently drop that
# metadata and produce divergent results vs vanilla.

The audit fix is correct in principle (avoid silent divergence) but on serving-time vLLM, chunked-prefill always populates these fields when max-num-batched-tokens < prompt_len, which on a 24 GB card is mandatory to load 60K-token prompts at all.

Net: PN59 protects a path that doesn't actually run on Ampere consumer + long context.

Diagnosis surface

• PN59 dispatcher is reached (proven by injected logger trapping the OOM raised from _vanilla_path)
• has_no_chunk_metadata is False during chunked-prefill — most likely the failing eligibility condition (single-seq + production_eligible + T>threshold are all confirmed True via direct probe)
• except Exception: pass masks the OOM so operators can't tell PN59 silently bypassed; we only saw it after the on-disk vanilla also OOM'd

Reproducer assets

• Compose: models/qwen3.6-27b/vllm/compose/docker-compose.long-text.yml (in noonghunna/club-3090, branch v7.72.2-uplift)
• 60K probe script: paste below or /tmp/pn59_high_t_probe.py
• Boot env shows: [Genesis Dispatcher] APPLY PN59 — Streaming-GDN orchestrator (Variant D Phase 2)

# /tmp/pn59_high_t_probe.py — single-shot 60K-token prompt to long-text.yml
# (60K Python code filler + 100-token output budget)
import json, time, urllib.request, subprocess
ENDPOINT, MODEL = "http://localhost:8020", "qwen3.6-27b-autoround"
filler = ("def handle_webhook(p): ...\n" * 12000)[:240000]
req = urllib.request.Request(
    f"{ENDPOINT}/v1/chat/completions",
    data=json.dumps({"model": MODEL, "messages": [
        {"role": "system", "content": "Reply concisely."},
        {"role": "user", "content": "Find bugs:\n```python\n" + filler + "\n```"},
    ], "max_tokens": 200, "temperature": 0.4}).encode(),
    headers={"Content-Type": "application/json"}, method="POST",
)
with urllib.request.urlopen(req, timeout=600) as r:
    print(r.status, r.read()[:200])

Fix proposals (operator → upstream priority)

1. Make has_no_chunk_metadata rejection optional — gate it behind a separate env (GENESIS_PN59_STRICT_NO_METADATA=1, default off) so single-seq chunked-prefill takes the streaming path. The "silent metadata drop" risk the audit was guarding against is real but at minimum the operator should be able to opt-in to "stream anyway, accept the metadata-drop risk" since the alternative is a hard OOM.

2. Thread chunk_indices/chunk_offsets into _streaming_path — the structurally correct fix. Streaming would respect the chunk metadata across windows.

3. Surface the eligibility-bypass reason without GENESIS_PN59_DEBUG=1 — the silent except Exception: pass in chunk.py:54 masks the OOM raised from _vanilla_path. At minimum a logger.info of the bypass reason so operators don't guess. (Possibly: log only when streaming is enabled-but-bypassed, to keep noise low on configs that don't engage PN59 at all.)

4. Update the v7.72 release-notes claim — "Single 24 GiB card now sustains 92K-context generation for an hour without OOM" doesn't hold on Ampere consumer + chunked-prefill (which is the only way 92K-context fits on a 24 GB card). Either restrict the claim to long-prefill-without-chunked-prefill, or document the missing protection envelope.

What I'd like to confirm

If has_no_chunk_metadata is the failing gate (high probability per the source comment), a one-line A/B with that condition relaxed to True should make PN59 actually engage on this rig, and the OOM should disappear. Happy to run that A/B if you want to confirm before designing the proper fix.

[Sandermage]

Hey @lexhoefsloot — apologies for AI-translated English; writing from Odessa.

Did a deep cross-engine + memory-layer research pass after your report (FLA chunk metadata semantics, vLLM KV pool sizing math, PyTorch caching allocator fragmentation, llama.cpp graph allocator, SGLang's lack of streaming, jhsmith409 TQ port). Findings: this is a systemic memory + fragmentation issue, not just a strict-gate bug, and the proper fix is a 2-step plan.

Step 1 — env-only recipe shipped today (commit a7ef419)

The launch script scripts/launch/bare_metal_27b_int4_TQ_k8v4_single_card.sh now bakes in the right combination:

# Cliff 2b chunking — splits FLA T-dim 60K → 4×16K sub-prompts
export GENESIS_ENABLE_P103=1 GENESIS_FLA_FWD_H_MAX_T=16384

# Window-iterative streaming on top
export GENESIS_ENABLE_PN59_STREAMING_GDN=1

# Stop fragmentation churn (was unset, defaulted to 1.0 = never GC)
export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True,max_split_size_mb:256,garbage_collection_threshold:0.85"

vllm serve ... --gpu-memory-utilization 0.85   # was 0.93

Effect on the Cliff:

• h-tensor peak (B, NT, H, V, K) at T=60K: 1.37 GiB → 365 MiB (P103 splits NT=938 → ~256 per sub-call)
• activation headroom: 1.6 GiB → 3.5 GiB (gpu_memory_utilization 0.93 → 0.85, since real KV usage at single-stream 60K is ~3 GiB out of the 20.5 GiB pool — the 0.05 we give back is "paper" capacity)
• fragmentation pressure: low (gc_threshold 0.6 was too aggressive on 22 GiB KV pool — gc fired constantly, churning. 0.85 = gc only under genuine pressure)

P103 was already in Genesis for weeks but nobody had documented the right combination for your case. That's on me — adding to docs/OOM_RECIPES.md as the "noonghunna/club-3090#22 recipe" section.

Please pull dev tip + restart with the new launch script and confirm whether the OOM at chunk_o.py:161 is gone with this stack alone. If yes, this issue is closed by Step 1; if not, the Step 2 structural fix below addresses any residual.

Step 2 — PN59 structural fix (in flight, ~3 days, v7.72.5)

Your option-3 (thread chunk_indices/chunk_offsets through the streaming path) is the structurally correct answer and I'm implementing it now:

1. Per-window metadata slicing — pure arithmetic for single-seq case (which PN59 already gates on): cu_seqlens_w = [0, T_w], chunk_indices_w = stack([zeros(cur_NT), arange(cur_NT)], dim=1), chunk_offsets_w = [0, cur_NT]. Bit-equivalent to upstream prepare_chunk_indices at the window level. No GPU sync, ~5 µs CPU per window.
2. State-chain bug fix — research turned up a latent bug your report would have surfaced: streaming_gdn_driver.py:364 does state = h_w[:, -1].to(float32) which is the state before the kernel applies the last chunk's update (the kernel writes h[i] before the b_h += b_k @ b_v accumulation). Fix is output_final_state=True per intermediate window with state_next chained instead. Strict gate hid this for 4 days.
3. GdnScratchPool production wiring — the pool primitives exist in kernels/gdn_scratch_pool.py but the production driver bypasses them (audit P2.5 honesty banner). Wiring them per the PN12/P22 pattern + adding acquire_o_output() for chunk_o.py:161 cuts allocation churn from ~3000 torch.empty()/forward to 4 boot-time allocations. Eliminates the fragmentation root cause.
4. Drop GENESIS_PN59_STRICT_NO_METADATA env requirement — once 1+2+3 land, PN59 is metadata-correct on every path. Env-flag stays as escape hatch for paranoia, but default = "just works".

Plus +18-25 TDD covering numerical equivalence vs vanilla, slicer edge cases, state-chain bit-equivalence, pool reuse across calls.

Step 3 — engine-level prealloc_v2 (next sprint, ~1-2 weeks)

Cross-engine research found a 5-7 GB gap between vLLM and llama.cpp on the same hardware/model — comes from llama.cpp's ggml_gallocr_reserve() static memory plan that PyTorch eager can't replicate. Genesis can approximate it via "first-call profile + replay" using the existing GenesisPreallocBuffer framework. Closes the Cliff systematically across all FLA/Mamba paths, not just PN59. Tracking separately.

What I need from you

When you have a window after pulling Step 1:

1. confirm the OOM is gone (or report new symptom + new logs)
2. share the new boot summary's [Genesis] structured boot summary block — want to verify P103 + PN59 both register APPLY in your env
3. (optional) any drift you notice in tool-call quality with the recipe — should be neutral but worth eyeballing

Thanks for the careful report — the option-2 + option-3 framing in your original issue is what made the cross-engine research tractable. The chunk_o.py:161 site is now indexed in three different research artifacts so this OOM class never silently re-emerges.

[noonghunna]

@Sandermage — small note first: the issue was filed by noonghunna (us, on cross-rig 1× RTX 3090), not lex. Lex is on club-3090#57 (the related xgrammar bug). Easy mix-up given the cross-thread context-switching this past week. No problem either way — the substance below is what matters.

Cross-rig validation of Step 1 recipe

Result: OOM still fires on club-3090's long-text.yml (TQ3 KV) — Step 1 doesn't fully close it on this config. Honest report below; please use it as another data point for Step 2.

Test A — alloc-conf change ONLY (kept our existing mem-util/max-len)

Applied:

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:256,garbage_collection_threshold:0.85

Unchanged: --gpu-memory-utilization 0.93, --max-model-len 180000. Our compose already had GENESIS_ENABLE_P103=1 + GENESIS_FLA_FWD_H_MAX_T=16384 + GENESIS_ENABLE_PN59_STREAMING_GDN=1 from the v7.72.2-uplift earlier.

Observable wins from gc_threshold change alone:

• Boot KV pool: 78,432 → 241,016 tokens (~3× larger). gc_threshold lifting from default to 0.85 freed substantial allocator churn at boot — confirms the GC framing was right.
• Boot succeeds clean.

But the 60K probe still HTTP 500s:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 276.00 MiB.
GPU 0 has a total capacity of 23.56 GiB of which 274.50 MiB is free.
22.52 GiB is allocated by PyTorch, with 413.90 MiB reserved but unallocated.
File ".../torch/_ops.py", line 865, in __call__

(allocation failed inside an Inductor-compiled kernel under qwen3_next forward)

Test B — full Sander recipe (mem-util 0.85)

Could not boot at our config: --gpu-memory-utilization 0.85 + TQ3 KV + max-model-len 65536 → KV pool admission failure (1.6 GiB needed for max seq len, only 0.29 GiB available).

The reason: our long-text.yml uses TQ3 KV (turboquant_3bit_nc), not k8v4 (which your bare_metal_27b_int4_TQ_k8v4_single_card.sh is built around). TQ3's per-token activation footprint is materially higher than k8v4's ~1 GB/card per club-3090#47. At 0.85 mem-util on TQ3 the model+activation budget consumes the room your recipe assumed would land in the pool.

So your recipe's tuning constants (0.85 mem-util specifically) are calibrated for k8v4 KV. For TQ3 we'd need a different mem-util landing point. The gc_threshold change is portable across KV formats, but the headroom reservation isn't.

Boot summary block (per ask #2)

[Genesis] structured boot summary:
  Genesis:  v7.63.x   (running pin 7b9fd319 — v7.72.2 release commit)
  Patches:  78 total → 45 APPLY | 33 SKIP

  By category:
    • compile_safety         APPLY=  3  SKIP=  1
    • hybrid                 APPLY=  2  SKIP=  0   ← PN59 + P103 both APPLY
    • kernel                 APPLY=  1  SKIP=  0
    • kernel_safety          APPLY=  1  SKIP=  0
    • memory_savings         APPLY=  2  SKIP=  1
    • model_correctness      APPLY=  2  SKIP=  0
    • perf_hotfix            APPLY=  8  SKIP=  9
    • quantization           APPLY=  1  SKIP=  0
    • spec_decode            APPLY= 12  SKIP= 18
    • structured_output      APPLY=  6  SKIP=  6

PN59 dispatcher applied at boot. P103 self-install also active. Confirms patches are loading correctly; the issue is purely runtime activation pressure on TQ3 + chunked-prefill at T=4128 chunks.

What this tells us about Step 2

Your option-3 (thread chunk_indices/chunk_offsets through streaming path) is structurally correct independent of KV format — once PN59 actually engages on chunked-prefill calls (instead of bypassing to _vanilla_path), the h-tensor materialization disappears and the OOM at chunk_o.py:161 should go too. We're holding for v7.72.5.

The state-chain bug at streaming_gdn_driver.py:364 you found is great — the output_final_state=True per-window chain fix means the streaming path will produce bit-identical results vs vanilla, removing the strict-gate's defense-in-depth burden. Awesome catch surfacing through the cross-rig pressure.

GdnScratchPool production wiring (Step 2 item 3) sounds like it'll close the residual fragmentation we're seeing in Test A: 22.5 GiB allocated by PyTorch with 413 MiB unallocated-reserved is exactly the churn signature you described.

Optional — would you like us to test k8v4-converted long-text?

If Step 1 was supposed to close it on a k8v4 config too, we could spin up a k8v4-KV variant of long-text.yml with mem-util=0.85 and re-test. Not sure if you've already validated this on your A5000 PROD or just shipped the launch script blind. Happy to run if useful.

Side-finding worth flagging

GENESIS_PN59_DEBUG=1 doesn't surface bypass reasons in the runtime path because chunk.py's outer try/except catches the OOM raised from _vanilla_path before the eligibility-bypass log fires. We trapped it via injected exception logger (you can see the [PN59_PROBE] line in our earlier diagnosis on this issue). Maybe Step 2 should also add a logger.warning("PN59 streaming path raised: %s") in the silent-fall-through clause? Operators can't currently tell from container logs alone that PN59 is failing — we only know because we hand-instrumented.

Genuinely thanks for the rapid turnaround + the depth of the Step 2 plan. Holding for v7.72.5.

[Sandermage]

An update is coming in a couple of hours once the tests are done. Let's base all future testing on that one. This should finally resolve the OOM problem.

[noonghunna]

Acknowledged — holding off further testing on this rig until your update lands. Ready to validate the moment it ships. Thanks for the rapid iteration on this 🙌

[raphaeldelors-code]

@Sandermage hi and thanks for you great work ! Any update on this fix landing almost ready you mentionned 3 days ago ? Thanks a lot in advance !
@noonghunna fyi

[Sandermage]

In the discussion thread, I’ve posted an estimated timeline for the update and explained the reasons for the delay. I am currently double-checking, reworking, and updating a significant number of components to ensure everything runs smoothly.

[raphaeldelors-code]

Hi @Sandermage ! Do you need any help with this ? Any challenge faced preventing the release ?
@noonghunna fyi

[noonghunna]

@Sandermage — hope you're doing well! Just checking in on the v7.73.x memory-rework — no rush at all, mostly wanted to see how it's coming along and whether there's anything we can help test or unblock.

On our side we've moved the club-3090 defaults to overlay-free stock vLLM v0.22.0 for now, so there's genuinely zero time pressure from us. But TurboQuant + the Cliff-2b fix are still the features people ask about most — someone opened noonghunna/club-3090#342 about exactly this just today — and we'd love to re-integrate Genesis the moment you tag a stable release.

Happy to throw cross-rig validation (1× / 2× 3090) at any RC whenever you're ready. Thanks again for all the work on this — take the time you need. 🙏