Stale queue

[haizhongzheng]

Title:

feat(dataflow): earliest-deadline-first (EDF) rollout buffer consumption

Body:

Problem: FIFO consumption creates a difficulty bias in staleness drops

Long generations span several weight updates while they are being produced, so
their min_version is already old the moment they land in the rollout buffer.
Consuming in completion order (FIFO) then parks them behind fresher short
samples until they exceed max_staleness and get dropped.

The two effects compound into a systematic bias: the long/hard samples a
model most needs to learn are preferentially discarded. In our measurements,
FIFO's staleness-dropped samples are 1.3–1.7× longer than the samples it
trains on.

Change: consume by staleness deadline instead of arrival order

The rollout buffer is now a priority queue consumed in ascending min_version
order (earliest deadline first) — the samples closest to expiring are
trained on first, before they cross the staleness bound.

• New config key dataflow.buffer.queue_order: edf (default) or fifo
  (the historical behavior, kept for comparison).
• The per-token staleness invariant is unchanged: EDF only reorders which
  buffered samples are consumed first; nothing older than max_staleness is
  ever trained on, and the replay path is untouched.
• Ties on min_version break by arrival order, so rollout groups stay
  contiguous; unversioned samples are consumed eagerly.
• New W&B metrics to observe the bias directly: buffer/consumed_len_avg,
  buffer/skipped_stale_len_avg, plus buffer-eviction accounting.
• Checkpoint compatibility: state_dict round-trips across queue_order
  settings (priorities are recomputed from metadata on load), and legacy
  deque-format checkpoints still load.
• 14 new unit tests: FIFO parity, EDF ordering/ties, a deterministic
  mixed-length load simulation showing the bias and its fix, and
  state-dict round-trips (astraflow/dataflow/tests/test_rollout_buffer_queue_order.py).

A/B results (Qwen3-4B, M2PO on math, 8×H100)

Controlled pair under real staleness pressure (6 GPUs RaaS vs 2 GPUs FSDP,
max_staleness: 8, ctx 16k / max_new_tokens 14000). Recipes included in
this PR: examples/math/qwen3-4b-m2po-edf/ and .../qwen3-4b-m2po-fifo/.

Overall avg@k across AIME24/25, AMC, Minerva, MATH500:

step	edf	fifo	Δ
0	42.2	43.1	—
100	57.6	52.5	+5.1
200	59.2	56.9	+2.3
250	59.5	58.1	+1.4
350	58.6	59.6	−1.0 (converged)

• The @100 gap reproduced across two independent edf runs (57.6 and 57.8).
• Gains are largest on the hardest sets — @100: AIME25 +9.4, AIME24 +5.4,
  AMC +5.0.
• Mechanism confirmed by the new metrics: fifo drops are length-biased
  (dropped/consumed length ratio 1.3–1.7×) with 8.3–17.1% drop rate; edf is
  near-fair (~1.1×) at 7.4–9.6%.
• Net effect: sample efficiency — edf reaches fifo's eval plateau ~150
  steps earlier; with enough steps both arms converge.

Also in this PR

• docs: installation.md now warns that flash-attn must be installed from
  outside the repo directory — [tool.uv] exclude-dependencies makes
  uv pip install flash-attn a silent no-op inside it, and a leftover
  namespace stub makes import flash_attn falsely succeed. The verify step
  now imports flash_attn_func and checks the installed version.

Compatibility / rollout notes

• Default changes to edf. Existing YAMLs without queue_order pick it up
  automatically; set dataflow.buffer.queue_order: fifo to restore the old
  behavior exactly.
• No API changes; config plumbing is additive
  (loader → AgentConfig → AstraFlow → MultiModelDataServing → RolloutBuffer).