eval: sub-0.1% perplexity deltas are sequence artifacts — two deterministic harnesses disagree 5.7x on identical inputs #429

Found while independently reproducing PR #413's calibration curve. Not a defect in that PR — a measurement-methodology limit that affects how any perplexity threshold in this repo should be derived.

The finding

Two perplexity harnesses, each byte-reproducible in isolation, disagree on identical inputs:

Same corpus (byte-diffed identical), same options, same layer partition. A 5.7x disagreement between two deterministic pipelines.

Determinism was verified, not assumed: repeating clean / 5% / 0.5% / both rounding configs at widely separated schedule positions gave spread = 0.000E+000 on every point, and a full 15-point curve reproduced byte-identically in a second fresh process.

The disagreement scales with the number of layer-window boundaries:

Mechanism unproven. Allocation history and cuBLAS algorithm selection are the leading hypothesis — stated as a hypothesis, not a conclusion.

Why it matters beyond one PR

A sub-0.1% NLL delta measured on this hardware is not a constant — it is a property of the run sequence. Any threshold derived at that scale is measuring the harness, not the model.

That directly touches how this project derives its bounds:

• The test(coverage): no end-to-end cross-backend gate for quantization kernels — how #254 reached dev green #256 cross-backend quant gate's 0.994 cosine bound was derived from a sweep — and .docs/HANDOFF.md already records that sweep as contaminated by a different artifact (seed pinning), requiring re-derivation. This is a second, independent reason to re-derive from a clean distribution rather than a single cell.
• PR feat(eval): perplexity layer-select/offload plus a layer-cycling mode (#395) #413's docs/GPU.md sensitivity table publishes several sub-0.1% figures (0.054% rounding, 0.043% uniform, 0.036% whole-device) as though constant. Being corrected there under this issue.
• Any future "backend X matches CPU to N%" claim below ~0.1% inherits the same problem.

A second finding from the same curve

The perturbation response plateaus and is non-monotonic below ~2%: an element-wise 0.5% boundary perturbation moved the figure more (0.1375%) than a 1.0% one (0.1311%) — corroborated independently by the other harness's 1% -> 0.123%. Below roughly 2%, the effect is dominated by which tokens flip, not by perturbation magnitude.

So "dose-response" framing is misleading in that regime, and a tolerance set there cannot be reasoned about linearly.

Ask

• Establish and document a floor of meaningfulness for perplexity/NLL deltas on this hardware — below which a difference must be reported as indistinguishable rather than as a value.
• Require any published sensitivity or tolerance figure below that floor to be given as a range with the measurement conditions, not a scalar.
• Re-derive the test(coverage): no end-to-end cross-backend gate for quantization kernels — how #254 reached dev green #256 cosine bound from a clean distribution, treating this alongside the seed-pinning contamination already recorded.
• Investigate the mechanism (allocation history vs cuBLAS algorithm selection) far enough to know whether it can be pinned — e.g. by fixing the cuBLAS workspace or algorithm — or must simply be bounded.
• Note in docs/ that below ~2% boundary error the response is token-flip dominated and non-monotonic.

Evidence

Measured first-hand on the local RTX 3060 at e5d3b553; full calibration curve run twice, byte-identical each time. Related: #395, #413, #256, #417.