quantization_retention_seed_tasks.jsonl

{"id": "quant_001", "capability": "quantization_retention", "domain": "inference", "difficulty": "medium", "prompt": "A model at f16 scores 0.74 on the planning suite. After q4_K_M quantization the planning score drops to 0.61 but repair_success holds at 0.71. Does this candidate pass Gate v2's quantization_retention floor of 0.90 relative retention? Show your calculation.", "reference": {"rubric": {"quality_delta": "bounded", "calibration_delta": "bounded", "repair_delta": "bounded"}}, "scoring": "rubric", "tags": ["quantization", "retention", "gate"]}
{"id": "quant_002", "capability": "quantization_retention", "domain": "inference", "difficulty": "medium", "prompt": "You are comparing a q8_0 and a q5_K_S export of the same model. The q8 variant is 2.1x larger on disk. At what quality retention threshold would you prefer q5_K_S over q8_0 for a deployment where disk is the primary constraint?", "reference": {"rubric": {"quality_delta": "bounded", "calibration_delta": "bounded", "repair_delta": "bounded"}}, "scoring": "rubric", "tags": ["quantization", "tradeoff", "disk"]}
{"id": "quant_003", "capability": "quantization_retention", "domain": "inference", "difficulty": "hard", "prompt": "A GGUF export at q4_K_M passes planning and critique retention checks but fails the calibration check (score drops from 0.83 to 0.71, a 14.5% drop vs the 10% ceiling). Propose a remediation strategy that does not require retraining from scratch.", "reference": {"rubric": {"quality_delta": "bounded", "calibration_delta": "bounded", "repair_delta": "bounded"}}, "scoring": "rubric", "tags": ["quantization", "calibration", "remediation"]}
{"id": "quant_004", "capability": "quantization_retention", "domain": "inference", "difficulty": "medium", "prompt": "State the key behavioural difference between quantization schemes q8_0 and q4_K_M. In what task category would you expect the largest quality gap, and why?", "reference": {"rubric": {"quality_delta": "bounded", "calibration_delta": "bounded", "repair_delta": "bounded"}}, "scoring": "rubric", "tags": ["quantization", "comparison"]}
{"id": "quant_005", "capability": "quantization_retention", "domain": "inference", "difficulty": "hard", "prompt": "A byte-level transformer with vocab_size=256 is exported to GGUF. Unlike word-piece models, every token is a single byte. How does this affect quantization sensitivity compared to a 32K-vocabulary model, and what does it imply for the safe quantization floor?", "reference": {"rubric": {"quality_delta": "bounded", "calibration_delta": "bounded", "repair_delta": "bounded"}}, "scoring": "rubric", "tags": ["quantization", "byte-level", "architecture"]}
{"id": "quant_006", "capability": "quantization_retention", "domain": "inference", "difficulty": "medium", "prompt": "You have a q4_K_M export and a user reports that the model's outputs on the reflection capability feel 'less self-correcting' than the f16 version. Describe how you would construct a targeted micro-benchmark to confirm or refute this observation.", "reference": {"rubric": {"quality_delta": "bounded", "calibration_delta": "bounded", "repair_delta": "bounded"}}, "scoring": "rubric", "tags": ["quantization", "evaluation", "reflection"]}
{"id": "quant_007", "capability": "quantization_retention", "domain": "inference", "difficulty": "medium", "prompt": "The Gate v2 spec requires quantization_retention >= 0.90. A candidate scores 0.91 at q8_0 but only 0.87 at q4_K_M. The deployment target requires q4_K_M. What is the gate decision and what options does the operator have?", "reference": {"rubric": {"quality_delta": "bounded", "calibration_delta": "bounded", "repair_delta": "bounded"}}, "scoring": "rubric", "tags": ["quantization", "gate", "operator"]}
{"id": "quant_008", "capability": "quantization_retention", "domain": "inference", "difficulty": "hard", "prompt": "Explain why perplexity on a held-out text corpus is a necessary but not sufficient quality metric for an instruction-tuned model. What additional evaluation would you require before accepting a quantized export into the ledger?", "reference": {"rubric": {"quality_delta": "bounded", "calibration_delta": "bounded", "repair_delta": "bounded"}}, "scoring": "rubric", "tags": ["quantization", "evaluation", "perplexity"]}
{"id": "quant_009", "capability": "quantization_retention", "domain": "inference", "difficulty": "medium", "prompt": "A model is quantized with llama.cpp's q4_K_M scheme. The ANCF wrapper records both the f16 SHA-256 and the q4 SHA-256. A user re-exports from the same f16 source and gets a different q4 SHA-256. Is this expected or a sign of corruption? Explain.", "reference": {"rubric": {"quality_delta": "bounded", "calibration_delta": "bounded", "repair_delta": "bounded"}}, "scoring": "rubric", "tags": ["quantization", "integrity", "sha256"]}
{"id": "quant_010", "capability": "quantization_retention", "domain": "inference", "difficulty": "hard", "prompt": "A team argues that for a retrieval-augmented system the base model's quantization level barely matters because context dominates the response. Under what conditions is this argument valid, and when does it break down?", "reference": {"rubric": {"quality_delta": "bounded", "calibration_delta": "bounded", "repair_delta": "bounded"}}, "scoring": "rubric", "tags": ["quantization", "rag", "reasoning"]}