REGRESSION: Format conversion still produces large diffs after #177 fix

[noahgift]

REGRESSION: Format Conversion Still Failing After #177 Fix

Status: REGRESSION from closed #177
Severity: P0 (CRITICAL - Data Corruption)
Component: apr-rosetta / realizear
Discovered By: apr-model-qa-playbook requalification (2026-01-30)
Blocking: Model qualification certification

---

Executive Summary

Issue #177 was closed, but requalification testing on 2026-01-30 shows format conversion still fails with large output differences. The Jidoka detection is working (diffs are flagged), but the root cause fix is incomplete.

---

Regression Evidence

Test Environment

Date: 2026-01-30T14:59:00Z
Host: noah-Lambda-Vector
Model: Qwen/Qwen2.5-Coder-1.5B-Instruct (GGUF Q4_K_M)
Path: /home/noah/.cache/huggingface/hub/models--Qwen--Qwen2.5-Coder-1.5B-Instruct-GGUF/snapshots/.../qwen2.5-coder-1.5b-instruct-q4_k_m.gguf
Playbook: qwen2.5-coder-1.5b-ci.playbook.yaml

Test Results

Total scenarios: 57
Passed: 50
Failed: 7  ← ALL 7 ARE FORMAT CONVERSION
Pass rate: 89.3%  ← Should be 100%

Detailed Failures

Gate	Conversion	Diff	Tolerance	Verdict
F-CONV-001	GGUF → APR	6.77e-1	1.00e-6	❌ FAIL (677,000× over tolerance)
F-CONV-002	APR → GGUF	4.16e-1	1.00e-6	❌ FAIL (416,000× over tolerance)
F-CONV-003	GGUF → SafeTensors	Infrastructure error	-	❌ FAIL (see below)
F-CONV-004	SafeTensors → GGUF	4.16e-1	1.00e-6	❌ FAIL
F-CONV-005	APR → SafeTensors	Infrastructure error	-	❌ FAIL (see below)
F-CONV-006	SafeTensors → APR	6.77e-1	1.00e-6	❌ FAIL
F-CONV-RT-001	Round-trip	Blocked	-	❌ FAIL

Raw Evidence from evidence.json

{
  "gate_id": "F-CONV-G-A",
  "outcome": "Falsified",
  "reason": "Conversion Gguf → Apr produced different output (diff: 6.77e-1, ε: 1.00e-6)",
  "output": "6de63189564fc936",
  "timestamp": "2026-01-30T14:07:23.xxx"
}

{
  "gate_id": "F-CONV-A-G", 
  "outcome": "Falsified",
  "reason": "Conversion Apr → Gguf produced different output (diff: 4.16e-1, ε: 1.00e-6)",
  "output": "0356a3e657672e25",
  "timestamp": "2026-01-30T14:07:35.xxx"
}

---

Comparison: Before vs After #177 Fix

Metric	Before #177	After #177	Status
NaN detection	❌ Silent	✅ Detected	FIXED
Inf detection	❌ Silent	✅ Detected	FIXED
Output diff (GGUF→APR)	8.46e-1	6.77e-1	WORSE → BETTER (15% improvement)
Output diff (APR→GGUF)	6.34e-1	4.16e-1	WORSE → BETTER (34% improvement)
Within tolerance (ε=1e-6)	❌ No	❌ No	STILL FAILING
Round-trip lossless	❌ No	❌ No	STILL FAILING

Conclusion: #177 fix improved detection and reduced diff magnitude, but diffs are still 400,000× to 700,000× above tolerance.

---

Root Cause Hypothesis

The #177 fix addressed:

1. ✅ NaN/Inf detection (Jidoka working)
2. ✅ Some quantization parameter handling

But did NOT address:

1. ❌ Quantization scale/offset precision loss
2. ❌ Block-wise quantization metadata transfer
3. ❌ Q4_K_M super-block structure preservation

Technical Detail

Q4_K_M uses a two-level quantization structure:

Super-block (256 elements):
  - Scale (fp16)
  - Min (fp16)
  - 32× Sub-blocks of 8 elements each
    - Sub-scale (6-bit)
    - 4-bit quantized weights

If the super-block scales are truncated or misaligned during conversion, all weights in that block will be off by a multiplicative factor, leading to the large cumulative diffs we observe.

---

Suggested Additional Fixes

1. Preserve Full Quantization Metadata

struct Q4KMSuperBlock {
    d: f16,      // Super-block scale - MUST preserve full precision
    dmin: f16,   // Super-block min - MUST preserve full precision
    scales: [u8; 12],  // Sub-block scales - MUST preserve bit-exact
    qs: [u8; 128],     // Quantized values
}

// During conversion, ensure:
// 1. d and dmin are NOT downcast to f32 then back to f16
// 2. scales array is copied bit-exact, not recomputed
// 3. Block alignment matches source format

2. Add Tensor-Level Validation

fn validate_conversion(source: &Tensor, converted: &Tensor) -> Result<()> {
    let diff = (source.to_f32() - converted.to_f32()).abs().max();
    if diff > EPSILON {
        return Err(ConversionError::LossyConversion { 
            diff, 
            tolerance: EPSILON,
            tensor_name: source.name.clone(),
        });
    }
    Ok(())
}

3. Test Each Quantization Type Separately

# Test suite should cover:
apr rosetta convert model_q4_k_m.gguf test.apr && apr rosetta convert test.apr model_back.gguf
apr rosetta convert model_q5_k_m.gguf test.apr && apr rosetta convert test.apr model_back.gguf
apr rosetta convert model_q8_0.gguf test.apr && apr rosetta convert test.apr model_back.gguf
apr rosetta convert model_f16.gguf test.apr && apr rosetta convert test.apr model_back.gguf
# All should produce diff < 1e-6

---

MQS Impact

Metric	Current	Required
Score	41.1/100	87+/100
Grade	F	B or higher
Conversion gates	0/7	7/7
Lost points	~45	0

---

Verification Criteria

Issue is resolved when:

cd ../apr-model-qa-playbook
cargo run --bin apr-qa -- run playbooks/models/qwen2.5-coder-1.5b-ci.playbook.yaml \
  --subprocess --model-path <model.gguf> --no-gpu --output output/verify

# Required:
# - F-CONV-001 through F-CONV-006: ALL PASS (diff < 1e-6)
# - F-CONV-RT-001: PASS (round-trip lossless)
# - MQS Score: 87+/100
# - Pass rate: 100%

---

References

• Original issue: P0 CRITICAL: Format conversion introduces NaN/Inf corruption in tensor weights #177 (CLOSED - but regression detected)
• Evidence file: ../apr-model-qa-playbook/output/qwen-requalify/evidence.json
• MQS report: ../apr-model-qa-playbook/output/qwen-requalify/mqs.json
• Verification playbook: ../apr-model-qa-playbook/playbooks/verify/TICKET-177.yaml
• Spec: Section 4 (Format Conversion Testing), tolerance = 1e-6

---

Filed by: apr-model-qa-playbook requalification (automated)
Related: #177 (regression), #172 (original P0)

[noahgift]

Five Whys Root Cause Analysis

Why #	Question	Answer
Why 1	Why do converted models produce different output?	Because tensor values are different after conversion (diff = 4e-1 to 7e-1)
Why 2	Why are tensor values different?	Because quantization parameters (scales, mins) are not preserved exactly
Why 3	Why aren't quantization parameters preserved?	Because Q4_K_M uses 2-level block structure that isn't transferred correctly
Why 4	Why isn't the block structure transferred correctly?	Because conversion dequantizes to f32, then requantizes - losing block alignment
Why 5	ROOT CAUSE: Why does requantization lose alignment?	The conversion path is: Q4_K_M → f32 → APR → f32 → Q4_K_M. Each dequant/requant step introduces cumulative error because block boundaries and sub-block scales are recomputed rather than preserved.

---

Detailed Technical Analysis

Q4_K_M Block Structure (GGUF)

SuperBlock (256 weights, 140 bytes):
├── d: f16 (2 bytes)      ← SCALE for entire super-block
├── dmin: f16 (2 bytes)   ← MIN for entire super-block  
├── scales: [u8; 12]      ← 32 sub-block scales (6-bit packed)
└── qs: [u8; 128]         ← 256 quantized weights (4-bit packed)

Dequantization formula:
  weight[i] = (qs[i] * sub_scale[i/8] * d) + dmin

Current Conversion Path (LOSSY)

GGUF Q4_K_M                APR Format               GGUF Q4_K_M (converted)
┌─────────────┐           ┌─────────────┐           ┌─────────────┐
│ d=0.0234    │  dequant  │ f32 tensor  │  requant  │ d=0.0231    │ ← DIFFERENT!
│ dmin=-0.012 │ ────────► │ [0.1, -0.3, │ ────────► │ dmin=-0.011 │ ← DIFFERENT!
│ scales=[...] │           │  0.2, ...]  │           │ scales=[...] │ ← RECOMPUTED
│ qs=[...]    │           └─────────────┘           │ qs=[...]    │ ← DIFFERENT!
└─────────────┘                                     └─────────────┘

Error accumulation:
- d precision loss: f16 → f32 → f16 (rounding)
- dmin precision loss: f16 → f32 → f16 (rounding)  
- scales recomputed: optimal for NEW data, not original
- qs recomputed: based on new scales, compounds error

Required Conversion Path (LOSSLESS)

GGUF Q4_K_M                APR Format               GGUF Q4_K_M (converted)
┌─────────────┐           ┌─────────────┐           ┌─────────────┐
│ d=0.0234    │  preserve │ d=0.0234    │  restore  │ d=0.0234    │ ← IDENTICAL
│ dmin=-0.012 │ ────────► │ dmin=-0.012 │ ────────► │ dmin=-0.012 │ ← IDENTICAL
│ scales=[...] │  metadata │ scales=[...] │           │ scales=[...] │ ← IDENTICAL
│ qs=[...]    │           │ qs=[...]    │           │ qs=[...]    │ ← IDENTICAL
└─────────────┘           └─────────────┘           └─────────────┘

Key insight: Don't dequantize at all - transfer the quantized representation directly.

---

Suggested Fix Implementation

Step 1: Preserve Quantization Metadata

// In apr-rosetta/src/convert.rs

/// Quantization metadata that MUST be preserved for lossless conversion
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct QuantizationMetadata {
    pub quant_type: QuantType,  // Q4_K_M, Q5_K_M, etc.
    pub block_size: usize,      // 256 for K-quants
    pub sub_block_size: usize,  // 8 for K-quants
    
    // Per-block parameters (DO NOT RECOMPUTE)
    pub scales: Vec<f16>,       // Super-block scales
    pub mins: Vec<f16>,         // Super-block mins
    pub sub_scales: Vec<u8>,    // Sub-block scales (packed)
}

fn convert_gguf_to_apr(gguf: &GgufFile) -> Result<AprFile> {
    let mut apr = AprFile::new();
    
    for tensor in &gguf.tensors {
        // DON'T DO THIS (lossy):
        // let f32_data = tensor.dequantize();
        // apr.add_tensor(tensor.name, f32_data);
        
        // DO THIS (lossless):
        apr.add_quantized_tensor(
            &tensor.name,
            tensor.quantized_data.clone(),  // Raw quantized bytes
            tensor.extract_quant_metadata(), // Preserve ALL metadata
        );
    }
    
    Ok(apr)
}

Step 2: Add Tensor-Level Validation

fn validate_conversion(source: &Tensor, converted: &Tensor) -> Result<()> {
    // For quantized tensors: compare raw bytes
    if source.is_quantized() && converted.is_quantized() {
        if source.raw_bytes() != converted.raw_bytes() {
            return Err(ConversionError::QuantizedMismatch {
                tensor: source.name.clone(),
                source_hash: hash(source.raw_bytes()),
                converted_hash: hash(converted.raw_bytes()),
            });
        }
        return Ok(());
    }
    
    // For float tensors: compare with epsilon
    let diff = (source.to_f32() - converted.to_f32()).abs().max();
    if diff > 1e-6 {
        return Err(ConversionError::FloatMismatch {
            tensor: source.name.clone(),
            diff,
            tolerance: 1e-6,
        });
    }
    
    Ok(())
}

Step 3: Test Each Quantization Type

#[test]
fn test_roundtrip_q4_k_m() {
    let original = load_gguf("test_q4_k_m.gguf");
    let apr = convert_gguf_to_apr(&original).unwrap();
    let roundtrip = convert_apr_to_gguf(&apr).unwrap();
    
    for (orig_tensor, rt_tensor) in original.tensors.iter().zip(roundtrip.tensors.iter()) {
        // Quantized data must be BITWISE IDENTICAL
        assert_eq!(
            orig_tensor.raw_bytes(), 
            rt_tensor.raw_bytes(),
            "Tensor {} differs after roundtrip", orig_tensor.name
        );
    }
}

---

Suggested Testing Protocol

Phase 1: Unit Tests (Isolation)

# Test quantization metadata extraction
cargo test -p realizear test_extract_q4km_metadata

# Test metadata preservation during conversion
cargo test -p apr-rosetta test_preserve_quant_metadata

# Test round-trip for single tensor
cargo test -p apr-rosetta test_single_tensor_roundtrip

Phase 2: Integration Tests (Full Model)

# Convert and validate each direction
MODEL="test_models/qwen-1.5b-q4_k_m.gguf"

# Step 1: GGUF → APR
apr rosetta convert "$MODEL" test.apr
apr rosetta validate test.apr --compare-to "$MODEL"
# Expected: "All 322 tensors match (0 diffs)"

# Step 2: APR → GGUF  
apr rosetta convert test.apr test_back.gguf
apr rosetta validate test_back.gguf --compare-to "$MODEL"
# Expected: "All 322 tensors match (0 diffs)"

# Step 3: Inference comparison
PROMPT="What is 2+2?"
OUTPUT_ORIG=$(apr run "$MODEL" "$PROMPT" -n 32 --seed 42)
OUTPUT_CONV=$(apr run test_back.gguf "$PROMPT" -n 32 --seed 42)
diff <(echo "$OUTPUT_ORIG") <(echo "$OUTPUT_CONV")
# Expected: No diff (identical output)

Phase 3: Property-Based Tests (Exhaustive)

use proptest::prelude::*;

proptest! {
    #[test]
    fn roundtrip_preserves_output(
        seed in 0u64..1000,
        prompt in "[a-zA-Z ]{10,50}",
    ) {
        let original = load_model("test.gguf");
        let converted = roundtrip_convert(&original);
        
        let out1 = inference(&original, &prompt, seed);
        let out2 = inference(&converted, &prompt, seed);
        
        prop_assert_eq!(out1, out2, "Output differs for seed={}, prompt={}", seed, prompt);
    }
}

Phase 4: Verification via apr-qa-playbook

cd ../apr-model-qa-playbook
cargo run --bin apr-qa -- run playbooks/verify/TICKET-181.yaml \
  --subprocess \
  --model-path /path/to/model.gguf \
  --no-gpu \
  --output output/verify-181

# Expected output:
# V-TICKET-181-001: ✅ PASS (diff < 1e-6)
# V-TICKET-181-002: ✅ PASS (diff < 1e-6)
# V-TICKET-181-003: ✅ PASS (diff < 1e-6)
# V-TICKET-181-004: ✅ PASS (diff < 1e-6)
# V-TICKET-181-005: ✅ PASS (diff < 1e-6)
# V-TICKET-181-006: ✅ PASS (diff < 1e-6)
# V-TICKET-181-007: ✅ PASS (round-trip lossless)

---

Acceptance Criteria Checklist

• Unit test: Q4_K_M metadata extraction preserves all fields
• Unit test: Single tensor round-trip is bitwise identical
• Integration test: Full model round-trip (GGUF→APR→GGUF) produces identical tensors
• Integration test: Inference output identical before/after conversion
• Property test: 1000 random prompts produce identical output
• apr-qa-playbook: All 7 V-TICKET-181-* gates pass
• MQS score: ≥87/100 for Qwen2.5-Coder-1.5B