Context
Research on competitor rtk-ai/rtk#582 revealed a critical failure mode of bash-output compressors: aggressive compression of input can increase total cost by 18% because the LLM generates 50% more output tokens trying to compensate for missing structural context.
Concrete numbers from that issue:
- Input savings: ~80%
- Output tokens: 1.6k → 2.4k (+50%)
- Total cost: $0.68 → $0.80 (+18%)
- Duration: +26%
We currently measure compression ratio but not information preservation — we are flying blind on whether our compression is causing Claude to re-investigate.
Goal
Add an information-preservation metric to cargo run --release -- benchmark that:
- Counts preserved signal terms in compressed output vs baseline: file paths (
path/to/file.rs:NN), line numbers, error markers (error/warning/FAIL/ERROR), test verdicts (passed/failed).
- Reports preservation_pct per scenario alongside reduction_pct.
- Emits a compression-risk warning when
reduction_pct > 90% AND preservation_pct < 70% — the regime where rtk regression occurred.
- Aggregates risk score across all scenarios.
Acceptance criteria
src/economy/preservation.rs exposes fn info_preservation(baseline: &str, compressed: &str) -> f64 returning value in [0.0, 1.0].
- Detects: file paths, line numbers (
:NN and (NN,NN)), error keywords, test verdicts.
ScenarioResult gains info_preservation: f64 field.
- Benchmark output prints the new column.
- Risk warning emitted when threshold crossed.
- Unit tests cover: high preservation, low preservation (over-compression), edge cases.
Out of scope
Live Claude API calls — follow-up PR (requires API key). This PR delivers the offline proxy metric.
Context
Research on competitor rtk-ai/rtk#582 revealed a critical failure mode of bash-output compressors: aggressive compression of input can increase total cost by 18% because the LLM generates 50% more output tokens trying to compensate for missing structural context.
Concrete numbers from that issue:
We currently measure compression ratio but not information preservation — we are flying blind on whether our compression is causing Claude to re-investigate.
Goal
Add an information-preservation metric to
cargo run --release -- benchmarkthat:path/to/file.rs:NN), line numbers, error markers (error/warning/FAIL/ERROR), test verdicts (passed/failed).reduction_pct > 90% AND preservation_pct < 70%— the regime where rtk regression occurred.Acceptance criteria
src/economy/preservation.rsexposesfn info_preservation(baseline: &str, compressed: &str) -> f64returning value in [0.0, 1.0].:NNand(NN,NN)), error keywords, test verdicts.ScenarioResultgainsinfo_preservation: f64field.Out of scope
Live Claude API calls — follow-up PR (requires API key). This PR delivers the offline proxy metric.