Skip to content

P0: Add output-token inflation guard to benchmark #1

Description

@jhonatanjunio

Context

Research on competitor rtk-ai/rtk#582 revealed a critical failure mode of bash-output compressors: aggressive compression of input can increase total cost by 18% because the LLM generates 50% more output tokens trying to compensate for missing structural context.

Concrete numbers from that issue:

  • Input savings: ~80%
  • Output tokens: 1.6k → 2.4k (+50%)
  • Total cost: $0.68 → $0.80 (+18%)
  • Duration: +26%

We currently measure compression ratio but not information preservation — we are flying blind on whether our compression is causing Claude to re-investigate.

Goal

Add an information-preservation metric to cargo run --release -- benchmark that:

  1. Counts preserved signal terms in compressed output vs baseline: file paths (path/to/file.rs:NN), line numbers, error markers (error/warning/FAIL/ERROR), test verdicts (passed/failed).
  2. Reports preservation_pct per scenario alongside reduction_pct.
  3. Emits a compression-risk warning when reduction_pct > 90% AND preservation_pct < 70% — the regime where rtk regression occurred.
  4. Aggregates risk score across all scenarios.

Acceptance criteria

  • src/economy/preservation.rs exposes fn info_preservation(baseline: &str, compressed: &str) -> f64 returning value in [0.0, 1.0].
  • Detects: file paths, line numbers (:NN and (NN,NN)), error keywords, test verdicts.
  • ScenarioResult gains info_preservation: f64 field.
  • Benchmark output prints the new column.
  • Risk warning emitted when threshold crossed.
  • Unit tests cover: high preservation, low preservation (over-compression), edge cases.

Out of scope

Live Claude API calls — follow-up PR (requires API key). This PR delivers the offline proxy metric.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions