Skip to content

Honesty: unknown-model output ceilings are fabricated as 'documented' (Anthropic 64K floor, Codex OAuth 4096) #5440

Description

@Hmbown

Class (from the self-improving-harness audit)

"Silent fallback masquerading as a verified capability." Same family as #5239
(context windows), different axis: max output tokens.

Finding 1 — Anthropic-family unknown models: 64K floor labeled "documented"

crates/tui/src/config.rs:637: max_output: Some(max_output_tokens_for_model(resolved_model).unwrap_or(64_000)) with the comment "64K is the documented Anthropic Messages floor, so it stays a known cap rather than an unknown" (config.rs:634-636).

The floor is real, but for an unknown model it becomes the effective cap on every request via output_ceiling_sourceOutputCeilingSource::Documented(64_000) (route_budget.rs:163-173) → clamp at route_budget.rs:178-193 (request sites: turn_loop.rs:945, client.rs:2028, subagent/mod.rs:10647, review.rs:508, verify.rs:458). Receipts label it "documented" (route_budget.rs:126-131); the picker shows 65536 out (model_picker.rs:1849-1850).

Truth: an unknown Anthropic-family model's real ceiling is unverified — could be 64K, could be 1M class. Silently clamping every response to the floor trades late-failure for early-truncation without telling the user which happened or why.

Finding 2 — Codex OAuth route: every response clamped to 4096

config.rs:654: max_output: Some(4096) with the admission "The OAuth cache does not publish an output ceiling" (:651-653). Pinned by test engine/tests.rs:13295 (gpt-5.5 → 4_096). The picker already refuses to display it ("omitting it is more truthful", model_picker.rs:2108-2113) — the right instinct — but the request-side clamp and the "documented" receipt label persist for every Codex OAuth session.

Direction

Mirror the #5239 pattern: OutputCeilingSource gains provenance (Documented floor vs Unverified default vs Configured override), unverified ceilings stop claiming "documented" in receipts, and the config override (if one exists for max_output — verify) is surfaced as the active source. Decision point for @HmbownBown: for unverified ceilings, clamp-with-honest-label (current behavior, labeled) vs no-clamp-plus-warning. Clamping to a floor is defensible; lying about why is not.

Acceptance

  • OutputCeilingSource distinguishes documented floor from unverified default.
  • Receipts/picker never render an unverified ceiling as "documented".
  • Tests: unknown Anthropic model + no override; known model; Codex OAuth route.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions