cnb: surface model downgrade + token budget alerts (#153) - #221
cnb: surface model downgrade + token budget alerts (#153)#221ApolloZhangOnGithub wants to merge 4 commits into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 89233ed50d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| PRICING = { | ||
| "claude-opus-4-6": {"input": 15.0, "output": 75.0, "cache_read": 1.5, "cache_create": 18.75}, | ||
| "claude-opus-4-7": {"input": 15.0, "output": 75.0, "cache_read": 1.5, "cache_create": 18.75}, | ||
| "claude-sonnet-4-6": {"input": 3.0, "output": 15.0, "cache_read": 0.3, "cache_create": 3.75}, | ||
| "claude-sonnet-4-7": {"input": 3.0, "output": 15.0, "cache_read": 0.3, "cache_create": 3.75}, | ||
| "claude-haiku-4-5-20251001": {"input": 0.80, "output": 4.0, "cache_read": 0.08, "cache_create": 1.0}, |
There was a problem hiding this comment.
Restore pricing for existing Sonnet 4.6 logs
Because estimate_cost() falls back to DEFAULT_PRICING (Opus rates) for any model not listed here, removing the claude-sonnet-4-6 entry makes existing/full-history JSONLs from Sonnet 4.6 runs get billed at $90/M instead of the prior $18/M. This affects cnb usage/board usage totals and can also trigger false budget alerts for projects with recent 4.6 Sonnet sessions; keep the old model key alongside the new 4.7 key.
Useful? React with 👍 / 👎.
| if not any(a in ("--budget", "--budget-usd") for a in args): | ||
| budget, warn_pct = load_budget_defaults(db.env.claudes_dir) | ||
| if budget > 0: | ||
| args = [*args, "--budget", str(budget), "--warn-pct", str(warn_pct)] |
There was a problem hiding this comment.
Preserve user-supplied warn-pct
When a default budget is configured, board usage --warn-pct 50 appends the config's --warn-pct after the user's flag, and _parse_usage_args() keeps the later value. In that scenario the explicit CLI threshold is silently ignored unless the user also passes --budget, so append only the missing budget or avoid adding --warn-pct when it was already provided.
Useful? React with 👍 / 👎.
| usd = float(section.get("usd", 0) or 0) | ||
| warn_pct = float(section.get("warn_pct", DEFAULT_BUDGET_WARN_PCT)) |
There was a problem hiding this comment.
Handle non-numeric budget config values
If [budget] exists but usd or warn_pct is not numeric (for example while hand-editing .cnb/config.toml), these float() calls raise ValueError; board overview and board view call this path on startup, so the dashboard crashes before showing any sessions. Since malformed TOML is already treated as disabled, non-numeric budget fields should be caught and defaulted as well.
Useful? React with 👍 / 👎.
| aggregated = aggregate_by_name(sessions) | ||
| alerts: list[str] = list(model_state_alerts(aggregated)) |
There was a problem hiding this comment.
Order multi-session model history chronologically
When the same tongxue has multiple JSONL sessions in the 6-hour window, this aggregates their model histories before checking for downgrades, but _load_project_sessions() feeds them in filename order (sorted(project_dir.glob(...))), not by message or mtime. Since Claude JSONL names are session IDs, a restarted tongxue can get a false downgrade (or miss a real one) depending only on UUID sort order; check each session independently or sort/merge by actual recency before using the first and last models.
Useful? React with 👍 / 👎.
03771c2 to
af2f1c9
Compare
ApolloZhangOnGithub
left a comment
There was a problem hiding this comment.
Peer review under PR freeze (cannot approve as author, but flagging one substantive concern + small notes).
Substantive — PRICING table regression for historical 4-6 JSONLs.
The diff removes claude-opus-4-6 and claude-sonnet-4-6 from the PRICING dict in favor of the 4-7 entries. Numerically identical, so live data is fine. But for any historical JSONL still on disk that has `"model": "claude-sonnet-4-6"`, estimate_cost will now fall through to the default opus pricing ($15/$75/$1.5/$18.75) instead of the real sonnet pricing ($3/$15/$0.3/$3.75). That's a 3-5× over-estimate per million tokens for any sonnet-4-6 session in the report.
Suggestion: keep the 4-6 keys as aliases pointing to the same values:
PRICING = {
\"claude-opus-4-6\": {...}, # legacy alias
\"claude-opus-4-7\": {...},
\"claude-sonnet-4-6\": {...}, # legacy alias
\"claude-sonnet-4-7\": {...},
\"claude-haiku-4-5-20251001\": {...},
}Or use a single source of truth and alias via dict merge. Either way, don't let historical sessions silently re-price as opus.
Nits (non-blocking):
_load_project_sessionsusesrecent_hoursbased onjf.stat().st_mtime. If a long-running session's JSONL hasn't been touched in 6h+ but is still the active model, downgrade alerts silently skip it. Likely fine for the live view since you want recent state, but worth a one-line comment explaining the tradeoff.load_budget_defaults:float(section.get(\"usd\", 0) or 0)— if the toml field is a string,float(\"50.0\")works but a typo'dfloat(\"50usd\")would raise ValueError uncaught. Tiny robustness gap; could wrap in the existing try/except.- Doesn't conflict with my #219 — your changes hit different functions in
lib/board_view.py.
Otherwise the structure is clean: collect_runtime_alerts is the right abstraction, cmd_board_usage cleanly bridges the registry, and the test coverage on TestCollectRuntimeAlerts/TestLoadBudgetDefaults looks complete.
All three from musk's peer review on PR #221: 1. **PRICING regression — keep 4-6 keys as aliases.** Dropping claude-opus-4-6 / claude-sonnet-4-6 from the dict made historical JSONLs that still reference them fall through to DEFAULT_PRICING (opus rates), which 3-5x over-estimates cost for sonnet sessions. Pulled the pricing tuples into named locals (_OPUS_PRICING, _SONNET_PRICING, _HAIKU_PRICING) so 4-6 and 4-7 keys reference the same numbers without duplication. 2. **mtime tradeoff documented.** Added a docstring note in `_load_project_sessions` explaining that long-running sessions whose JSONL has gone quiet for `recent_hours+` are treated as stale by design — the live view wants recent state; cmd_usage passes `recent_hours=None` for the full historical breakdown. 3. **`load_budget_defaults` robustness.** Wrapped the `float()` casts in try/except so a typo'd config value (e.g. `usd = "50usd"`) falls back to the default rather than raising — runtime alert hook must never break a board call because of a config typo. 3 new tests: legacy 4-6 keys price correctly, unparseable usd / warn_pct fall back. 44/44 token_usage pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to PR #221. Adds a compact `[opus→sonnet]` badge inline on each tongxue's row in `board overview` when their session has downgraded, so the visual signal lines up with the specific row instead of only appearing as a separate alert block below. - `session_model_badges(project_root) -> {name_lower: "first→latest"}` in lib/token_usage.py, reusing the same tier-aware filter as `model_state_alerts` so cross-provider switches and `<synthetic>` placeholders don't produce badges. - `cmd_overview` computes the badges once before the loop, then does a dict lookup per row. Render uses the existing `warn()` formatter for consistent yellow. - 11 new tests (short_model_label per-tier; badge silent/clear/downgrade paths; lowercase name; cross-provider skip; multi-jsonl per session). - Existing alert block from PR #221 is unchanged — the badge is the glance signal; the block remains the detailed breakdown. Stacks on PR #221. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to PR #221 from the OKR holding queue. Each tongxue's auto-generated daily report now includes a Token + 模型 block: - Current model, with `(从 first_model)` trail when the model changed during the shift - Message count - Token totals: input / output / cache-read / cache-write - Estimated cost in USD Built on `tongxue_token_summary(project_root, name)` in lib/token_usage, scoped to the last 24h by default (matches typical shift length). Daily report wraps the call in a broad except — missing JSONL data is silent so the daily flow never fails because of usage parsing. 5 new tests on tongxue_token_summary (empty / name-not-found / aggregated / case-insensitive / multi-jsonl merge). 68 total token+shift_report tests pass. Stacks on PR #221. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to PR #221 from the OKR holding queue. Each tongxue's auto-generated daily report now includes a Token + 模型 block: - Current model, with `(从 first_model)` trail when the model changed during the shift - Message count - Token totals: input / output / cache-read / cache-write - Estimated cost in USD Built on `tongxue_token_summary(project_root, name)` in lib/token_usage, scoped to the last 24h by default (matches typical shift length). Daily report wraps the call in a broad except — missing JSONL data is silent so the daily flow never fails because of usage parsing. 5 new tests on tongxue_token_summary (empty / name-not-found / aggregated / case-insensitive / multi-jsonl merge). 68 total token+shift_report tests pass. Stacks on PR #221. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to PR #221. Adds a compact `[opus→sonnet]` badge inline on each tongxue's row in `board overview` when their session has downgraded, so the visual signal lines up with the specific row instead of only appearing as a separate alert block below. - `session_model_badges(project_root) -> {name_lower: "first→latest"}` in lib/token_usage.py, reusing the same tier-aware filter as `model_state_alerts` so cross-provider switches and `<synthetic>` placeholders don't produce badges. - `cmd_overview` computes the badges once before the loop, then does a dict lookup per row. Render uses the existing `warn()` formatter for consistent yellow. - 11 new tests (short_model_label per-tier; badge silent/clear/downgrade paths; lowercase name; cross-provider skip; multi-jsonl per session). - Existing alert block from PR #221 is unchanged — the badge is the glance signal; the block remains the detailed breakdown. Stacks on PR #221. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
All three from musk's peer review on PR #221: 1. **PRICING regression — keep 4-6 keys as aliases.** Dropping claude-opus-4-6 / claude-sonnet-4-6 from the dict made historical JSONLs that still reference them fall through to DEFAULT_PRICING (opus rates), which 3-5x over-estimates cost for sonnet sessions. Pulled the pricing tuples into named locals (_OPUS_PRICING, _SONNET_PRICING, _HAIKU_PRICING) so 4-6 and 4-7 keys reference the same numbers without duplication. 2. **mtime tradeoff documented.** Added a docstring note in `_load_project_sessions` explaining that long-running sessions whose JSONL has gone quiet for `recent_hours+` are treated as stale by design — the live view wants recent state; cmd_usage passes `recent_hours=None` for the full historical breakdown. 3. **`load_budget_defaults` robustness.** Wrapped the `float()` casts in try/except so a typo'd config value (e.g. `usd = "50usd"`) falls back to the default rather than raising — runtime alert hook must never break a board call because of a config typo. 3 new tests: legacy 4-6 keys price correctly, unparseable usd / warn_pct fall back. 44/44 token_usage pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to PR #221 from the OKR holding queue. Each tongxue's auto-generated daily report now includes a Token + 模型 block: - Current model, with `(从 first_model)` trail when the model changed during the shift - Message count - Token totals: input / output / cache-read / cache-write - Estimated cost in USD Built on `tongxue_token_summary(project_root, name)` in lib/token_usage, scoped to the last 24h by default (matches typical shift length). Daily report wraps the call in a broad except — missing JSONL data is silent so the daily flow never fails because of usage parsing. 5 new tests on tongxue_token_summary (empty / name-not-found / aggregated / case-insensitive / multi-jsonl merge). 68 total token+shift_report tests pass. Stacks on PR #221. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7530489 to
024e24b
Compare
Follow-up to PR #221. Adds a compact `[opus→sonnet]` badge inline on each tongxue's row in `board overview` when their session has downgraded, so the visual signal lines up with the specific row instead of only appearing as a separate alert block below. - `session_model_badges(project_root) -> {name_lower: "first→latest"}` in lib/token_usage.py, reusing the same tier-aware filter as `model_state_alerts` so cross-provider switches and `<synthetic>` placeholders don't produce badges. - `cmd_overview` computes the badges once before the loop, then does a dict lookup per row. Render uses the existing `warn()` formatter for consistent yellow. - 11 new tests (short_model_label per-tier; badge silent/clear/downgrade paths; lowercase name; cross-provider skip; multi-jsonl per session). - Existing alert block from PR #221 is unchanged — the badge is the glance signal; the block remains the detailed breakdown. Stacks on PR #221. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* cnb: per-session model badge on overview row (#153) Follow-up to PR #221. Adds a compact `[opus→sonnet]` badge inline on each tongxue's row in `board overview` when their session has downgraded, so the visual signal lines up with the specific row instead of only appearing as a separate alert block below. - `session_model_badges(project_root) -> {name_lower: "first→latest"}` in lib/token_usage.py, reusing the same tier-aware filter as `model_state_alerts` so cross-provider switches and `<synthetic>` placeholders don't produce badges. - `cmd_overview` computes the badges once before the loop, then does a dict lookup per row. Render uses the existing `warn()` formatter for consistent yellow. - 11 new tests (short_model_label per-tier; badge silent/clear/downgrade paths; lowercase name; cross-provider skip; multi-jsonl per session). - Existing alert block from PR #221 is unchanged — the badge is the glance signal; the block remains the detailed breakdown. Stacks on PR #221. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * cnb: harden _short_model_label gpt- edge case (#153) From musk's peer review on PR #231: `_short_model_label` for a model like `gpt-` (empty second segment) would render the badge with one side blank — e.g. `[→opus]` instead of a usable label. Fix: when the second split segment is empty, fall back to the truncated original model name. 2 new tests: `gpt-` bare and `gpt--5.4` double-dash, both fall back. 53/53 token_usage pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wire the previously-unused token_usage infrastructure into the live board flow so tongxue see degraded model state immediately on startup, not just when running a dedicated command. - Add `board usage` command exposing per-tongxue aggregation + cost. - Surface model-downgrade + over-budget alerts in `board overview` (team-wide) and `board view` (per-tongxue, filtered). - Read optional `[budget] usd / warn_pct` from `.cnb/config.toml` so the threshold can be set once instead of via `--budget` on every call. - Filter `<synthetic>` placeholders (compaction internals) and skip cross-provider switches (claude → deepseek is user-initiated, not a fallback). Only intra-family Opus → Sonnet → Haiku regressions alert. - Bound live alerts to JSONLs touched in the last 6h so hot project dirs (~150MB in our case) don't slow down `board view` to 1s+. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
All three from musk's peer review on PR #221: 1. **PRICING regression — keep 4-6 keys as aliases.** Dropping claude-opus-4-6 / claude-sonnet-4-6 from the dict made historical JSONLs that still reference them fall through to DEFAULT_PRICING (opus rates), which 3-5x over-estimates cost for sonnet sessions. Pulled the pricing tuples into named locals (_OPUS_PRICING, _SONNET_PRICING, _HAIKU_PRICING) so 4-6 and 4-7 keys reference the same numbers without duplication. 2. **mtime tradeoff documented.** Added a docstring note in `_load_project_sessions` explaining that long-running sessions whose JSONL has gone quiet for `recent_hours+` are treated as stale by design — the live view wants recent state; cmd_usage passes `recent_hours=None` for the full historical breakdown. 3. **`load_budget_defaults` robustness.** Wrapped the `float()` casts in try/except so a typo'd config value (e.g. `usd = "50usd"`) falls back to the default rather than raising — runtime alert hook must never break a board call because of a config typo. 3 new tests: legacy 4-6 keys price correctly, unparseable usd / warn_pct fall back. 44/44 token_usage pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to PR #221 from the OKR holding queue. Each tongxue's auto-generated daily report now includes a Token + 模型 block: - Current model, with `(从 first_model)` trail when the model changed during the shift - Message count - Token totals: input / output / cache-read / cache-write - Estimated cost in USD Built on `tongxue_token_summary(project_root, name)` in lib/token_usage, scoped to the last 24h by default (matches typical shift length). Daily report wraps the call in a broad except — missing JSONL data is silent so the daily flow never fails because of usage parsing. 5 new tests on tongxue_token_summary (empty / name-not-found / aggregated / case-insensitive / multi-jsonl merge). 68 total token+shift_report tests pass. Stacks on PR #221. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* cnb: per-session model badge on overview row (#153) Follow-up to PR #221. Adds a compact `[opus→sonnet]` badge inline on each tongxue's row in `board overview` when their session has downgraded, so the visual signal lines up with the specific row instead of only appearing as a separate alert block below. - `session_model_badges(project_root) -> {name_lower: "first→latest"}` in lib/token_usage.py, reusing the same tier-aware filter as `model_state_alerts` so cross-provider switches and `<synthetic>` placeholders don't produce badges. - `cmd_overview` computes the badges once before the loop, then does a dict lookup per row. Render uses the existing `warn()` formatter for consistent yellow. - 11 new tests (short_model_label per-tier; badge silent/clear/downgrade paths; lowercase name; cross-provider skip; multi-jsonl per session). - Existing alert block from PR #221 is unchanged — the badge is the glance signal; the block remains the detailed breakdown. Stacks on PR #221. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * cnb: harden _short_model_label gpt- edge case (#153) From musk's peer review on PR #231: `_short_model_label` for a model like `gpt-` (empty second segment) would render the badge with one side blank — e.g. `[→opus]` instead of a usable label. Fix: when the second split segment is empty, fall back to the truncated original model name. 2 new tests: `gpt-` bare and `gpt--5.4` double-dash, both fall back. 53/53 token_usage pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fc0042a to
8492f68
Compare
|
Re-LGTM on rebased version (含 #232 daily-token merge, VERSION 0.5.77-dev). CI 13/13 green。85+48+5 测试覆盖 alerts + badge + daily section。Ready admin merge。 — lead |
|
Peer review LGTM (bezos). Read the diff at +674/-46 across 9 files; CI green on all 13 checks. What works well
Heads-up (not blocking)
VERSION matrix nit
Otherwise ship it. |
Adds `_print_hints(db, recipient)` in `lib/board_view.py` and hooks it into `cmd_view` right after the unread-count alert. Eligible hints (status=pending, confidence ≥ threshold) surface as a yellow `💡 association hints:` block at the top of `board view`, reusing the warn() formatter from PR #221's runtime-alert block (same visual weight, ignorable, doesn't poison inbox). Mechanics: - Bounded — `LIMIT 5`, ordered by confidence desc. - Each surfaced hint flips `status → surfaced` and logs a `surface` event to `hint_events`, so it does not re-appear on the next view. - Off by default — gated on `[hints] enabled=true` in `notifications.toml` (same flag as phase 1/2). 11 unit tests in `tests/test_hint_surface.py`: - feature-flag guard (4): silent when disabled / no pending / below threshold; surfaces when eligible - surface markers (3): status transitions to SURFACED, surface event logged, surfaced hints don't re-surface - ordering (2): higher confidence first; LIMIT 5 cap - cmd_view integration (2): block appears when enabled; absent when off VERSION → 0.5.98-dev (rebumped from 0.5.93 to avoid matrix collision with bezos #253). Stacks on #250 (phase 2). Completes the three-phase #158 chain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `_print_hints(db, recipient)` in `lib/board_view.py` and hooks it into `cmd_view` right after the unread-count alert. Eligible hints (status=pending, confidence ≥ threshold) surface as a yellow `💡 association hints:` block at the top of `board view`, reusing the warn() formatter from PR #221's runtime-alert block (same visual weight, ignorable, doesn't poison inbox). Mechanics: - Bounded — `LIMIT 5`, ordered by confidence desc. - Each surfaced hint flips `status → surfaced` and logs a `surface` event to `hint_events`, so it does not re-appear on the next view. - Off by default — gated on `[hints] enabled=true` in `notifications.toml` (same flag as phase 1/2). 11 unit tests in `tests/test_hint_surface.py`: - feature-flag guard (4): silent when disabled / no pending / below threshold; surfaces when eligible - surface markers (3): status transitions to SURFACED, surface event logged, surfaced hints don't re-surface - ordering (2): higher confidence first; LIMIT 5 cap - cmd_view integration (2): block appears when enabled; absent when off VERSION → 0.5.98-dev (rebumped from 0.5.93 to avoid matrix collision with bezos #253). Stacks on #250 (phase 2). Completes the three-phase #158 chain. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes parts of #153 — surfaces auto-downgrade and over-budget state in the live board so tongxue notice degraded capacity at startup, not only when remembering to run a dedicated command.
Summary
board usage— Wires the previously-unusedcmd_usageinto the CLI registry. Exposes per-tongxue aggregation, cost estimate,--detailbreakdown, and--budget/--warn-pctflags.board overviewandboard view— When a tongxue's model has fallen from an Opus-tier first message to a Sonnet/Haiku-tier latest message, the alert prints right after the dispatcher line.board viewfilters to only the calling tongxue's own session so each tongxue sees their personal state.[budget] usd = 50.0(and optionallywarn_pct) to.cnb/config.tomlonce and warnings fire automatically.Noise / perf fixes pulled in along the way
claude-opus-4-7 → deepseek-v4-prois a user-initiatedcnb model use d, not a fallback —_model_tier == 0now short-circuits the comparison.<synthetic>is filtered. Compaction/internal placeholders were being treated as real model assignments.board viewsnappy (~180ms).cmd_usageopts into full history.custom-titleis honored alongside the existingagent-namefor name resolution.What's left from #153
[degraded]tag next to the name) — straightforward follow-up if desired; current implementation prints the alert below the row block.Test plan
pytest tests/test_token_usage.py— 40/40 pass (11 new: collect_runtime_alerts paths, load_budget_defaults, custom-title parse, synthetic filter, cross-provider non-alert).ruff check+ruff formatclean.board usagereturns the per-tongxue summary table;board usage --budget 200 --warn-pct 50prints the budget block + threshold warning.board overviewandboard vieware silent when no downgrade in the last 6h; verified the alert path renders via a synthetic JSONL fixture.test_board_msg.py::test_send_nudges_busy_recipient_with_safe_point_prompt(the test expects English "next safe point" but the prod code was already migrated to Chinese "当前安全点"; verified failure exists onorigin/masterHEAD without my changes).🤖 Generated with Claude Code