Skip to content

Record why a subscription reading was refused, and read the outcome from the last word - #169

Merged
drevendev merged 1 commit into
masterfrom
policy/168-telemetry-diagnostics
Sep 5, 2026
Merged

drevendev merged 1 commit into
masterfrom
policy/168-telemetry-diagnostics

Conversation

@drevendev

Copy link
Copy Markdown
Owner

Closes #168

Achieved outcome

A refused subscription reading now records the API's own error type in its reason (HTTP 403 permission_error) and keeps the error message in the reading, which reaches the private ledger only; the public warning prints the reason alone, and the token appears in neither. The outcome of a run is classified from its final message, and a handoff or a verdict marker there means completed whatever else the message mentions; blocker markers are consulted only after that.

Tested revision

d8a41a5

Changed artifacts

  • scripts/subscription_usage.py — error_detail() reads at most 64 KB of a refusal body and returns error.type and a 200-character error.message; unavailable() and the success reading carry an error_message field; the HTTP branch of read() uses both.
  • scripts/record_usage.py — COMPLETED markers (handoff, verdict, a named pull request); final_text() returns the result text or the last assistant text; classify_outcome() checks COMPLETED before BLOCKED; main() classifies from final_text() while mentions and rework still read the whole transcript.
  • scripts/tests/test_subscription_usage.py — refusal with a JSON body, without a JSON body, with no body, and with an overlong message; the token never appears in a reading.
  • scripts/tests/test_record_usage.py — the Implement buildInitialWorld() 17-step initialization (REQ-CONFIG-003) #163 case (handoff plus a blocked mention is completed), acceptor verdicts, a blocker heading without a handoff, and final_text() in three shapes.

Acceptance criteria

  • A 403 whose body names permission_error is recorded with reason HTTP 403 permission_error and the message in the reading; a body that is not JSON leaves reason HTTP 403 and no message; the token appears nowhere in either — test_a_refusal_records_the_status_and_the_error_type_and_keeps_the_message, test_a_refusal_without_a_json_body_records_the_status_code_only, test_a_refusal_with_no_body_at_all_is_still_recorded.
  • A final message carrying ## AUTHOR handoff and a pull request number classifies as completed even if it also says something is blocked; a final message carrying ## AUTHOR blocker classifies as blocked — test_a_handoff_is_completed_whatever_else_it_mentions, test_a_blocker_heading_without_a_handoff_is_still_blocked.
  • python -m unittest discover -s scripts/tests passes — 275 tests at d8a41a5.

Checks

Check Outcome Evidence
python -m unittest discover -s scripts/tests passed 275 tests, OK, at d8a41a5
python scripts/policy_guard.py --base origin/master passed policy paths only, no violation
dotnet build --configuration Release / dotnet test not_run no product code changed; the required check runs it on this pull request. Risk left open: none for this diff
npm run typecheck && npm test && npm run build not_run same

Not checked

The live 403 body. The first run after merge records it; the reason then names the error type and the ledger holds the message, which is what decides the next step on the subscription figure.

Assumptions and unknowns

  • Anthropic error bodies are JSON of the shape {"type": "error", "error": {"type": ..., "message": ...}}. Any other shape degrades to the status code alone.
  • The error message never contains the token. It is capped at 200 characters and stored in the private ledger only, so even a surprising message stays off the public log.
  • COMPLETED is a heuristic like the rest of the classifier and is marked heuristic in the record. A run that opens a pull request and then reports a blocker in the same final message is recorded as completed; that is the intended reading — a pull request exists.

Highest-risk area for review

COMPLETED in record_usage.py: too broad a pattern would hide genuinely blocked runs behind any mention of a pull request. It is anchored to headings for the role markers and requires a number for the pull request forms; OutcomeTests carries the negative controls.

Remaining gate

None mandatory. Narrowing in substance: nothing new is printed except an error type, and no credential enters any workflow.

…rom the last word

Every reading since the ledger gained the subscription block has been HTTP 403 and
nothing more, which cannot say whether the token lacks a scope or the endpoint moved.
The refusal body now contributes its error type to the reason and its message to the
private reading. The outcome classifier reads the run's final message, where a handoff
or a verdict means completed; the first live record had called a run that opened #163
blocked because its transcript discussed an Issue that was. Refs #168.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@drevendev drevendev added type:process Development workflow, governance, or coordination change area:tooling CI, scripts, guards, developer tooling policy Control-plane change: workflows, scripts, runbooks, AGENTS.md. Operator-owned, never AUTHOR. labels Sep 5, 2026
@drevendev
drevendev merged commit 9d9618c into master Sep 5, 2026
6 checks passed
@drevendev
drevendev deleted the policy/168-telemetry-diagnostics branch September 5, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:tooling CI, scripts, guards, developer tooling policy Control-plane change: workflows, scripts, runbooks, AGENTS.md. Operator-owned, never AUTHOR. type:process Development workflow, governance, or coordination change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Record why a subscription reading was refused, and classify the outcome from the handoff

1 participant