Repository navigation
The usage log says which call it was - #996
Merged
Merged
Conversation
A document of 270 tokens costs 31,000 to process and the log could not say on what: each `llm usage` line had the model and two numbers, and the typed bench attributed them to phases by time window, where votes, rule proposals and readings run interleaved. Each line now carries `call`, the first sentence of the call's system message: those are constants and differ per kind of call, so no caller changes. The field comes last, the scripts that read `model=… prompt=… completion=…` still match. `usageByCall` in the bench library groups a log by it, and the typed bench prints the breakdown per kind of call and per document. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
WaylandYang
added a commit
that referenced
this pull request
Sep 28, 2026
* A document that states no date is not asked for one 0064 decision 1 says the separate call that read a date off the opening goes away. Cut 2 kept it as a fallback for documents where extraction reported no entry. With usage labelled by call (#996), twenty Re-DocRED documents: sixteen took the fallback, each asking the question extraction had just answered with "none" and getting the same answer, about 940 tokens a document. On the time bench every document that states a date had it reported by extraction; the fallback never found one that was missed. The resolver no longer calls the model for the document's date. A document whose extraction reported none has none: a relative expression in it waits at grade C and its statements carry no attestation. A date stored by the old call on a document extracted before cut 2 is still read. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com> * The record names the pull request Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com> --------- Signed-off-by: Wayland Yang <wayland0916@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A 270-token document costs about 31,000 tokens to process, and the server log could not say on what: each
llm usageline carried the model and two numbers. The typed bench attributes them to phases by time window, and inside alignment the two votes, rule proposals and readings run interleaved.Each line now carries
call, the first sentence of the call's system message (truncated to 60 characters, spaces as underscores). System messages are constants and differ per kind of call, so no caller changes. The field comes last in the line; scripts that readmodel=… prompt=… completion=…still match.usageByCallinscripts/bench/lib.mjsgroups a log by it;typed.mjsprints the breakdown per kind of call and per document and stores it in the result.No behaviour change. This is the first step of bringing the cost per document down: measure by kind of call, then cut.
🤖 Generated with Claude Code