Repository navigation
A bench for time words: what each sentence of a written document should be dated (0064 cut 1) - #990
Merged
Merged
Conversation
…ld be dated ADR 0064 cut 1, no behaviour change. `scripts/bench/timewords.mjs` uploads five documents written for the purpose (three Chinese, two English, 31 sentences whose time is known) into a fresh base per run and checks each sentence: was it extracted, was its time expression recorded as a time word, does its start fall in the expected interval, was a sentence with no time words given a start, is its as-of right, and does the outcome repeat across runs. Baseline on dev, three runs: time expressions recorded 16 of 21; starts right 4 of 4 absolute, 7 of 11 relative to the text's now, 3 of 3 relative to a dated event, 0 of 2 under a section heading; no sentence without time words was given a start; as-of 0 of 10 (the anchor is the processing moment); 28 of 31 sentences the same in every run; 8–10k tokens per chunk. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ADR 0064 cut 1 (the record is #989). No behaviour change.
scripts/bench/timewords.mjsuploads five documents written for the purpose (three Chinese, two English, 31 sentences whose time is known) into a fresh base per run, then checks each sentence: extracted or not, its time expression recorded as a time word or not, its start inside the expected interval, a start given to a sentence that has no time words, its as-of, and whether the outcome repeats across runs. Tokens come from the server log.Baseline (dev d7756a4, gemini-3.5-flash, three runs)
28 of 31 sentences have the same outcome in every run. What is lost: "三个月后发布" and "下个月启动" are not recorded as time words (the words end up in the object); "上周" is recorded and not resolved; "in two months" is right one run in three; a quarter on a section heading does not reach the sentences under it. The record's own "today" column (2 of 7 on the weekly report) came from a looser hand check and is corrected by these numbers with cut 2.
🤖 Generated with Claude Code