Repository navigation
The competency bench records whether an answer went through the graph (0061 cut 3, second slice) - #962
Merged
Conversation
ADR 0061 cut 3, second slice. The first slice's 11 of 11 on the bench base said nothing about the ontology: chat can answer from document text alone, and the expected-answer judge cannot tell that apart from an answer read off the typed graph (the ADR's open question). The bench now reads the chat turn's step events: facts, neighbors, timeline and paths are graph tools, search and document read text. Each result carries `via` (graph, text, both, none) and the steps it saw; the report and the workbench add two numbers beside "answered": answered after fetching graph facts, and answered without reading text at all. The second is the ontology's own number. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
The server writes the kind of that step as "path", so the bench's "paths" never matched and an answer reached only through paths counted as none. The workbench chip now words via in the reader's language instead of showing the stored token. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ADR 0061 cut 3, second slice. #955's 11 of 11 on the bench base said nothing about the ontology: chat can answer from document text alone, and the expected-answer judge cannot tell that apart from an answer read off the typed graph (the ADR's open question).
What lands
competency.mjsreads the chat turn's step events:facts,neighbors,timeline,pathsare graph tools,searchanddocumentread text. Each result carriesvia(graph / text / both / none) and the steps it saw;POST /kbs/{id}/questions/{qid}/resultacceptsvia.Measured on the bench base (same eleven questions, asked again)
The one miss is instructive: "Where did Allen Francis Moore die?" was expected "Monticello Cemetery" (the typed
placeOfDeathfact), but chat read the text and answered San Antonio, with Monticello as the burial place. The expected answer was copied from a wrong typed fact; the bench caught the graph, not the chat. The shape-only question (houses) counts as answered by shape but was answered via text: the shape check says the ontology could answer it, not that it did.Not changed: the judge, the questions, chat.
🤖 Generated with Claude Code