Repository navigation
The recall bench marks a round that did not extract every chunk and keeps it out of the table - #1014
Merged
Merged
Conversation
…eeps it out of the table A round whose extraction failed chunk by chunk (the endpoint down for the whole round) ends with an empty queue, so the settle wait is satisfied at once and the record carries settled=true with a score of 0. --table then pulls it into the group means. The record now says complete=false when the extracted count is below the chunk count; --table leaves such rounds out of every group and names them under the table. Records from before this change are judged by their per-document counts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1002 and #1005 (#588).
Problem
A round whose extraction failed chunk by chunk, because the endpoint was down or refused payment for the whole round, ends with an empty queue. The settle wait is satisfied at once, and the record carries
settled: truewith a score of 0/52 and a handful of chunks.--tablethen pulls it into the group means: on the 2026-09-29 measurement the shown group read 38.0 ± 16.5 instead of 47.3 ± 0.6 until the two void records were moved out by hand.Change
complete; a round below the count says so on the console.--tableleaves incomplete rounds out of every group and names them under the table with their chunk counts. Records from before this change have nocompletefield and are judged by their per-documentextracted/chunks.Verified
node --checkpasses.--tableon the six valid records of the The chunks of one document are extracted one at a time #588 measurement plus the four void ones: the four are named and left out, and the means are the ones posted on The chunks of one document are extracted one at a time #588 (47.3 ± 0.6 in both groups, overlap 48.5 / 46.9 within and 34.3 across).🤖 Generated with Claude Code