feat(benchmark): expose vote aggregation on leaderboards - #816
Conversation
Whether a leaderboard binarizes its responses was only reachable through a raw PATCH against the API — it is absent from the dashboard UI and was never surfaced on the SDK, so every SDK-created leaderboard silently inherited the server default of AllVotes. With AllVotes each individual response counts as its own matchup, so a matchup answered 5 times outweighs one answered twice and within-matchup annotator noise flows straight into the standings. Adds a public `VoteAggregation` enum and threads it through `create_leaderboard`, defaulting to `MAJORITY_VOTE`: each matchup collapses to a single win for the majority side, ties split 0.5/0.5, so every matchup weighs the same. `RapidataLeaderboard.vote_aggregation` reads it back and can change it in place. Standings are derived from the raw votes on every read (the binarization lives in the leaderboard service's ClickHouse scoreboard query, keyed by rapidId), so switching the aggregation also re-counts already-collected responses. That is why this is a settable property rather than create-only like the prompt-tag filters. The value resolves lazily: `QueryLeaderboardsByBenchmarkEndpoint` does not carry `voteAggregation` in its output — unlike the sibling list, get-by-id, and create endpoints — so a leaderboard read from `benchmark.leaderboards` fetches it once on first access instead of guessing the default. An unresolved value is omitted from the update payload so an unrelated setter cannot rewrite it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: lino <68745352+LinoGiger@users.noreply.github.com>
| @vote_aggregation.setter | ||
| def vote_aggregation(self, vote_aggregation: VoteAggregation): | ||
| """ | ||
| Sets how the responses on a single matchup are aggregated. | ||
|
|
||
| Standings are derived from the raw responses on every read, so switching this | ||
| also changes how already-collected responses are counted — no re-evaluation | ||
| needed. | ||
| """ | ||
| with tracer.start_as_current_span( | ||
| "RapidataLeaderboard.vote_aggregation.setter" | ||
| ): | ||
| if not isinstance(vote_aggregation, VoteAggregation): | ||
| raise ValueError( | ||
| "Vote aggregation must be one of: " | ||
| + ", ".join( | ||
| f"VoteAggregation.{member.name}" for member in VoteAggregation | ||
| ) | ||
| ) | ||
|
|
||
| logger.debug( | ||
| f"Setting vote aggregation to {vote_aggregation.name} for leaderboard {self.name}" | ||
| ) | ||
| self.__vote_aggregation = vote_aggregation | ||
| self._update_config() |
There was a problem hiding this comment.
i feel like these setters are overused, can you get rid of them in this entire class and just expose a patch endpoint?
There was a problem hiding this comment.
Done — all four setters are gone (name, level_of_detail, min_responses_per_matchup, vote_aggregation), replaced by a single update() in 1846f99:
leaderboard.update(
name="Realism v2",
level_of_detail="high",
min_responses_per_matchup=5,
vote_aggregation=VoteAggregation.MAJORITY_VOTE,
)It only sends the arguments you actually pass, which fixes a real bug the old setters had: each one built a full PATCH from the object's current in-memory state, so renaming a leaderboard also resent its response budget and min-responses, and a stale field could overwrite a value changed elsewhere. Now an omitted field is left alone by patch semantics, and several changes go out in one request. Local state is only updated after the request succeeds.
Also removed _custom_config, which was an unused second path to the same PATCH.
The properties are read-only now, so this is a breaking change for anyone assigning to them — flagged as BREAKING CHANGE: in the commit. Added tests/rapidata_client/benchmark/test_leaderboard_update.py covering the per-field validation, the only-named-fields behaviour, and that the properties reject assignment.
| ### Vote Aggregation | ||
|
|
||
| Each matchup — one comparison of two models on one prompt — collects several | ||
| responses (see `min_responses_per_matchup`). `vote_aggregation` decides how those | ||
| responses become that matchup's result: | ||
|
|
||
| | `VoteAggregation` | Effect | | ||
| |---|---| | ||
| | `MAJORITY_VOTE` (default) | The matchup collapses to a single win for the side the majority picked, ties split 0.5/0.5. Every matchup weighs the same. | | ||
| | `ALL_VOTES` | Every individual response counts as its own matchup, so a heavily-answered matchup outweighs a lightly-answered one. | | ||
|
|
||
| `MAJORITY_VOTE` is what you want in almost every case: it removes annotator noise | ||
| within a matchup and keeps the standings from being skewed by uneven response | ||
| counts across matchups. | ||
|
|
||
| ```python | ||
| from rapidata import VoteAggregation | ||
|
|
||
| leaderboard = benchmark.create_leaderboard( | ||
| name="Realism", | ||
| instruction="Which image is more realistic?", | ||
| vote_aggregation=VoteAggregation.ALL_VOTES, # opt out of majority collapsing | ||
| ) | ||
| ``` | ||
|
|
||
| Standings are derived from the raw responses on every read, so switching the | ||
| aggregation on an existing leaderboard also changes how its already-collected | ||
| responses are counted — no re-evaluation needed. | ||
|
|
||
| ```python | ||
| print(leaderboard.vote_aggregation) # VoteAggregation.ALL_VOTES | ||
| leaderboard.vote_aggregation = VoteAggregation.MAJORITY_VOTE | ||
| ``` | ||
|
|
There was a problem hiding this comment.
please don't make this sound so negatively. i think it's fine if for now we just mention that the matchup winner is chosen by majority vote without going into too much detail
There was a problem hiding this comment.
Toned down in d474c7b. Dropped the comparison table and the "this is what you want in almost every case" framing; it now just states that the matchup winner is the side the majority picked (ties split 0.5/0.5) and shows how to opt out:
Each matchup — one comparison of two models on one prompt — collects several responses. By default the matchup's winner is the side the majority of those responses picked, with ties split 0.5/0.5.
Pass
vote_aggregation=VoteAggregation.ALL_VOTESto count every individual response as its own matchup instead.
I kept the note that switching it re-counts already-collected responses, since that is genuinely surprising and easy to get wrong — happy to cut it too if you think it is more detail than the page needs.
Every mutable leaderboard setting went through its own property setter, each one issuing a full PATCH built from the object's current state. That made a rename resend the response budget, and it meant a stale in-memory field could be written back over a value someone else had changed. `update()` takes all four settings as optional keyword arguments and sends only the ones the caller named, so an omitted field is left alone by patch semantics rather than overwritten with a guess. Several changes now go out in one request. Local state is refreshed only after the request succeeds. Removes `_custom_config`, which was an unused second path to the same PATCH. BREAKING CHANGE: `name`, `level_of_detail`, `min_responses_per_matchup` and `vote_aggregation` are read-only properties now. Use `leaderboard.update(<field>=...)` instead of assigning to them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: lino <68745352+LinoGiger@users.noreply.github.com>
State plainly that the matchup winner is chosen by majority vote and how to opt out, instead of editorialising about what the alternative does to the standings. Switches the mutation examples over to update(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: lino <68745352+LinoGiger@users.noreply.github.com>
2 participants
Why
Whether a leaderboard binarizes its responses was only reachable through a raw
PATCH /leaderboard/{id}— it has no dashboard UI and was never surfaced on the SDK. So every SDK-created leaderboard silently inherited the server default ofAllVotes, where each individual response counts as its own matchup: a matchup answered 5 times outweighs one answered twice, and within-matchup annotator noise flows straight into the standings.This came out of a real board (
rlb_1RNU9vh78AqXX1on the Benchmark.ai Image Benchmark) sitting onAllVoteswhile its three sibling leaderboards were all onMajorityVote. At 5 responses per matchup, binarizing moved the top participants' win rates by 3–5 points.What
VoteAggregationenum (MAJORITY_VOTE/ALL_VOTES), exported fromrapidataandrapidata.types, following the existingGender/AgeGroupwrapper-enum pattern.RapidataBenchmark.create_leaderboard(..., vote_aggregation=VoteAggregation.MAJORITY_VOTE)— majority is now the SDK default, so new leaderboards binarize unless the caller opts out.RapidataLeaderboard.vote_aggregationreads it back and can change it in place.mri_advanced.md.Notes for the reviewer
The default changes behaviour for existing SDK users. Leaderboards created through
create_leaderboardafter this lands binarize where they previously did not. Existing leaderboards are untouched.Switching the aggregation is retroactive. Standings are derived from the raw votes on every read — the binarization lives in the leaderboard service's ClickHouse scoreboard query, grouped by
rapidId— so changing it also re-counts already-collected responses. That's why this is a settable property rather than create-only like the prompt-tag filters. (Note: the backend'sUpdateLeaderboardEndpointXML doc claims "Only affects future runs", which is wrong and ships in the published OpenAPI spec. Worth a separate backend fix.)The value resolves lazily.
QueryLeaderboardsByBenchmarkEndpointdoes not carryvoteAggregationin its output — unlike the siblingQueryLeaderboardsEndpoint, get-by-id, and create endpoints — so a leaderboard obtained frombenchmark.leaderboardsfetches it once on first access rather than guessing the default. An unresolved value is omitted from the update payload so an unrelated setter (name,level_of_detail) can't rewrite it. Adding the field to that endpoint'sOutputinrapidata-backendwould remove the extra request; happy to open that PR if you want it.Open question: the backend default is still
AllVotes, so leaderboards created through the API or dashboard don't get this. Should that flip too?Verification
uv run pytest tests/→ 112 passed. The 4 failures intests/rapidata_client/audience/are pre-existing onmain(verified on a clean tree).uv run pyright src/rapidata/rapidata_client→ 0 errors.uv run black src/rapidata/rapidata_client→ clean (unrelated pre-existing reformatting left out of this diff).uv run --group docs mkdocs build→ builds; reference page auto-generates for the new module.🔗 Session: session-86eedabe
🤖 Generated with Claude Code