feat/3957/disqualify-non-primary-bots

[lsabor]

closes #3957
removes non primary bots from aggregations, scoring, and ranks

Summary by CodeRabbit

• Improvements

  • Enhanced bot forecast filtering in scoring aggregations with differentiated handling of primary vs. non-primary bots, improving ranking accuracy.

• Tests

  • Expanded unit test coverage for ranking and exclusion logic across various bot status scenarios.

[coderabbitai]

📝 Walkthrough

Walkthrough

Changes introduce differential handling for primary versus non-primary bots across scoring and aggregation logic. Non-primary bots are now unconditionally excluded from rankings and aggregations, while primary bots are preserved when applicable. Affects evaluation question scoring, ranking assignments, and forecast aggregations.

Changes

Cohort / File(s)	Summary
Bot filtering in scoring logic scoring/score_math.py, scoring/utils.py	Modified bot-filtering logic to distinguish between primary and non-primary bots. Non-primary bots (author__is_bot=True and author__is_primary_bot=False) are now excluded unconditionally, while primary bots are preserved in base forecasts and ranking calculations.
Forecast aggregation filtering utils/the_math/aggregations.py	Added conditional filter to include forecasts by primary bots when no explicit user IDs are specified. Uses (author__is_bot=False) OR (author__is_primary_bot=True) condition in get_aggregations_at_time and get_aggregation_history.
Expanded test coverage tests/unit/test_scoring/test_utils.py	Added comprehensive test scenarios for ranking and exclusion logic, including MedalExclusionRecord interactions, bot status variations, and exclusion scope coverage (project vs leaderboard levels).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Poem

🐰 Primary bots hop freely, approved and adored,
Non-primary ones must wait beyond the scoring board,
With filters refined and ranks rearranged,
The aggregation logic is thoughtfully changed,
Tests bloom to verify each distinction made!

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat/3957/disqualify-non-primary-bots' directly describes the main change: disqualifying non-primary bots from aggregations, scoring, and ranks.
Linked Issues check	✅ Passed	The pull request implements removal of non-primary bots from aggregations, scoring, and ranks across four files, directly addressing the requirements in issue #3957.
Out of Scope Changes check	✅ Passed	All changes are scoped to implementing non-primary bot restrictions; the test additions in test_utils.py support the core functionality changes without introducing unrelated modifications.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feat/3957/disqualify-non-primary-bots

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

🚀 Preview Environment

Your preview environment is ready!

Resource	Details
🌐 Preview URL	https://metaculus-pr-4269-feat-3957-disqualify-non-prima-preview.mtcl.cc
📦 Docker Image	ghcr.io/metaculus/metaculus:feat-3957-disqualify-non-primary-bots-6884839
🗄️ PostgreSQL	NeonDB branch preview/pr-4269-feat-3957-disqualify-non-prima
⚡ Redis	Fly Redis mtc-redis-pr-4269-feat-3957-disqualify-non-prima

Details

• Commit: 49ccb3fcffca874980b27f84dacc678f440c720d
• Branch: feat/3957/disqualify-non-primary-bots
• Fly App: metaculus-pr-4269-feat-3957-disqualify-non-prima

---

ℹ️ Preview Environment Info

Isolation:

• PostgreSQL and Redis are fully isolated from production
• Each PR gets its own database branch and Redis instance
• Changes pushed to this PR will trigger a new deployment

Limitations:

• Background workers and cron jobs are not deployed in preview environments
• If you need to test background jobs, use Heroku staging environments

Cleanup:

• This preview will be automatically destroyed when the PR is closed

[coderabbitai]

Actionable comments posted: 3

🤖 Fix all issues with AI agents

In `@scoring/score_math.py`:
- Around line 365-369: The inline comment on the if not only_include_user_ids
branch is incorrect: the code excludes non-primary bots
(base_forecasts.exclude(author__is_bot=True, author__is_primary_bot=False)) when
user IDs are NOT specified, so update the comment to accurately describe that
behavior (e.g., "exclude forecasts by non-primary bots when user ids are not
explicitly specified") so it matches the conditional around
only_include_user_ids and the base_forecasts exclusion.

In `@utils/the_math/aggregations.py`:
- Around line 752-756: The comment is misleading: change the comment above the
forecasts.filter(...) call to state that non-primary bot forecasts are excluded
unless user IDs are explicitly specified (i.e., "exclude non-primary bot
forecasts unless user IDs are explicitly specified"); keep the filter logic
using forecasts.filter(Q(author__is_bot=False) | Q(author__is_primary_bot=True))
intact and ensure the corrected comment sits directly above that line so it
accurately documents the behavior.
- Around line 994-998: The comment above the filter is misleading: the code on
forecasts.filter(Q(author__is_bot=False) | Q(author__is_primary_bot=True))
actually includes non-bot authors OR primary bots (i.e., it excludes non-primary
bots), so update the comment to reflect that behavior (or if the intended
behavior was different, change the filter to match the original intent). Refer
to the forecasts variable and the filter call in this function (same pattern as
in get_aggregations_at_time) and either reword the comment to state "include
only non-bot authors or primary bots when user ids are not specified" or adjust
the Q(...) expressions to implement the original "include forecasts by
non-primary bots only when user ids explicitly specified" semantics.