Skip to content

[code_review] Do not use new examples in the experimental environment #5002

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 8, 2025

Conversation

suhaibmujahid
Copy link
Member

@suhaibmujahid suhaibmujahid commented May 7, 2025

Resolves #4995

@suhaibmujahid
Copy link
Member Author

The evaluation results:

--------------------
Variant Name: exclude-new
--------------------
New Comments: 330
New Valid Comments: 44
New Invalid Comments: 94
New Unevaluated Comments: 191
--------------------
Old Comments: 273
Old Valid Comments: 83
Old Invalid Comments: 188
--------------------
Recalled comments: 52.74725274725275
Recalled valid comments: 53.01204819277109
Recalled invalid comments: 53.191489361702125
--------------------
Missed valid comments: 46.98795180722892
Missed invalid comments: 46.808510638297875

This can be compared to #4984 (comment). It seems that the results did not improve as much as was shown in the approved examples PR.

@suhaibmujahid suhaibmujahid marked this pull request as ready for review May 7, 2025 18:26
@suhaibmujahid suhaibmujahid requested review from marco-c and Copilot May 7, 2025 18:26
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR removes the use of new examples in the experimental environment by adding an optional time filter for generated comment examples and issuing a warning when running in an experimental environment.

  • Introduces a new attribute (must_range) to the QueryFilter for handling range queries.
  • Adds an experimental environment flag that filters examples based on patch creation date and prints a warning.
  • Updates several function signatures to accept an optional created_before parameter for filtering comment examples.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
bugbug/vectordb.py Adds the must_range filter support in QueryFilter.
bugbug/tools/code_review.py Introduces experimental environment handling and updates function signatures to include an optional created_before parameter.

marco-c
marco-c previously approved these changes May 7, 2025
Copy link
Collaborator

@marco-c marco-c left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a pity the improvement is gone :(

@suhaibmujahid suhaibmujahid merged commit 5898b7d into mozilla:master May 8, 2025
6 checks passed
@suhaibmujahid suhaibmujahid deleted the exclude-new branch May 8, 2025 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[code_review] Exclude approved examples sourced from the same revision
2 participants