Skip to content

[code_review] Add approved generated comments as examples in the prompt #4984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
May 5, 2025

Conversation

suhaibmujahid
Copy link
Member

Resolves #4320

@suhaibmujahid suhaibmujahid force-pushed the approved-generated branch 3 times, most recently from cc258af to 958a137 Compare April 24, 2025 18:00
@suhaibmujahid
Copy link
Member Author

Here is the report:

--------------------
Variant Name: approved-examples
--------------------
New Comments: 295
New Valid Comments: 50
New Invalid Comments: 86
New Unevaluated Comments: 158
--------------------
Old Comments: 274
Old Valid Comments: 83
Old Invalid Comments: 189
--------------------
Recalled comments: 52.55474452554745
Recalled valid comments: 60.24096385542169
Recalled invalid comments: 49.735449735449734
--------------------
Missed valid comments: 39.75903614457831
Missed invalid comments: 50.264550264550266

This can be compared to #4966 (comment):

--------------------
Variant Name: llm-gpt-4.1
--------------------
New Comments: 316
New Valid Comments: 42
New Invalid Comments: 84
New Unevaluated Comments: 190
--------------------
Old Comments: 272
Old Valid Comments: 83
Old Invalid Comments: 189
--------------------
Recalled comments: 51.10294117647059
Recalled valid comments: 50.602409638554214
Recalled invalid comments: 51.32275132275132
--------------------
Missed valid comments: 49.39759036144578
Missed invalid comments: 48.67724867724868

The numbers look better overall. However, there is a chance of bias since the examples could come from revisions in the evaluation dataset.

@suhaibmujahid suhaibmujahid requested a review from marco-c April 25, 2025 16:55
@suhaibmujahid suhaibmujahid marked this pull request as ready for review April 25, 2025 16:55
marco-c
marco-c previously approved these changes Apr 29, 2025
Co-authored-by: Marco Castelluccio <mcastelluccio@mozilla.com>
@suhaibmujahid suhaibmujahid merged commit f6fd3ba into mozilla:master May 5, 2025
6 checks passed
@suhaibmujahid suhaibmujahid deleted the approved-generated branch May 5, 2025 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[code_review] Use approved generated comments as examples
2 participants