-
Notifications
You must be signed in to change notification settings - Fork 41
Save pass@k result & use custom tokenizer #20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@@ -277,6 +277,12 @@ def stucking_checker(): | |||
if not os.path.isfile(result_path): | |||
with open(result_path, "w") as f: | |||
json.dump(results, f, indent=2) | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe adding if not os.path.isfile(pass_at_k_path):
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, I think a better way is to check if at least the Pass@1 scores are the same and decide whether we need to rewrite the result_path and pass_at_k_path. Wdyt?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, sounds good
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! I'll merge the PR after your update.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
@marianna13 LGTM, only minor updates are needed. Did you test w/ these changes with some models? |
I tried with |
Hey there!
Two changes that would be nice to have: