-
Notifications
You must be signed in to change notification settings - Fork 256
fix custom model example #766
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR updates the custom model example to fix the configuration parameter name and refines the Google Translate example’s iteration and mapping.
- Rename
model
tomodel_name
inCustomModelConfig
instantiation. - Update
google_translate_model.py
to useconfig.model_name
and refactor dataset iteration.
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
File | Description |
---|---|
src/lighteval/main_custom.py | Fixed parameter name passed to CustomModelConfig |
examples/custom_models/google_translate_model.py | Mapped model_name correctly and switched to splits_iterator with nested loops |
Comments suppressed due to low confidence (2)
examples/custom_models/google_translate_model.py:142
- [nitpick] The variable name
r
is ambiguous. Consider renaming it to a more descriptive name such asrequest
orexample
for clarity.
for r in tqdm(split, desc="Batch", position=1, disable=False):
examples/custom_models/google_translate_model.py:139
- Hard-coding
disable=False
overrides the class-leveldisable_tqdm
setting. Consider usingdisable=self.disable_tqdm
to respect the configured flag.
disable=False, # self.disable_tqdm,
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are a few more instances:
examples/custom_models/local_mt_model.py
src/lighteval/models/endpoints/openai_model.py
#768 for convenience |
No description provided.