Skip to content

fix custom model example #766

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 21, 2025
Merged

fix custom model example #766

merged 1 commit into from
May 21, 2025

Conversation

NathanHB
Copy link
Member

No description provided.

@NathanHB NathanHB self-assigned this May 21, 2025
@NathanHB NathanHB added bug Something isn't working ignore-for-release will be ignored when doing release note labels May 21, 2025
@NathanHB NathanHB linked an issue May 21, 2025 that may be closed by this pull request
@HuggingFaceDocBuilderDev
Copy link
Collaborator

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@NathanHB NathanHB requested a review from Copilot May 21, 2025 12:34
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@NathanHB NathanHB requested a review from Copilot May 21, 2025 14:15
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the custom model example to fix the configuration parameter name and refines the Google Translate example’s iteration and mapping.

  • Rename model to model_name in CustomModelConfig instantiation.
  • Update google_translate_model.py to use config.model_name and refactor dataset iteration.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/lighteval/main_custom.py Fixed parameter name passed to CustomModelConfig
examples/custom_models/google_translate_model.py Mapped model_name correctly and switched to splits_iterator with nested loops
Comments suppressed due to low confidence (2)

examples/custom_models/google_translate_model.py:142

  • [nitpick] The variable name r is ambiguous. Consider renaming it to a more descriptive name such as request or example for clarity.
for r in tqdm(split, desc="Batch", position=1, disable=False):

examples/custom_models/google_translate_model.py:139

  • Hard-coding disable=False overrides the class-level disable_tqdm setting. Consider using disable=self.disable_tqdm to respect the configured flag.
disable=False,  # self.disable_tqdm,

Copy link

@mjpost mjpost left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a few more instances:

  • examples/custom_models/local_mt_model.py
  • src/lighteval/models/endpoints/openai_model.py

@mjpost
Copy link

mjpost commented May 21, 2025

#768 for convenience

@NathanHB NathanHB merged commit cce0bfc into main May 21, 2025
5 checks passed
hynky1999 pushed a commit that referenced this pull request May 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working ignore-for-release will be ignored when doing release note
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] pydantic throws error with custom evaluator
3 participants