Allow for model kwargs when loading transformers from pretrained #754
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Pull Request Overview
This PR enhances the TransformersModelConfig to accept arbitrary keyword arguments when loading a pretrained Hugging Face model, removes the hardcoded generation_size setting, and updates the internal loader to pass through these custom kwargs.
- Introduce
model_loading_kwargsin the config - Remove the
generation_sizefield - Update
_create_auto_modelto merge and passmodel_loading_kwargsinstead of a localkwargsdict
Comments suppressed due to low confidence (1)
src/lighteval/models/transformers/transformers_model.py:139
- Removing the
generation_sizefield is a breaking change for users. Consider deprecating it first or updating documentation and release notes to guide consumers through the migration.
generation_size: PositiveInt = 256
| if "quantization_config" not in pretrained_config.to_dict(): | ||
| kwargs["quantization_config"] = quantization_config | ||
| self.config.model_loading_kwargs["quantization_config"] = quantization_config |
There was a problem hiding this comment.
Mutating config.model_loading_kwargs in place can lead to unexpected state if _create_auto_model is called multiple times. Consider merging into a local dict and passing that to from_pretrained.
Copilot uses AI. Check for mistakes.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull Request Overview
This PR introduces support for passing custom keyword arguments when loading pretrained transformer models, enabling more flexible configuration of model loading. It also replaces the fixed "generation_size" parameter with a more general "model_loading_kwargs" field.
- Removed the fixed generation_size parameter.
- Added a new model_loading_kwargs field to the configuration.
- Updated the auto model creation to copy the provided kwargs.
Comments suppressed due to low confidence (1)
src/lighteval/models/transformers/transformers_model.py:139
- The removal of the fixed generation_size parameter may be a breaking change. If this change is intentional, please update the documentation to clarify the impact on users and ensure that all dependent functionalities are adjusted accordingly.
- generation_size: PositiveInt = 256
## Pull Request Overview This PR introduces support for passing custom keyword arguments when loading pretrained transformer models, enabling more flexible configuration of model loading. It also replaces the fixed "generation_size" parameter with a more general "model_loading_kwargs" field. - Removed the fixed generation_size parameter. - Added a new model_loading_kwargs field to the configuration. - Updated the auto model creation to copy the provided kwargs. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * suggestion from copilot --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
No description provided.