Align EOS token ID between tokenizer and generation config #663
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lewtun
commented
May 27, 2025
| - `grpo.py`: trains a model with GRPO on a given dataset. | ||
| - `sft.py`: performs a simple SFT of a model on a dataset. | ||
| - `evaluate.py`: evaluates a model on the R1 benchmarks. |
There was a problem hiding this comment.
No longer exists since we migrated the evals to lighteval natively
lewtun
commented
May 27, 2025
| @@ -140,6 +140,9 @@ def make_conversation(example, prompt_column: str = script_args.dataset_prompt_c | |||
| # Save model and create model card | |||
| ################################## | |||
| logger.info("*** Save model ***") | |||
| # Align the model's generation config with the tokenizer's eos token | |||
| # to avoid unbounded generation in the transformers `pipeline()` function | |||
| trainer.model.generation_config.eos_token_id = tokenizer.eos_token_id | |||
There was a problem hiding this comment.
Here is the key change to avoid the generation footgun
|
ok checking |
edbeeching
approved these changes
May 27, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses a nasty footgun in
transformers, where a change to the base model's tokenizer must ALSO be propagated to the model's generation config. Without this change, thepipeline()function produces unbounded generations because it relies on the base model's EOS token (e.g.<|endoftext|>) instead of the one we set in the tokenizer config (e.g.<|im_end|>)Note that this has no impact on
vllmsince there the EOS token is inferred from thetokenizer_config.jsonfile.cc @qgallouedec @kashif for viz as this might be necessary to include on the SFT script of TRL too.
I also took the opportunity to clean up some of the "demo" recipes as I think it's better to have a single source of truth for well-tested recipes.