Skip to content

Conversation

@daiyunwei1998
Copy link

@daiyunwei1998 daiyunwei1998 commented Jan 19, 2026

Summary

Update Gemma tokenizer usage in convergence tests for Transformers v5 by removing deprecated GemmaTokenizerFast imports and renaming usages to the supported non-fast tokenizer class. This fixes the No module named transformers.models.gemma.tokenization_gemma_fast error when running convergence tests under Transformers v5.

Details

Transformers v5 moves away from parallel “fast” and “slow” tokenizer implementations and adopts a single tokenizer implementation (see huggingface/transformers#40936).

  • Convergence tests were importing and instantiating the fast tokenizer class, causing import errors.
  • This change updates both: 1) the import path, and 2) the tokenizer class name used in code (GemmaTokenizerFastGemmaTokenizer), following the new Transformers v5 API.

Testing Done

  • Hardware Type: A100-40G-PCIe
  • run make test to ensure correctness
  • run make checkstyle to ensure code style
  • run make test-convergence to ensure convergence

@Tcc0403 Tcc0403 force-pushed the transformers-5.0.0rc1 branch from df188d7 to 2cd6e39 Compare January 20, 2026 06:44
Copy link
Collaborator

@Tcc0403 Tcc0403 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@Tcc0403 Tcc0403 merged commit 6a28a3f into linkedin:transformers-5.0.0rc1 Jan 20, 2026
@Tcc0403 Tcc0403 mentioned this pull request Jan 20, 2026
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants