Skip to content

Commit 804038f

Browse files
patrickvonplatenCyrilvallez
authored andcommitted
Add AutoTokenizer mapping for mistral3 and ministral (#42198)
* WIP * WIP
1 parent ede92a8 commit 804038f

File tree

1 file changed

+18
-0
lines changed

1 file changed

+18
-0
lines changed

src/transformers/models/auto/tokenization_auto.py

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -423,6 +423,15 @@
423423
"GPT2TokenizerFast" if is_tokenizers_available() else None,
424424
),
425425
),
426+
(
427+
"ministral",
428+
(
429+
"MistralCommonTokenizer"
430+
if is_mistral_common_available()
431+
else ("LlamaTokenizer" if is_sentencepiece_available() else None),
432+
"LlamaTokenizerFast" if is_tokenizers_available() and not is_mistral_common_available() else None,
433+
),
434+
),
426435
(
427436
"mistral",
428437
(
@@ -432,6 +441,15 @@
432441
"LlamaTokenizerFast" if is_tokenizers_available() and not is_mistral_common_available() else None,
433442
),
434443
),
444+
(
445+
"mistral3",
446+
(
447+
"MistralCommonTokenizer"
448+
if is_mistral_common_available()
449+
else ("LlamaTokenizer" if is_sentencepiece_available() else None),
450+
"LlamaTokenizerFast" if is_tokenizers_available() and not is_mistral_common_available() else None,
451+
),
452+
),
435453
(
436454
"mixtral",
437455
(

0 commit comments

Comments
 (0)