New Voxtral model support

[bing9]

Hi team,

Thanks for the great tool!

Mistal AI just launched Voxtral transcribe tool, better than the Whisper.

https://huggingface.co/mistralai/Voxtral-Mini-3B-2507/tree/main

Could we add model support?
Thanks!

[jooray]

That would be amazing, it seems it's much better at least in English.

[stduhpf]

I think it would probably make more sense to support it via llama.cpp, as it is not "just" a speech-to-text model like whisper, but a LLM with audio input modality.

[am17an]

I was trying to add this in llama.cpp, but they don't seem to add tokenizer.json or tokenizer_config.json in this HF repo vs other Mistral repos, so I'm having trouble making a gguf at the moment
But I see someone managed to do it https://huggingface.co/SaisExperiments/Voxtral-3B-But-4B-Text-Only-GGUF/, so it might just be my incompetence

[Kreijstal]

I was trying to add this in llama.cpp, but they don't seem to add tokenizer.json or tokenizer_config.json in this HF repo vs other Mistral repos, so I'm having trouble making a gguf at the moment But I see someone managed to do it https://huggingface.co/SaisExperiments/Voxtral-3B-But-4B-Text-Only-GGUF/, so it might just be my incompetence

that thing has the whisper layers ripped out, which defeats the point

[am17an]

I was trying to add this in llama.cpp, but they don't seem to add tokenizer.json or tokenizer_config.json in this HF repo vs other Mistral repos, so I'm having trouble making a gguf at the moment But I see someone managed to do it https://huggingface.co/SaisExperiments/Voxtral-3B-But-4B-Text-Only-GGUF/, so it might just be my incompetence

that thing has the whisper layers ripped out, which defeats the point

Yes I know, but it successfully manages to do the text part, which is the first part of this puzzle. Which I am unable to do at the moment. Maybe @CISC has some ideas.

[stduhpf]

Isn't the text part just Ministral 3B (with some more training)?

[CISC]

Yes I know, but it successfully manages to do the text part, which is the first part of this puzzle. Which I am unable to do at the moment. Maybe @CISC has some ideas.

The tekken.json file contains all the tokenization info, see mistral-commons tekken.py for info.

[CISC]

@am17an ggml-org/llama.cpp#14737 should interest you.