Speculative Decoding / MTP

[davidsyoung]

Wanted to ask a question here as I don’t fully understand the implementation details, but is something like speculative decoding or multi token prediction in Deepseek MoE models possible with the way llama.cpp does inference?

It would be an incredible feature to have, but I’m not even sure if it’s realistic.

Thank you!

[ejrydhfs]

Llama.cpp already has a form of speculative decoding implemented based on trees but I believe it's not in the main example program, I wonder if it would be possible for llama.cpp to do multi token prediction next, as well as more advanced forms of speculative decoding