llama: support Qwen3

[CISC]

Initial draft based on huggingface/transformers#36878

In case models are released before I can have a look at them this weekend:

TODO

• Set type for all layer sizes in llama_model::load_hparams
• Test conversion and inference on all models

[x0wllaar]

Are you planning to add MoE support?

[CISC]

Are you planning to add MoE support?

I'm focusing on non-MoE for now, so if someone wants to work on Qwen3MoE in the mean time they are more than welcome to. :)

[x0wllaar]

Thank you! I not sure I'm up to the task though lol

[ngxson]

I had a look at the quen3 MoE python code, it's not much difference from qwen2 MoE. Diff are:

• Shared experts are removed
• Added k_norm and q_norm (similar to qwen3 dense)

[CISC]

I had a look at the quen3 MoE python code, it's not much difference from qwen2 MoE.

That was my initial impression too, I can have a stab at it if no-one else volunteers, just didn't want to bite off too much at once (esp. given the flustercuck 57B-A14B was). :)

[CISC]

Superseded by #12828