Wire QwenNetworkLoader into CLI for proper Qwen3 inference

[michalharakal]

Context

The CLI (Main.kt) always routes GGUF models through LlamaIngestion → LlamaRuntime, which works for Llama-architecture models. Qwen3 models load successfully (same tensor names), but produce garbage output because LlamaRuntime doesn't handle Qwen3-specific features:

• QK-norm (query/key normalization via attn_q_norm.weight / attn_k_norm.weight)
• RoPE base frequency (1,000,000 vs Llama's 10,000)
• BOS token differences

The correct loader (QwenNetworkLoader in llm-inference:qwen) exists but isn't wired into the CLI.

Scope

• Add :llm-inference:qwen dependency to :llm-runtime:kllama
• Detect qwen* architecture from GGUF metadata in Main.kt
• Route to QwenNetworkLoader.fromGguf() for Qwen models
• Wire the Qwen DSL network module into a runtime compatible with AgentLoop
• Validate end-to-end with Qwen3-1.7B-Q8_0.gguf --demo

Related

• Parent: Generalize tool-calling support across model families #35 (Generalize tool-calling support)
• Depends on architecture detection added in Generalize tool-calling support across model families #35 commits

Acceptance Criteria

• Qwen3-1.7B-Q8_0.gguf --demo produces coherent output
• Tool calling works through the Qwen chat template
• Llama models continue working unchanged

[michalharakal]

Closing as resolved — the Qwen3 inference goal here is met, though via a different implementation than this issue originally proposed.

What shipped (instead of wiring QwenNetworkLoader into the CLI):

• LlamaWeightLoader / LlamaRuntime were extended to handle Qwen specifics, and Qwen GGUFs are routed through that path.
• CLI detection: Main.kt:355 branches on family == "qwen" → Main.kt:363-395 uses LlamaWeightLoader with acceptedArchitectures = {"qwen2", "qwen3", "qwen35"}.
• Qwen3 features the issue called out:
  • QK-norm: LlamaRuntime.kt:92 (hasQKNorm), applied at :150 and :295; tensor names resolved in LlamaWeightLoader.kt:65-66.
  • RoPE base: read from GGUF metadata and passed via Main.kt:388 (runtimeWeights.metadata.ropeFreqBase).
  • EOS / BOS / rmsNormEps: pulled from metadata at Main.kt:393-395.

Relevant commits:

• 1fb532c feat(Wire QwenNetworkLoader into CLI for proper Qwen3 inference #46): wire Qwen3 loading into unified kllama CLI
• 5f84f15 Fix Qwen3 inference: read RoPE base, QK-norm, BOS token from GGUF metadata
• abb710f qwen working with smoke tests

Follow-up: the standalone :llm-inference:qwen module (QwenNetworkLoader, QwenNetworkDef, QwenGgufWeightSource, QwenHfTensorNameMapper) is now unused — no dependency from :llm-runtime:kllama and no references in llm-runtime/, llm-core/, or llm-api/. Worth a separate issue to either remove it or repurpose it (e.g., migrate the working Qwen path off the LlamaRuntime hybrid onto a proper architecture-specific runtime).