feat(llm-core): shared decoder transformer body + DecoderModelMetadata#109
Merged
Conversation
…lMetadata Adds an architecture-neutral decoder-only transformer body builder that each per-model `xNetwork(metadata)` can compose with its own knobs (RoPE base, RMSNorm eps, QK-norm) instead of duplicating the transformer DAG. Today `qwenNetwork()` is a 3-line stub delegating to `llamaNetwork()`, and `llamaNetwork()` hardcodes `eps = 1e-5f`, `qkNorm = false`, and the default RoPE base of 10000 — none of which are right for Qwen3. The intended fix is to collapse both `*NetworkDef.kt` files into thin callers of the shared `decoderTransformerNetwork` introduced here, with each passing its architecture-specific knobs explicitly. That collapse ships in the next PR; this PR is purely additive in :llm-core. The `DecoderModelMetadata` interface captures the shape fields plus the common defaults (`ropeFreqBase`, `rmsNormEps`, BOS/EOS) that every decoder LLM in this repo carries. `LlamaModelMetadata` adopts it in the follow-up so it can be passed directly. Tests verify: - module tree shape (token_embd / blk.N / output_norm / output) honors the layer count - `qkNorm` flips real `q_norm` / `k_norm` submodules into the MHA tree - `ropeBase` and `eps` defaults flow from metadata without code change Refs the no-model-duplication architectural plan (see ~/.claude/plans/snazzy-wibbling-dewdrop.md), and the closed #46. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
decoderTransformerNetwork) and aDecoderModelMetadatainterface to:llm-core. Every per-modelxNetwork(metadata)function can now compose it with that model's specific knobs (RoPE base, RMSNorm eps, QK-norm) instead of duplicating the transformer DAG.qwenNetwork()is a 3-line stub that delegates tollamaNetwork(), andllamaNetwork()hardcodeseps = 1e-5f,qkNorm = false, RoPE base = 10_000 — none of which are right for Qwen3. This PR is the foundation for collapsing both*NetworkDef.ktfiles into thin callers of the shared builder; that collapse and Llama-named loader rename ship as separate follow-up PRs per the no-model-duplication plan.Scope
:llm-core; no existing source modified.DecoderModelMetadata.kt(interface) — common shape fields +ropeFreqBase,rmsNormEps, BOS/EOS that every decoder LLM in this repo carries.DecoderTransformerNetwork.kt(builder) —Embedding → N × (RMSNorm → MHA(RoPE, KVCache, [QK-norm]) → Residual → RMSNorm → SwiGLU FFN → Residual) → RMSNorm → output Dense. Knobs (ropeBase,eps,qkNorm) default from metadata so callers can override per-architecture.DecoderTransformerNetworkTest.kt— module-tree-shape tests with a synthetic in-testDecoderModelMetadataimpl. The full integration with realLlamaModelMetadatalives in the follow-up*NetworkDefcollapse PR (which adds: DecoderModelMetadatato that data class).Test plan
./gradlew :llm-core:compileKotlinJvm :llm-core:compileCommonMainKotlinMetadata— clean compile, only pre-existing warnings../gradlew :llm-core:jvmTest— 9 suites / 85 tests pass, including the 3 new ones inDecoderTransformerNetworkTest.Refs the closed #46. No behavior change for downstream consumers — nothing imports the new code yet.
🤖 Generated with Claude Code