LLM Scratch Scratch

Intro

This is a scratchpad project that implements different LLM parts from scratch, also builds and trains small model variants of popular LLM architectures.

Contents

Architecture

• Attention
  • scaled dot-product attention
  • Multi-head variants
    • Multi-Head Attention (MHA)
    • Multi-Query Attention (MQA)
    • Grouped-Query Attention (GQA)
    • Multi-head Latent Attention (MLA)
• MoE
  • MoE overview (EL, sparsity, granularity, shared expert)
  • load balancing/routing
• Hybrid
  • SSM/Mamba-2
  • Linear Attention/Gated Attention (https://github.com/fla-org/flash-linear-attention)
  • Sparse Attention
  • Lightning Attention (MiniMax-01 https://arxiv.org/pdf/2501.08313)
• Positional Encoding
  • RoPE
  • Learned Absolute Positional Encoding
  • Relative Positional Encoding
• Context extention
• Context extention overview
• RoPE and YaRN
• Activation
  • Overview and historical methods (ReLU, GeLU)
  • SwiGLU
• Normalization
  • RMSNorm
• Embedding
• Tokenizer
  • BPE
  • (references)
    • Andrej Karpathy
    • Hugging Face
    • Getting the most out of your tokenizer for pre-training and domain adaptation

Models From Scratch

Training / Fine-tuning

• Hyperparameters
  • batch size
  • learning rate
• Reinforcement Learning
  • PPO
  • DPO
  • GRPO
  • DAPO
  • GSPO
  • ORPO
  • KTO

Optimization / Distributed

Acknowledgments

Book

• Sebastian Raschka 's amazing book Build a Large Language Model From Scratch

Papers

• Transformer paper Attention Is All You Need (arxiv.org/abs/1706.03762)
• MQA paper Fast Transformer Decoding: One Write-Head is All You Need (arxiv.org/abs/1911.02150)
• GQA paper GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (arxiv.org/abs/2305.13245)
• DeepSeek-v2 paper (proposed MLA) DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Blogs, Online articles

• Sebastian Raschka's The Big LLM Architecture Comparison
• Sebastian Raschka's Beyond Standard LLMs
• Hugging Face's The Smol Training Playbook: The Secrets to Building World-Class LLMs
• The Annotated Transformer

Code

• Frank Odom's GQA PyTorch implementation

History

• 2025/06/26 Project start
• 2025/06/27
  • Add scaled dot-product attention
  • Add multi-head attention (MHA)
• 2025/06/29
  • Add multi-query attention (MQA)
  • Add attention and MHA variants explanation in attention readme
• 2025/07/24
  • Add multi-head latent attention (MLA)
• 2025/08/05
  • Update MLA implementation to follow DeepSeek-v2 official formula