A clean, single-file PyTorch implementation of Attention Residuals (Kimi Team, MoonshotAI, 2026), integrated with Grouped Query Attention (GQA), SwiGLU feed-forward networks, and Rotary Position Embeddings (RoPE).
open-source research ai ml transformers torch pytorch attention diffusion residuals llms open-implementation attention-transformer
-
Updated
Mar 16, 2026 - Python