multi-token-prediction

DMax: Aggressive Parallel Decoding for dLLMs

Faithful from-scratch reimplementation of DeepSeek-V3 (MLA + MoE + MTP), scaled for Chinchilla-optimal 422M training on a single A100 80GB

PyTorch implementations of modern open-source LLM architectures (Llama, Qwen, DeepSeek, Gemma, GPT-OSS, Kimi, and more) — written from scratch for readability and learning, based on Sebastian Raschka's LLM Architecture Gallery.

MTP draft-head research harness for Qwen3.5-9B. Verdict: function cuts (pruning, low-rank, distillation) pay a 10-19pt off-distribution tax fidelity cuts do not — the zero-training trimmed-vocab NVFP4 recipe wins at 1.8-2.7x end-to-end in memra. Negative-result ledgers included.

A controlled 180-run study of DeepSeek-inspired MLA, sparse MoE routing, V3-style load balancing, and multi-token prediction under constrained compute.

From-scratch PyTorch: frontier LLM techniques as of 2026-Q1 — the Muon optimizer and Multi-Token Prediction, plus a base BPE tokenizer. Self-contained, self-checking modules.

Some benchmarks on running speculative decoding with Qwen-3.6-35B-A3B with EAGLE3, DFlash and MTP.

A lightweight experimental generative model for chemistry, with mini Qwen2-like architecture and horizon loss and biologically-aware RL fine-tuning on SELFIES molecular representations.

Fork of DeepSeek-V3 with MTP speculative-decoding improvements, a cache-aware mask fix, CPU test suite, and docs cleanup.

Benchmarks Multi-Token Prediction in llama.cpp against standard autoregressive baselines. Measures tokens per second, perplexity delta, and speculative acceptance rate across model sizes.

Research code for ProbeRoute, a probe-initialized sparse routing method for frozen-backbone multi-token prediction

Multi-Ensemble Memory-Elastic Token Prediction — a numpy implementation of DeepSeek's DSpark speculative decoding + the MEMTP elastic-ensemble extension, on AMD Strix Halo (gfx1151).

ChemMiniQ3-SAbRLo is a lightweight experimental generative model for chemistry, built on mini Qwen2-like arch, designed for rapid prototyping of HuggingFace AutoModel and AutoTokenizer compatibility, and fast iteration of Multi-Token Prediction (MTP) and RL fine-tuning algorithms/rewards.