Train speculative decoding models effortlessly and port them smoothly to SGLang serving.

Best practices & guides on how to write distributed pytorch training code

Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)

Llama-style transformer in PyTorch with multi-node / multi-GPU training. Includes pretraining, fine-tuning, DPO, LoRA, and knowledge distillation. Scripts for dataset mixing and training from scratch.

Fast and easy distributed model training examples.

A script for training the ConvNextV2 on CIFAR10 dataset using the FSDP technique for a distributed training scheme.

Simple and efficient implementation of 671B DeepSeek V3 that trainable with FSDP+EP and minimal requirement of 256x A100/H100, targeted for HuggingFace ecosystem

A minimal, hackable pre-training stack for GPT-style language models

Minimal yet high performant code for pretraining llms. Attempts to implement some SOTA features. Implements training through: Deepspeed, Megatron-LM, and FSDP. WIP

Implementations of some popular approaches for efficient deep learning training and inference

Framework, Model & Kernel Optimizations for Distributed Deep Learning - Data Hack Summit

Dataloading for JAX

Scalable multimodal AI system combining FSDP, RLHF, and Inferentia optimization for customer insights generation.

Training Qwen3 to solve Wordle using SFT and GRPO

Mini-FSDP for PyTorch. Minimal single-node Fully Sharded Data Parallel wrapper with param flattening, grad reduce-scatter, AMP, and tiny GPT/BERT training examples.

A foundational repository for setting up distributed training jobs using Kubeflow and PyTorch FSDP.

Fully Sharded Data Parallel (FSDP) implementation of Transformer XL

This repository showcases hands-on projects leveraging distributed multi-GPU training to fine-tune large language models (LLMs).

Custom FSDP + LoRA training loop for Ministral 3 (axolotl workaround)