Implement a LLM step by step with PyTorch from basic torch utilize to a LLM with MoE. Code can run on CPU, GPU, NPU(Ascend).
Important
To have a better reading experience. You'd better change the Monospaced fonts to the one you like, such as Consolas and Cascadia Code etc. Or when you reading the notebook, the code is very ugly maybe.
Note
this repository is under coding. Cause I'm a student, the developing process may be slow.
| Direcroty | SubSection | Status |
|---|---|---|
| basic_pytorch | tensor | ✅ |
| basic_pytorch | data | |
| basic_pytorch | activation_func | |
| basic_pytorch | nn | |
| basic_pytorch | autograd | |
| optimizer | Gradientprop | |
| optimizer | SGD | |
| optimizer | RMSprop | |
| optimizer | Adam | |
| optimizer | AdamW | |
| optmizer | Muon | |
| tokenizer | \ | ✅ |
| embedding | \ | ✅ |
| transformer | \ | |
| llm | GPT | |
| llm | Llama | |
| llm | Qwen | |
| llm | DeepSeek | |
| llm | ChatGLM | |
| llm | Kimi | |
| train | \ | |
| train | DeepSpeed | |
| infer | LLaMA.cpp | |
| infer | Ollama | |
| infer | SGLang | |
| Multimodal | \ | |
| MoE | \ | |
| Finetuning | Classification | |
| Finetuning | Instruction | |
| Finetuning | LoRA | |
| Finetuning | QLoRA | |
| Finetuning | LLaMA-Factory | |
| Finetuning | LLaMA-Adapter | |
| Finetuning | RLHF | |
| Distill | \ | |
| quantization | \ | |
| compression | \ | |
| deployment | Fastapi | |
| deployment | onnx-runtime | |
| deployment | TensorRT | |
| deployment | vllm | |
| corpus | \ |