Lecturer: Binhang Yuan.
Teaching Assistant: You Peng, Yukun Zhou, Yuxuan Li.
In recent years, foundation models have fundamentally revolutionized the state-of-the-art of artificial intelligence. Thus, the computation in the training or inference of the foundation model could be one of the most important workflows running on top of modern computer systems. This course unravels the secrets of the efficient deployment of such workflows from the system perspective. Specifically, we will i) explain how a modern machine learning system (i.e., PyTorch) works; ii) understand the performance bottleneck of machine learning computation over modern hardware (e.g., Nvidia GPUs); iii) discuss four main parallel strategies in foundation model training (data-, pipeline-, tensor model-, optimizer- parallelism); and iv) real-world deployment of foundation model including efficient inference and fine-tuning.
| Date | Topic |
|---|---|
| W1-02/04,02/06 | Introduction and Logistics [Slides] & ML Preliminary [Slides] [Notebook] |
| W2-02/11,02/13 | Stochastic Gradient Descent [Slides] [Notebook] & Automatic Differentiation [Slides] [Notebook] |
| W3-02/18,02/20 | Language Model Architecture [Slides] & Large Scale Pretrain Overview [Slides] |
| W4-02/25,02/27 | Nvidia GPU Performance & Collective Communication Library |
| W5-03/04,03/06 | Data-, Pipeline- Parallel Training & Tensor Model-, Optimizer- Parallel Training |
| W6-03/11,03/13 | Sequence-, MoE- parallelism & Mid-Term Review |
| W7-03/18,03/20 | Mid-Term Exam & Generative Inference |
| W8–03/25,03/27 | Inference Alogirhtm Optimizations & Inference System Optimizations |
| W9-04/01,04/03 | Spring Break |
| W10-04/08,04/10 | Prompt Engineering & Inference Scaling |
| W11-04/15,04/17 | RAG & LLM Agent |
| W12-04/22,04/24 | Parameter Efficient Fine-Tuning & RL Alignment |
| W13-04/29 | LLM Evaluation |
| W14-05/06,05/08 | Guest Speech & Final Review |
- 4 Homework (4
$\times$ 5%$=$ 20%); - Mid-term exam (30%);
- Final exam (50%).
| Topic | Release | Due |
|---|---|---|
| Homework1 | 2025/02/13 ✔️ | 2025/02/22 |
| Homework2 | 2025/03/04 | 2025/03/12 |
| Homework3 | 2025/03/20 | 2025/04/01 |
| Homework4 | 2025/04/22 | 2025/04/31 |