🚀 Efficient-LLM-Inference

旧时王谢堂前燕，飞入寻常百姓家。这或许正是大语言模型高效推理的意义所在：通过优化大模型推理，将性能和效率提升至极致，让大模型不再只是少数人（富人、科技巨头公司）的专属。为实现将简单易用、高效、低成本的大模型推理服务带给每个人而不懈奋斗。

❤️ 课程

• MIT Song Han | Model Compression and Acceleration Techniques for AI Computing

• Stanford CS25: Transformers United V5

• Stanford CS336: Language Modeling from Scratch

• Washington University | CSE 5610: Large Language Models (2025 Fall)

• University of Pennsylvania | CIS 7000: Large Language Models

• UC Berkeley & Google | Large Language Model Agents

• California Institute of Technology | Large Language Models for Reasoning

• Datawhale 发布 Happy-LLM：刚上线一周就收获 2.3K+ Star 的大模型 "顶流教程"

• 上海交通大学 - 《动手学大模型》系列编程实践教程

🎮 视频教程

• ZOMI 酱在 B 站上传了一系列视频教程

• 高效的大语言模型推理 - Efficiency 论文分享 @ICML & ACL 2025

• 如何加速大语言模型推理？万字长文综述大语言模型高效推理技术

• ICML-2024 Towards Efficient Generative Large Language Model Serving: A Tutorial from Algorithms to Systems

• 高效大模型推理技术与应用 | 无问芯穹首席科学家戴国浩 GenAICon 2025 报告

• 国产大算力 GPU 如何迎接 DeepSeek 新机遇 | 壁仞科技 AI 软件首席架构师丁云帆 GenAICon 2025 演讲

• 一些已成为 LLM 推理引擎中事实标准的方法

• 一小时精讲高性能 LLM 推理框架及细节优化

• Meta AI 的研究科学家 Kai Sheng Tai | 稀疏性促进高效 LLM 推理

• 英伟达机器学习和深度学习专家 Christian Merkwirth | 优化 LLM 推理：挑战与最佳实践

• 英伟达高级数据科学家 Mark Moyou | 从理论到具有成本效益的部署，掌握 LLM 推理优化

• 北大刘譞哲 | 面向 AI Infra 的系统软件：挑战、思考和实践

• NVIDIA GTC 技术干货 | 超大型 Transformer 模型的高效推理

• 硅谷知名华人 AI 科学家田渊栋 | 支持长文本上下文的 LLMs 的高效推理

• Julien Simon 曾任亚马逊（AWS）人工智能和机器学习全球布道师 | 深度探究：LLM 推理优化

• OneFlow - 大语言模型的推理

• KTransformers 团队分享异构推理架构思路：基于内存的大模型推理成本优化

🛞 开源项目

• vLLM - Easy, Fast, and Cheap LLM Serving for Everyone

• SGLang - A Fast Serving Framework for Large Language Models and Vision Language Models

• Nano-vLLM - A Lightweight vLLM Implementation Built from Scratch

• llm-d: A Kubernetes-Native High-Performance Distributed LLM Inference Framework

• Xinference - Deploy AI Models Fast And Seamless Enterprise Ready

• Tencent KsanaLLM - A High Performance and Easy-To-Use Engine for LLM Inference and Serving

• NVIDIA - LM Studio

• LMDeploy - A Toolkit for Compressing, Deploying, and Serving LLMs

• HUAWEI Ascend MindIE

• Ollama - Get up and Running with Llama 3.3, Deepseek-R1, Phi-4, Gemma 3, Mistral Small 3.1 and Other LLMs.

• Hugging Face TGI - Text Generation Inference

• NVIDIA TensorRT-LLM - A Tensorrt Toolbox for Optimized Large Language Model Inference

• DeepSeek - The Path to Open-Sourcing the DeepSeek Inference Engine

• RTP-LLM: Alibaba's High-Performance LLM Inference Engine for Diverse Applications

• Chitu - High-Performance Inference Framework for Large Language Models, Focusing on Efficiency, Flexibility, and Availability.

• NVIDIA Dynamo - A Datacenter Scale Distributed Inference Serving Framework

• llama.cpp - LLM Inference in C/C++

• MLC LLM - Universal LLM Deployment Engine with ML Compilation

• DashInfer - A Native LLM Inference Engine Aiming to Deliver Industry-Leading Performance Atop Various Hardware Architectures, Including CUDA, x86 and ARMv9

📝 博客

• AAAI 2025 收录，LLM 推理加速新范式！加州大学洛杉矶分校提出 DSBD 技术，有效结合推测解码与束采样

• 突破百万 Token 处理瓶颈：Context Parallelism 让大模型推理提速 10 倍

• 俞星凯 Nano-vLLM：从零开始构建了一个轻量级 vLLM 实现，将代码简化到了 1200 行

• 大语言模型推理优化技术综述（The Art of LLM Inference）

• 昇腾 × 盘古：模型与系统全链路优化，打造大模型推理 "性能王炸" 组合！

• 谷歌之后，英伟达入局扩散型大语言模型，Fast-dLLM 推理速度狂飙 27.6 倍

• 大语言模型（LLM）推理框架的全面分析与选型指南（2025 年版）

• 告别卡脖子，华为黑科技破局！昇腾推理加速 1.6 倍打破 LLM 降智魔咒

• 昇腾超大规模 MoE 模型推理优化技术揭秘：MTP 调度 10 倍提速，INT 打平 FP8

• 大模型推理优化概述

• ICLR 2025 | 计算开销减半！Dynamic-LLaVA 刷新多模态大模型推理加速上限

• 帮大模型提速 80%，华为拿出昇腾推理杀手锏 FlashComm，三招搞定通算瓶颈

• 如何重现 DeepSeek 推理性能突破

• 华为 + DeepSeek，推理性能创新高！技术报告也公布出来了

• 华为曝光两大黑科技！打破推理延迟魔咒，大模型从此「秒回」

• 华为是怎么让大模型提速的？

• 一作解读！从 idea 视角，聊聊 Qwen 推出的新 Scaling Law —— Parallel Scaling

• 叶子豪、陈天奇等人开源项目 FlashInfer 入选，MLSys2025 最佳论文奖公布

• 结合 MindIE-LLM 框架的具体优化案例，分析大模型推理加速的关键技术

• 大模型推理利器：3 大开源框架揭秘｜漫谈 MFU（三）

• DeepSeek-V3 再发论文，梁神署名，探讨扩展挑战并反思硬件架构

• 大模型推理成本每年降低 10 倍的秘密：一文了解 vLLM、SGLang 等主流推理引擎

• 梁文锋新论文！DeepSeek 降本秘籍公开，突破算力瓶颈有六招

• 探索大语言模型 LLM 的推理优化秘密

• 两万六千字，大模型核心技术综述：微调、推理与优化指南

• 漫谈大模型推理优化技术系列 ——（二）模型推理服务

• 2024 年 - 开源大模型推理引擎现状及常见推理优化方法

• 10 种主流 LLM 推理框架的技术介绍与对比：从本地部署到企业级服务

• 一文读懂 llama.cpp/vLLM/SGLang/FastTransformer/TensorRT/TGI/MindIE 大模型推理引擎（附图）

• 阿里云 Higress 团队 - 大模型推理服务全景图

• 大模型推理优化 —— 技术突破与产业落地新范式

• 毫秒级响应的秘密：解密大模型推理层如何用 10% 成本实现 10 倍速性能

• 万字长文，大模型推理性能优化综述汇总

• 语言大模型的推理技巧

• AI 推理场景的痛点和解决方案

• 七种大模型推理加速的方法

• Lilian Weng 出品，必是精品 | 大型 Transformer 模型推理优化

• 大模型推理优化关键技术

• 高性能 LLM 推理框架的设计与实现

• 一文读懂大模型推理服务平台（Xinference/Ollama/GPUStack/KServe/Triton/LMDeploy）

• Semi-PD，P/D 半分离的调度策略，大模型推理范式新选择

• 一念 LLM 大语言模型推理加速

• 大模型推理显存优化的深度探索与实践

• 大模型推理引擎国产化思考和实践

• 大模型的模型压缩与有效推理要点总结

• 「乘法变加法」！MIT 清华校友全新方法优化 Transformer：Addition is All You Need

• 迈向 100 倍加速：全栈 Transformer 推理优化

💻 方法论文

• Chen, M., Hui, B., Cui, Z., Yang, J., Liu, D., Sun, J., ... & Liu, Z. (2025). Parallel Scaling Law for Language Models. arXiv preprint arXiv: 2505.10475.

• Li, Y., Wei, F., Zhang, C., & Zhang, H. (2025). EAGLE-3: Scaling Up Inference Acceleration of Large Language Models via Training-Time Test. arXiv preprint arXiv: 2503.01840.

• Liao, B., Xu, Y., Dong, H., Li, J., Monz, C., Savarese, S., ... & Xiong, C. (2025). Reward-Guided Speculative Decoding for Efficient LLM Reasoning. arXiv preprint arXiv: 2501.19324.

• Zhang, T., Sui, Y., Zhong, S., Chaudhary, V., Hu, X., & Shrivastava, A. (2025). 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float. arXiv preprint arXiv: 2504.11651.

• Ye, Z., Chen, L., Lai, R., Lin, W., Zhang, Y., Wang, S., ... & Ceze, L. (2025). FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. arXiv preprint arXiv: 2501.01005.

• Xu, J., Pan, J., Zhou, Y., Chen, S., Li, J., Lian, Y., ... & Dai, G. (2025). SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting. arXiv preprint arXiv: 2504.08850.

• Prabhu, R., Nayak, A., Mohan, J., Ramjee, R., & Panwar, A. (2025). vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention. In Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 1: 1133-1150.

• Huang, Y., Wan, L. J., Ye, H., Jha, M., Wang, J., Li, Y., ... & Chen, D. (2024). Invited: New Solutions on LLM Acceleration, Optimization, and Application. In Proceedings of the 61st ACM/IEEE Design Automation Conference (DAC), 1-4.

• Chen, Y., & Huang, G. (2024). GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments. arXiv preprint arXiv: 2412.04788.

• Dong, J., Feng, B., Guessous, D., Liang, Y., & He, H. (2024). Flex Attention: A Programming Model for Generating Optimized Attention Kernels. arXiv preprint arXiv: 2412.05496.

• Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., ... & Stoica, I. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP), 611-626.

💡 综述论文

• A Survey of LLM Inference Systems

• Zhen, R., Li, J., Ji, Y., Yang, Z., Liu, T., Xia, Q., ... & Zhang, M. (2025). Taming the Titans: A Survey of Efficient LLM Inference Serving. arXiv preprint arXiv: 2504.19720.

• Park, S., Jeon, S., Lee, C., Jeon, S., Kim, B. S., & Lee, J. (2025). A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiency. arXiv preprint arXiv: 2505.01658.

• Sui, Y., Chuang, Y. N., Wang, G., Zhang, J., Zhang, T., Yuan, J., ... & Hu, X. (2025). Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models. arXiv preprint arXiv: 2503.16419.

• 葛旭冉, 欧洋, 王博, 赵宇, 吴利舟, 王子聪, 陈志广, 肖侬. 大语言模型推理中的存储优化技术综述[J]. 计算机研究与发展, 2025, 62(3): 545-562. DOI: 10.7544/issn1000-1239.202440628

• Liu, Y., Wu, J., He, Y., Gao, H., Chen, H., Bi, B., ... & Hooi, B. (2025). Efficient Inference for Large Reasoning Models: A Survey. arXiv preprint arXiv: 2503.23077.

• Zhou, Z., Ning, X., Hong, K., Fu, T., Xu, J., Li, S., ... & Wang, Y. (2024). A Survey on Efficient Inference for Large Language Models. arXiv preprint arXiv: 2404.14294.

• Li, J., Xu, J., Huang, S., Chen, Y., Li, W., Liu, J., ... & Dai, G. (2024). Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective. arXiv preprint arXiv: 2410.04466.

• Chavan, A., Magazine, R., Kushwaha, S., Debbah, M., & Gupta, D. (2024). Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI), 7980-7988.

• Luohe, S., Zhang, H., Yao, Y., Li, Z & Zhao, H. (2024). Keep the Cost Down: A Review on Methods to Optimize LLM's KV-Cache Consumption. In First Conference on Language Modeling (COLM).

• Li, H., Li, Y., Tian, A., Tang, T., Xu, Z., Chen, X., ... & Chen, L. (2024). A Survey on Large Language Model Acceleration Based on KV Cache Management. arXiv preprint arXiv: 2412.19442.

• Wan, Z., Wang, X., Liu, C., Alam, S., Zheng, Y., Liu, J., ... & Zhang, M. (2024). Efficient Large Language Models: A Survey. Transactions on Machine Learning Research (TMLR).

• Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Jin, H., Chen, T., & Jia, Z. (2023). Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems. arXiv preprint arXiv: 2312.15234.

• Kim, S., Hooper, C., Wattanawong, T., Kang, M., Yan, R., Genc, H., ... & Gholami, A. Full Stack Optimization of Transformer Inference. In Architecture and System Support for Transformer Models (ASSYST@ ISCA 2023).