Awesome-LLM

Summary of LLM Resources.

Content

• 1. Survey
• 2. Model
• 3. Pre-training
• 3. Finetune
• 5. Inference
• 6. Small LLM
• 7. Miscellaneous

1. Survey

• Small Language Models: Survey, Measurements, and Insights
  2024.09 - Zhenyan Lu - Beijing University of Posts and Telecommunications
• Large Language Models: A Survey
  2024.02 - Shervin Minaee - Snap Inc., USA
• A Survey of Large Language Models
  2023.11 - Wayne Xin Zhao - Renmin Runiversity, China

2. Model

DeepSeek Series - DeepSeek-AI, China

• DeepSeek-V3 Technical Report - 2024.12
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model - 2024.05
• DeepSeek LLM: Scaling Open-Source Language Models with Longtermism - 2024.01

LLaMA Series - Meta Inc, USA

• The Llama 3 Herd of Models - 2024.07
• Llama 2: Open Foundation and Fine-Tuned Chat Models - 2023.07
• LLaMA: Open and Efficient Foundation Language Models - 2023.02

Mamba Series

• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
  2023.12 - Albert Gu, Tri Dao - CMU, Princeton University
• Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers
  2021.10 - Albert Gu - Stanford University

FlashAttention Series

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
  2022.05 - Tri Dao - Stanford University

3. Pre-training

• Teaching Transformers Causal Reasoning through Axiomatic Training✅
  2024.07 - Aniket Vashishtha - Microsoft Research, India

4. Finetune

• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
  2024.07 - Zeyu Han - Northeastern University, USA

5. Inference

• A Survey on Efficient Inference for Large Language Models
  2024.07 - Zixuan Zhou - Tsinghua University, China
• Efficient Memory Management for Large Language Model Serving with PagedAttention
  2023.09 - Woosuk Kwon - UC Berkeley
• Clipper: A Low-Latency Online Prediction Serving System
  2017.03 - Daniel Crankshaw - UC Berkeley, USA

6. Small LLM

• Small Models are Valuable Plug-ins for Large Language Models
  2024.05 - Canwen Xu - University of California, San Diego

7. Miscellaneous

• LiL'Log
• AI-System
• Deep Generative Models
• CS 886: Recent Advances on Foundation Models