[Roadmap] vLLM Roadmap Q2 2024

[simon-mo]

This document includes the features in vLLM's roadmap for Q2 2024. Please feel free to discuss and contribute to the specific features at related RFC/Issues/PRs and add anything else you'd like to talk about in this issue.

You can see our historical roadmap at #2681, #244. This roadmap contains work committed by the vLLM team from UC Berkeley, as well as the broader vLLM contributor groups including but not limited to Anyscale, IBM, NeuralMagic, Roblox, Oracle Cloud. You can also find help wanted items in this roadmap as well! Additionally, this roadmap is shaped by you, our user community!

Themes.

We categorized our roadmap into 6 broad themes:

• Broad model support: vLLM should support a wide range of transformer based models. It should be kept up to date as much as possible. This includes new auto-regressive decoder models, encoder-decoder models, hybrid architectures, and models supporting multi-modal inputs.
• Excellent hardware coverage: vLLM should run on a wide range of accelerators for production AI workload. This includes GPUs, tensor accelerators, and CPUs. We will work closely with hardware vendors to ensure vLLM utilizes the greatest performance out of the chip.
• Performance optimization:vLLM should be kept up to date with the latest performance optimization techniques. Users of vLLM can trust its performance to be competitive and strong.
• Production level engine: vLLM should be the go-to choice for production level serving engine with a suite of features bridging the gaps from single forward pass to 24/7 service.
• Strong OSS product: vLLM is and will be a true community project. We want it to be a healthy project with regular release cadence, good documentation, and adding new reviewers to the codebase.
• Extensible architectures: For vLLM to grow at an even faster pace, it needs good abstractions to support a wide range of scheduling policies, hardware backends, and inference optimizations. We will work on refactoring the codebase to support that.

Broad Model Support

• Encoder Decoder Models
  • T5 Add Encoder-decoder model support and T5 Model support #3117
  • Whisper
  • Embedding Supporting embedding models #3187
• Hybrid Architecture (Jamba) [New Model]: Jamba (MoE Mamba from AI21) #3690
• Decoder Only Embedding Models [Model][Misc] Add e5-mistral-7b-instruct and Embedding API #3734
• Prefix tuning support

Help Wanted:

• More vision transformers beyond llava
• Support private model registration How to serve a private HF model? #172
• Control vector support [Feature]: Control vectors #3451
• Fallback support for arbitrary transformers text generation model
• Long context investigation of LongRoPE
• RWKV

Excellent Hardware Coverage

• AMD MI300x: enhancing fp8 performance [enable FP8 compute]
• NVIDIA H100: enhancing fp8 performance
• AWS Trainium and Inferentia
• Google TPU
• Intel CPU
• Intel GPU
• Intel Gaudi

Performance Optimization

• Speculative decoding
  • Speculative decoding framework for top-1 proposals w/draft model
  • Proposer improvement: Prompt-lookup n-gram speculations
  • Scoring improvement: Make batch expansion optional
  • Scoring improvement: dynamic scoring length policy
• Kernels:
  • FlashInfer integration Import FlashInfer: 3x faster PagedAttention than vLLM #2767
  • Sampler optimizations leveraging triton compiler
• Quantization:
  • FP8 format support for NVIDIA Ammo and AMD Quantizer
  • Weight only quantization (Marlin) improvements: act_order, int8, Exllama2 compatibility, fused MoE, AWQ kernels.
  • Activation quantization (W8A8, FP8, etc)
  • Quantized lora support Add Support for QLORA/QA-QLORA weights which are not merged #3225
  • AQLM quantization

• Constrained decoding performance (batch, async, acceleration) and extensibility (Outlines [Feature]: Update Outlines Integration from FSM to Guide #3715, LMFormatEnforcer [Feature]: Integrate with lm-format-enforcer #3713, AICI AI Controller Interface (AICI) integration #2888 )

Help Wanted:

• Sparse kv cache (H2O, compression, FastDecode)
• Speculative decoding
  • Proposer/scoring/verifier improvement: Top-k “tree attention” proposals for Eagle/Medusa/Draft model
  • Proposer improvement: RAG n-gram speculations
  • Proposer improvement: Eagle/Medusa top-1 proposals
  • Proposer improvement: Quantized draft models
  • Verifier improvement: Typical acceptance

Production Level Engine

• Scheduling
  • Prototype Disaggregated prefill (How to use Splitwise(from microsoft) in vllm? #2370)
  • Speculative decoding fully merged in ([WIP] Speculative decoding using a draft model #2188)
  • Turn chunked prefill/sarathi/splitfuse on by default ([2/N] Chunked prefill data update #3538)
• Memory management
  • Automatic prefix caching enhancement

• TGI feature parity (stop string handling, logging and metrics, test improvements)
• Provide non-ray option for single node inference
• Optimize api server performance
• OpenAI server feature completeness (function calling) (OpenAI Tools / function calling v2 #3237)
• Model Loading
  • Optimize model weights loading by directly loading from hub/s3 [Feature]: Add model loading using CoreWeave's tensorizer #3533
  • Fully offline mode

Help Wanted:

• Logging serving FLOPs for performance analysis
• Dynamic LoRA adapter downloads from hub/S3

Strong OSS Product

• Continuous benchmarks (resource needed!)
• Commit to 2wk release cadence
• Growing reviewer and committer base

• Better docs
  • doc: memory and performance tuning guide
  • doc: apc documentation
  • doc: hardware support levels, feature matrix, and policies
  • doc: guide to horizontally scale up vLLM service
  • doc: developer guide for adding new draft based models or draft-less optimizations

• Automatic CD of nightly wheels and docker images

Help Wanted:

• ARM aarch-64 support for AWS Graviton based instances and GH200
• Full correctness test with HuggingFace transformers. Resources needed.
• Well tested support for lm-eval-harness (logprobs, get tokenizers)
• Local development workflow without cuda

Extensible Architecture

• Prototype pipeline parallelism
• Extensible memory manager
• Extensible scheduler
• torch.compile investigations
  • use compile for quantization kernel fusion
  • use compile for future proofing graph mode
  • use compile for xpu or other accelerators
• Architecture for queue management and request prioritization
• Streaming LLM, prototype it on new block manager
• Investigate Tensor + Pipeline parallelism (LIGER)

[Jeffwan]

@simon-mo for prefill disaggregation. from the splitwise and distserve paper, they all build solution on top of vLLM for evaluation. Any contribution from these teams? is vLLM community open for public contribution for this feature?

[simon-mo]

@Jeffwan yes! We are actively with the authors of both papers to integrate the work properly. We are also working with Sarathi's authors for chunked prefill as well.

[kanseaveg]

Any update for PEFT?

please consider support huggingface peft, thank you. #1129

[simon-mo]

Hi @kanseaveg, we do support LoRA and planning to add prefix tuning support, which should allow Hugging face PEFT model format. Which PEFT methods are you interested in?

[kanseaveg]

@simon-mo Thank you very much for your reply.There are three common types of tuning methods that I am currently concerned about:

• prefix-tuning / p-tuning v2
• adapter-tuning
• lora-tuning (currently supported)
  I hope the vllm framework can support this, which is what I mentioned in Q3 last year and Q1 this year.
  Thank you very much for your reply.

[accupham]

Maybe consider supporting QuaRot quantization scheme?

QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism and to the KV cache. The result is a quantized model where all matrix multiplications are performed in 4-bits, without any channels identified for retention in higher precision. Our quantized LLaMa2-70B model has losses of at most 0.29 WikiText-2 perplexity and retains 99% of the zero-shot performance. Code is available at: this https URL.

I think this would be huge for larger models like Command-R+ (104B) being able to fit into a single 80G A100 with negligible performance losses.

[zbloss]

Very excited to see both Embedding models and CPU support on the roadmap!

These being implemented would make vLLM my default model serving engine.

[sangstar]

Very excited to see that the tensorizer PR is in this roadmap! Sorry about all the pings, I'm just passionate about getting this to vLLM users :D More than happy to be of any assistance in getting that feature implemented :)

[PenutChen]

Will larger vocabulary size for multi-lora be supported in Q2 2024? Related: #3000

[yukavio]

I'm very interested in implementing tree attention for speculative decoding. @simon-mo