v0.4.3 Release Tracker

[simon-mo]

ETA May 30 (due to some blockers and US holiday).

Blockers

• Support Deepseek-V2 #4650
• [Kernel][Backend][Model] Blocksparse flash attention kernel and Phi-3-Small model #4799
• [Bugfix / Core] Prefix Caching Guards (merged with main) #4846

Nice to have

• [Frontend] Dynamic RoPE scaling #4638
• [Kernel] Initial Activation Quantization Support #4525
• [Metrics] add more metrics #4464

[sasha0552]

Hi, is it possible to include the following PRs?

• Add vllm serve to wrap vllm.entrypoints.openai.api_server #4167
• [Hardware][Nvidia] Enable support for Pascal GPUs #4409
• [Frontend] Dynamic RoPE scaling #4638

[simon-mo]

Thanks for bring these up @sasha0552!

#4167 is unlikely to be finished in time.
#4409 might need a little bit more discussion given what features are supported for Pascal GPUs and whether building from source might be a better option.
#4638 can be included if it gets merged in time.

We do commit to biweekly release cadence so don't worry many of these will get into soon enough!

[robertgshaw2-neuralmagic]

Thanks for bring these up @sasha0552!

#4167 is unlikely to be finished in time. #4409 might need a little bit more discussion given what features are supported for Pascal GPUs and whether building from source might be a better option. #4638 can be included if it gets merged in time.

We do commit to biweekly release cadence so don't worry many of these will get into soon enough!

re: #4409 --> I did not have any issues running an fp16 model on a P40 when I installed from source.

[robertgshaw2-neuralmagic]

I am going to try to get these in

• [Kernel] Initial Activation Quantization Support #4525
• [Bugfix / Core] Prefix Caching Guards (merged with main) #4846
• [Metrics] add more metrics #4464

probably will not make it but tracking to v0.4.4:

• add spec infer related into prometheus metrics. #4582
• [CORE] Quantized lm-head Framework #4442

[njhill]

Sounds like we may want to include #4894 @rkooo567?

[rkooo567]

Yeah +1 on that PR @njhill

[robertgshaw2-neuralmagic]

• Fix for mistral-v0.3: [Bugfix] Fix Mistral v0.3 Weight Loading #5005

[jasonacox]

re: #4409 --> I did not have any issues running an fp16 model on a P40 when I installed from source.

Hi @robertgshaw2-neuralmagic - was this without the patch? I couldn't get a source build to run on P100's without the patch of #4409. With the patch, like you, running fp16 models (Mistral 7B for example) with no issues.

[sasha0552]

With the patch, like you, running fp16 models (Mistral 7B for example) with no issues.

Not only fp16, but AQLM works well too (#5058)

[robertgshaw2-neuralmagic]

re: #4409 --> I did not have any issues running an fp16 model on a P40 when I installed from source.

Hi @robertgshaw2-neuralmagic - was this without the patch? I couldn't get a source build to run on P100's without the patch of #4409. With the patch, like you, running fp16 models (Mistral 7B for example) with no issues.

P40 requires building with the patch.

[vrdn-23]

Is there any particular PR that we're waiting for before cutting the release?

[robertgshaw2-neuralmagic]

Is there any particular PR that we're waiting for before cutting the release?

The model support for Phi and Deepseek

[AmazDeng]

Is there any particular PR that we're waiting for before cutting the release?

The model support for Phi and Deepseek

Excuse me, when will VLLM support embedding input?

[dongxiaolong]

Could we include #4109 ? Structured output is also very important, and it seems almost complete. @simon-mo

Link to PR #4109

[LSC527]

Is there any particular PR that we're waiting for before cutting the release?

The model support for Phi and Deepseek

Hi, is Deepseek v2 supported now?

[simon-mo]

Moving Deepseek to optional in #5224
Tracking #4109 as optional in #5224