[v0.3.1] Release Tracker

[WoosukKwon]

ETA: Feb 14-16 th

Major changes

TBD

PRs to be merged before the release

• Don't use cupy NCCL for AMD backends #2855
• Fix docker python version #2845
• Support per-request seed #2514
• Ensure memory release when LLM class is deleted. [BugFix] Fix GC bug for LLM class #2882
• Fix AttributeError: MixtralModel object has no attribute org_vocab_size. #2875 Align LoRA code between Mistral and Mixtral (fixes #2875) #2880

[WoosukKwon]

@simon-mo Please feel free to add more!

[simon-mo]

I would really like #2804 but it seems to be blocked by either FlashInfer or other libraries

[WoosukKwon]

@simon-mo I think the main concern here is AMD because the ROCm xformers patch uses xformers==0.0.23 while we're upgrading to xformers==0.0.24 for CUDA.

[simon-mo]

I see. We might be able to distribute different version with varying version pins...

[tutu329]

need supporting miqu-1-70b-sf-gptq. thanks a lot！

[umarbutler]

I would like to see a fix to #2795. I and two other users have been unable to use the latest version of vLLM with Ray, however, it works perfectly well after downgrading to the previous version.

[casper-hansen]

#2761 brings back support for quantized MoE models like Mixtral/Deepseek. Also brings a great speedup (2-3x).

Possible to include it in the next release so that quantized models are not broken?

[simon-mo]

@umarbutler In this release, we will disable to the custom all reduce, which should address #2795.
@casper-hansen We should definitely get this PR in in next two weeks, but we need to cut a release today to address critical issues.

[pcmoritz]

We will need to include #2875 in the release as well

[WoosukKwon]

@pcmoritz Added. Thanks!

[WoosukKwon]

@caoshiyi Would it be possible to merge #2517 before the release?

[caoshiyi]

@WoosukKwon Sorry for the delay. Will address the comments tonight.