v0.4.1 Release Tracker

[simon-mo]

ETA Monday April 22

• [Misc] Bump transformers to latest version #4176
• Support eos_token_id from generation_config.json #4182 (addressing [Usage]: Llama 3 8B Instruct Inference #4180)
• [Core] add an option to log every function call to for debugging hang/crash in distributed inference #4079
• [Bugfix] Fix CustomAllreduce pcie nvlink topology detection (#3974) #4159
• [Bugfix] Fix LoRA loading check #4138
• [Bug]: OpenAI API Server always reports 0 tokens/s #4209
• Performance Regression between v0.4.0 and v0.4.1 #4210
• [Core][Distributed] use absolute path for library file #4271
• [Misc] Reduce supported Punica dtypes #4304 (otherwise we cannot upload to PyPI :(

[DefTruth]

@simon-mo I think we should add this fix into release tracker, because it fixed a major bug in the prefix prefill kernel that would cause many model services to crash when using prefix caching, such as bloom, phi-2 series etc.

• [Bugfix][Kernel] allow non-power-of-two head sizes in prefix prefill #4128

By the way, when will 0.4.1 be released ? Because we want to use a stable release that fixes the prefix prefill issue.

[jeejeelee]

@simon-mo #4138 should be merged before release

[simon-mo]

@DefTruth, if it is merged (which seems to be the case), it will be in the release

[simon-mo]

Oh I missed the second question. ETA is Friday April 19. (updated ETA above)

[zhouyuan]

Hi Simon, can we also have #3993 included? This can fix the CPU api server issue

thanks
-yuan

[simon-mo]

Because the CPU distribution is currently built from source and Docker directly, and we do not release prebuilt version, we don't need to mark it release blocking. We will release every two weeks.

[esmeetu]

@simon-mo We should wait for #4150 for release.

[simon-mo]

@esmeetu good catch. Merged and I'll retag.

[simon-mo]

https://github.com/vllm-project/vllm/releases/tag/v0.4.1