[v0.4.0] Release Tracker

[zhuohan123]

ETA: Before Mar 28th

Major changes

TBD.

PRs to be merged before the release

• Support int8 KVCache Quant in Vllm #1507
• Add Automatic Prefix Caching #2762
• ...

[nivibilla]

@zhuohan123 do you think the second part for int8 inference should also be added into the roadmap?
#1508

[miko7879]

Would it be possible to request that the following is merged prior to release: #2961

It is preventing us from being able to deploy models using vLLM in certain contexts.

[Xu-Chen]

This amazing MR created by @chu-tianxiang was originally supposed to be merged into the previous version. Hope it can be merged this time. The speed improvement is very obvious.

[zhaoyang-star]

@zhuohan123 Hope Prefix Caching with FP8 KV cache support (#3234) could be merged.

[GennVa]

@zhuohan123 Would it be possible to request if #3233 can be merged in this release?
As solution for #2398 and #3118
Thanks

[robertgshaw2-neuralmagic]

Cmake -> #2830

[simon-mo]

I'm planning to merge usage reporting PR by Monday.

[JMHenri]

We are very hopeful to have the JAIS commit in v0.3.4 4c07dd2

[wsp317]

When do you plan to release v0.3.4?

[simon-mo]

We have a hard deadline to release it this week. Sorry about the delay.

[grandiose-pizza]

We are very hopeful to have the JAIS commit in v0.3.4 4c07dd2

Hi @JMHenri , Please let me know in case you face any issues with the JAIS models. Since we don't have a chat_template in the tokenizer, please feel to reach out in case of any issues.

You can certainly look into this for reference on usage.