[Feature]: Will vLLM support flash-attention 3 ?

[jorgeantonio21]

🚀 The feature, motivation and pitch

Flash attention 3 seems very promising for running efficient LLM inference on NVIDIA Hopper cards. Are there any plans to support it in the future ?

https://github.com/Dao-AILab/flash-attention

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

@WoosukKwon can you help answer this?

[DarkLight1337]

Also see #11194

[mgoin]

Yes, we are working directly with the authors to bring Flash Attention 3 to vLLM

[jorgeantonio21]

Really appreciate the effort guys ! Any estimate when this will be available ?

[mgoin]

Once the FA3 refactor branch lands (I believe currently at https://github.com/Dao-AILab/flash-attention/tree/decode), it should be ready to integrate. Hopefully within a few weeks

[jorgeantonio21]

@mgoin I remember that integrating FA2 with vLLM introduced a bunch of issues with block sizes less than 256 and how . I presume the same issues will have to be addressed with FA3..

[mgoin]

I think we will support small block sizes from the start of FA3 support, this is part of the reason why it is taking longer as we have this as a requirement.

[jorgeantonio21]

@mgoin @DarkLight1337 is there any progress on this ?

[mgoin]

We have landed initial support for FA3 with this PR! #12093

Currently only Hopper support is included due to binary size concerns, but look forward to it in this upcoming release.

[mgoin]

Tracking remaining features here #12429

[jorgeantonio21]

This is great work team, vLLM works much faster for me now. Thank you !