[Tracking Issue][Help Wanted]: FlashInfer backend improvements

[comaniac]

This issue tracks the progress and roadmap of integrating new FlashInfer kernels and enabling more vLLM features in FlashInfer backend. Items listed in Milestone 1 is a part of re-arch #8779 .

If you want any features available in FlashInfer backend but not listed here, or if you're interested in taking items, please feel free to comment.

Milestone 1

• Unify FlashInfer backend with multi-level cascade attention kernel. Adding Cascade Infer to FlashInfer #8132
• Support in-batch prefix sharing with 2-level cascade attention. @raywanb
• Support prefix caching.

Milestone 2

• Support chunked prefill.
• Support FP8 E4M3 kv-cache with kv scale.
• Support batch expansion in speculative decoding.
• Integrate RaggedTensor kernel.

cc @yzh119 @LiuXiaoxuanPKU @raywanb @simon-mo @WoosukKwon @pavanimajety