TE FP8 support?

[SinanAkkoyun]

Hi! Is adding FP8 transformer engine (H100) speedup to inference planned?
If not, could you please give me an outline of what needs to be done in order for me to work on that?

Thank you!

[WoosukKwon]

Hi @SinanAkkoyun, thanks for raising the issue! We are not familiar with the Transformer engine, and does not have access to H100 GPUs at the moment. We will let you know after we investigate more on that.

[casper-hansen]

Hi @WoosukKwon, any update on this issue? FLOPS is theoretically 2x of FP16. H100s can quite easily be rented from RunPod, Azure, AWS, or the like.

[wuchaooooo]

Hi @WoosukKwon , I also want to know that when vLLM will support FP8 in H100(H800)？FP8 is 2x faster than FP16.

[uncensorie]

any update on FP8 @WoosukKwon ?

[bdambrosio]

Second this, would be helpful for my batch biomed ontology extraction project.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[joydchh]

any update now?

[hmellor]

FP8 is supported https://docs.vllm.ai/en/latest/features/quantization/supported_hardware.html