Explore NVIDIA/TransformerEngine for speed/efficiency #1288

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like

• https://github.com/NVIDIA/TransformerEngine

  • A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper GPUs, to provide better performance with lower memory utilization in both training and inference.

Describe alternatives you've considered

• That this lib won't be useful in this repo, or that existing optimisations already do things as well as it could (or better).

Additional context

• Crossposted on:
  • [Feature Request]: Explore NVIDIA/TransformerEngine for speed/efficiency AUTOMATIC1111/stable-diffusion-webui#4721
• Other potential speed gain/improvement issues:
  • Explore potential speed benefits from implementing kernl (Up to 12X faster GPU inference) #1094
  • Community Integration: Making AIGC cheaper, faster, and more efficient. #1212
    • Community Integration: Making AIGC cheaper, faster, and more efficient. #1212 (comment)