Optimum-NVIDIA

[mshannon-sil]

https://huggingface.co/blog/optimum-nvidia

They're seeing up to 128x faster inference when using Optimum-NVIDIA. We should also check whether this can be used in tandem with BetterTransformer.

[ddaspit]

Optimum-NVIDIA currently only supports AutoModelForCausalLM, i.e. LLMs. It needs to support AutoModelForSeq2SeqLM to work with MT models.

[mshannon-sil]

Moving this to next quarter to check back in and see whether Optimum-NVIDIA adds support for AutoModelForSeq2SeqLM.

[mshannon-sil]

Support for other models has not been added yet, moving it to next quarter again to check back in.