intel-extension-for-pytorch vs. intel-extension-for-transformers

[NaamaVian]

Describe the issue

Hello,

I noticed that there exists intel-extension-for-transformers, though intel-extension-for-pytorch also allows the deployment of LLMs. What is the difference between the two, and which one is best for deploying LLama2-70b?

Thanks!

[YuningQiu]

Hello, thanks for bringing up this great question.

In general, ITREX supports optimizations of common models on CPUs. It can get better accuracy with data type int4 than IPEX, but so far, ITREX does not support GPUs.

On the other hand, IPEX supports GPUs,expecially Ponte Vecchio (PVC).

For LLama2-70b, we suggest IPEX. More details can be found at https://github.com/intel/intel-extension-for-pytorch/tree/main/examples/cpu/inference/python/llm

[jgong5]

In general, ITREX supports optimizations of common models on CPUs. It can get better accuracy with data type int4 than IPEX, but so far, ITREX does not support GPUs.

Please allow me to clarify this a bit. As of now, both IPEX and ITREX rely on INC to quantize the model to int4 so I don't expect the accuracy would be difference between them. The difference between ITREX and IPEX is the API usage style where ITREX is more like HuggingFace native while IPEX is PyTorch native. Users can choose what API style to use per their preferences. Internally, ITREX will leverage IPEX kernels. For IPEX usage, users can leverage the optimize_transformers API from IPEX to streamline the optimizations of out-of-the-box HuggingFace models.