CTranslate2

[Matthieu-Tinycoaching]

Hello,

Thanks for the great framework for deploying LLM.

Would it be possible to use a LLM model compiled with the CTranslate2 library?

[zhuohan123]

Thanks for bringing this up. We will investigate the CTranslate2 library and evaluate the difficulty and the potential benefit of adding it into vLLM.

[anujnayyar1]

Would love to see this, ct2 would be a great integration! It would give us easy access to fast 8 bit inference and plays nice with HF Transformers. Thank you for the library so far!!

[Matthieu-Tinycoaching]

Hi,

Any news regarding this integration? Ctranslate2 has already proven its speed within the TitanML framework for local LLM serving.

[manishiitg]

hi,

any news on this?

[Matthieu-Tinycoaching]

+1

[shixianc]

+11

[hmellor]

@zhuohan123 do you see any benefit of adding this to vLLM?

[hmellor]

Closing because if there was significant benefit, it would have been discussed more or even implemented by now.