[Question]: how much size of gpu should i use to deploy ragflow？

[sanwei111]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (Language Policy).
• Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
• Please do not modify this template :) and fill in all the required fields.

Describe your problem

i deploy by the gpu version， so embedding process and reranker process and parse process ，even the llm generation use the gpu。

so how much size of gpu should i use to deploy ragflow？to fix the request of concurrent performance

[KevinHuSh]

I think 80G is necessary for deploying a LLM.

[sanwei111]

I think 80G is necessary for deploying a LLM.

thank you for response,sir.

one more question for detail, now i deploy ragflow in 3090. i use the gpu version docker compose with embedding model and reranker. what's more, i deploy a ollama to run a 7B llm.

so, the overall models are as follow, file parser model in ragflow server, embedding model in ragflow server, reranker model in ragflow server. And lllm model in ollama server.

i think there are too many models, how much ram and vram should i prepare? and is it better to deploy embedding and reranker, llm in the ollama server instead of the ragflow server?

[KevinHuSh]

You could deploy the slim version of RAGFlow and utilize Ollama for embedding and LLM inference which are on GPUs.