[Question]: What type of GPU and how many GPUs are recommended to fasten the speed of parsing files?

[zengqingfu1442]

Describe your problem

I use CPU to parse a PDF file whose size is 55MB, but is cost about 1 hour. It's too slow to create knowledge base in ragflow. What type of GPU are recommended to use to fasten the speed of parsing files? And how many GPUs? Would RTX 4090 or RTX A6000 is recommended? Thanks.

[KevinHuSh]

Please deploy an embedding service with Ollama/Xinference on your GPUs.
That's gona accelerate much more.

[zengqingfu1442]

Please deploy an embedding service with Ollama/Xinference on your GPUs. That's gona accelerate much more.

I prefer vllm. So you recommend to deploy an embedding model and then add it as embedding model onto ragflow? Which embedding model is recommended?

[writinwaters]

Create a knowledge base is not time-consuming; parsing file is. Here are some tips: https://ragflow.io/docs/dev/accelerate_doc_indexing

Or, you can use docker-compose-gpu.yml to start your service. This accelerates DeepDoc tasks using GPU, requiring RAGFlow v0.16.0+.

[yuzhichang]

@zengqingfu1442 Yes. The recommended embedding model is bge-m3.

[zengqingfu1442]

Please deploy an embedding service with Ollama/Xinference on your GPUs. That's gona accelerate much more.

As shown below, this file whose size is only 1MB, is cost 13 minutes to parse it. But i find that OCR part is 3.31s, Layout analysis is 1.80s, embedding is 6.34s. But is waited about 12 minutes for the task to be received(i used the builtin BAAT/bge-large-zh-v1.5 model), i understand this because at that same time a big PDF file about 55M is being parsed on the same machine with CPU. So what i want to express is that the main spending time should not only include embedding, right?

[writinwaters]

You are right in that embedding is not the only stage that takes time. You toggled on RAPTOR, and this is another time consumer.

BTW, embedding models are for offline processing, it does not contribute to the performance of question answering, which is in real-time.

[zengqingfu1442]

So the OCR runs on CPU by default?

[writinwaters]

Yes, you are right.

[said-what-sakula]

Create a knowledge base is not time-consuming; parsing file is. Here are some tips: https://ragflow.io/docs/dev/accelerate_doc_indexing

Or, you can use docker-compose-gpu.yml to start your service. This accelerates DeepDoc tasks using GPU, requiring RAGFlow v0.16.0+.

I compiled a Docker image from the source code to start. If OCR uses GPU, LIGHT=0 is required. Docker-compose-gpu.yml does indeed use GPU to accelerate ORC, but there will be OOM issues, and in the case of multiple graphics cards, only the first graphics card will be used. Can you solve this problem? Thank you.

[yuzhichang]

@said-what-sakula onnxruntime-gpu leaking memory is a known issue. We've add code according to that onnxruntime issue but it doesn't help. v0.17.0 add Support the use of LLM to parse documents(experimental).

[zengqingfu1442]

Can RAPTOR and Extracting knowledge graph (GraphRAG) also run on GPU to accelerate?

You are right in that embedding is not the only stage that takes time. You toggled on RAPTOR, and this is another time consumer.

BTW, embedding models are for offline processing, it does not contribute to the performance of question answering, which is in real-time.

[writinwaters]

Thanks @yuzhichang for sharing the story behind this. @said-what-sakula You can take a look at Feature 5 of RAGFlow's latest release notes: https://ragflow.io/docs/dev/release_notes#v0170

[writinwaters]

Can RAPTOR and Extracting knowledge graph (GraphRAG) also run on GPU to accelerate?

You are right in that embedding is not the only stage that takes time. You toggled on RAPTOR, and this is another time consumer.
BTW, embedding models are for offline processing, it does not contribute to the performance of question answering, which is in real-time.

GPU can't be used to accelerate RAPTOR and knowledge graph.

[said-what-sakula]

Thanks @yuzhichang for sharing the story behind this. @said-what-sakula You can take a look at Feature 5 of RAGFlow's latest release notes: https://ragflow.io/docs/dev/release_notes#v0170

I didn't expect the update to be so fast. I will go check the update document and try it out