[Question]: Distributed file parse across multiple ragflows and machines for knowledge base construction？Improve parsing speed

[charmmy-workstation]

Describe your problem

my ragflow version: 0.14

I found that most of the time to build a knowledge base spent on file parsing .Can I scatter the analytical task to multiple Ragflow and multiple machines, and finally merge them again?

A single file parsing takes a staggering 2h in the task queue and analysis.

[KevinHuSh]

Of course you can. Start mutiple task executor by python rag/svr/task_executor.py 1,2,3....

[charmmy-workstation]

Thank you. I used Docker to run RagFlow, and now I can perform multiple file parsing tasks simultaneously through the following methods.

path/to/ragflow$ docker exec -d ragflow-server python rag/svr/task_executor.py 1
path/to/ragflow$ docker exec -d ragflow-server python rag/svr/task_executor.py 2
path/to/ragflow$ docker exec -d ragflow-server python rag/svr/task_executor.py 3
...
path/to/ragflow$ docker exec -d ragflow-server python rag/svr/task_executor.py 10

Can be seen in the panel

I think this should be added to the FAQ!

[charmmy-workstation]

These are other solutions:

Slow file parsing

The process of refining Question is slow

I am interested in ES optimization in it

Now use infinityflow

[19920716]

Consider referencing the Celery distributed Task framework @KevinHuSh

[charmmy-workstation]

I have a 30MB txt file that needs parsing, and here's the time it took:
Using multiple task_executor threads sped up the slicing file process. However, the embedding of the content stage took an exceptionally long time. I am using bge-m3 for this task, which has an 8k max token limit. I used xinference to run bge-m3. The image above shows the execution time of bge-m3 on a single GPU. When I switched bge-m3 to using 4 GPUs, there was no improvement in execution time, as shown in the following times.

Are there any suggestions?

[iot-future]

When I start the project with ‘docker-compose-gpu.yml’, I also use ’python rag/svr/task_executor.py 1,2,3.... ’.After multiple task actuators are enabled, some task parsing error occurs. The error message is

19:00:41 Page(1~100000001): [ERROR]Fail to bind embedding model: NCCL Error 2: unhandled system error (run with NCCL_DEBUG=INFO for details)
19:00:41 [ERROR][Exception]: NCCL Error 2: unhandled system error (run with NCCL_DEBUG=INFO for details)

[KevinHuSh]

FYI