Very poor file handling quality

[creatorofuniverses]

📦 Platform

Self hosting Docker

📦 Deploymenet mode

server db(lobe-chat-database image)

📌 Version

v1.79.6

💻 Operating System

Ubuntu

🌐 Browser

Chrome

🐛 Bug Description

The chat has the option to upload a file and send it to the model to work with it

And for the 4 different APIs I've tried, the quality of the model's response is as if the file had been verbally translated into Ancient Egyptian and back again.

📷 Recurrence Steps

1. Take a file with a scientific publication (paper)
2. Send it to the OpenAI website chat and ask for a structured summary through the system prompt.
3. Repeat the same for the same model in lobechat.

And there is a huge difference, the model in lobechat hallucinates, misses key elements and speculates wherever it can, while OpenAI model - acts as expected - very nice and useful summary

🚦 Expected Behavior

Same output as in original model provider service

📝 Additional Information

It works the same way with Claude, Gemini and Grok

The system agent settings have no effect, the result is the same.

Apparently the question is how lobechat processes the file and in what form it actually gives the models.

Either the model does not match there to get a vector representation, or the text is compressed and reduced to impossible, but one way or another it is impossible to use it.

(Unfortunately I can't give you examples right now, as my lobechat instance is not yet available due to ISP issues, but I will provide examples if possible. )

[lobehubbot]

👀 @creatorofuniverses

Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible.
Please make sure you have given us as much context as possible.
非常感谢您提交 issue。我们会尽快调查此事，并尽快回复您。 请确保您已经提供了尽可能多的背景信息。

[dosubot]

Hello @creatorofuniverses! I'm Dosu, here to help you solve bugs, answer questions, and contribute to your experience while you wait for a human maintainer. How can I assist you today?

Lobe-chat processes files through several steps before sending them to the model, which might affect the quality of the model's response. Here's a brief overview of the process:

1. File Hashing and Existence Check: The file is hashed to check if it already exists in the system. If it does, the existing metadata is used ^[1].

2. File Upload and Type Detection: If the file is new, it is uploaded, and its type is detected using a powerful file type detector ^[1].

3. Database Entry: The file's metadata, including type, hash, name, size, and URL, is stored in the database ^[1].

4. File Processing: After uploading, the file undergoes additional processing steps such as chunking and embedding, which are crucial for preparing the data for model usage ^[1].

The issue you're experiencing might be related to how the file is processed and represented before being sent to the model. Unfortunately, specific details on chunking, embedding, or vector representation optimizations in lobe-chat are not available in the current documentation. This could be a factor in the discrepancies you're observing between lobe-chat and the original model provider services.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[Sun-drenched]

As far as I know, the knowledge base enhancement is already in the plan, which will allow you to input the full text of the document to the large model to improve the quality of the answer.