[BUG] Status: 500 Error: documentStoreServices.insertIntoVectorStore - Error: documentStoreServices._insertIntoVectorStoreWorkerThread - AggregateError

[srdrkbs]

Describe the bug
I get 500 error when i try to upsert ollama embedings to redis store

To Reproduce
Steps to reproduce the behavior:

1. Go to 'Document Store'
2. Click on 'Embedings, search for ollama embedings and select. set local ollama parameters'
3. Next select Vector Store, search for redis, set redis parameters. after that click upsert'
4. See error 'Status: 500'

[Cirr0e]

Hey there! Looking at the 500 error you're getting with the Redis vector store upsert, I can help you resolve this.

The error is occurring in the vector store worker thread, which suggests there's an issue with either the connection or configuration parameters. Let's fix this step by step:

1. First, verify your Redis connection parameters are correct:

   • Check if Redis server is running and accessible
   • Verify the connection URL/host
   • Ensure the port is correct and accessible
   • Confirm any authentication credentials if required

2. For the Ollama embeddings configuration:

   • Make sure the Ollama service is running and accessible
   • Verify you're using a supported model for embeddings
   • Check that the model parameters match your local Ollama setup

3. When configuring in the UI:

   • In 'Document Store', double-check your Ollama parameters first
   • For Redis settings, ensure all required fields are properly filled
   • Try testing the connection to both services independently before upsert

The error suggests there might be an aggregate error, which typically means multiple things went wrong. Most commonly, this happens when either:

• Redis connection fails
• Ollama embedding generation fails
• Data format is incompatible between services

This is supported by similar issues I found in the codebase, particularly in documentStoreServices._insertIntoVectorStoreWorkerThread.

Could you share:

1. Your Redis connection details (excluding sensitive info)
2. The Ollama model configuration you're using
3. The complete error message if there's more

This will help me provide more specific guidance for your setup.

References:

• Error handling in document store services from server/src/services/documentstore/index.ts
• Similar Redis integration issues from previous tickets

Let me know if you need any clarification or have questions about these steps.

[shahzans]

I'm getting the same error, when trying to upsert vectors to Pinecone.

Status: 500
Error: documentStoreServices.insertIntoVectorStore - Error: documentStoreServices._insertIntoVectorStoreWorkerThread - AggregateError

[Cirr0e]

Hey! This is actually a known issue that's been fixed in v1.6.3. From looking at similar reports, it was related to the Pinecone dependency version. Can you upgrade to v1.6.3 or later? That should resolve the upsert error you're seeing.

If you're still seeing issues after upgrading, let me know and we can look into other potential causes.

(Reference: issue #1977 where this was resolved by pinning the Pinecone dependency)

[shahzans]

I'm on the latest release of Flowise. Looks like there was a glitch, deleted and created a new document store and it worked fine. Thanks for your help :)

[shahzans]

I found what is causing this issue. It is the Record Manager within the Document Store which is causing this issue. Without configuring the Record Manager, the upsertion works well. Once you configure the Record Manager (Supabase) and try to upsert then it throws the below error.

Status: 500
Error: documentStoreServices.insertIntoVectorStore - Error: documentStoreServices._insertIntoVectorStoreWorkerThread - AggregateError

After that even if you reset and remove the Record Manager configuration, even then it wouldn't allow you to upsert any documents. Looks like a bug to me.

[srdrkbs]

Hi guys,
I tried something diffrent. I decided to change the upserting document because error can be produced by document itself. and my tryout was succesful.
When I change the document which I try to upsert I could succesfully upsert the document. so my document which i successfully upserted is small text file. when i try to upsert another large pdf file was unsuccesfull. the couse maybe happening because of character set i'm using turkish. but same character set i used in minimal text file was succesfull.

So my opinion there is a bug with pdf files and character sets when upserting. could you try same please. And can you help me where i can find the upsertation logs. I dont know the path.

[Cirr0e]

After looking at your recent feedback about the PDF files and Turkish characters, I can see what's likely happening here. Let me break this down:

1. The issue appears to be related to PDF processing and character encoding, specifically when:
   • Working with larger PDF files
   • Handling non-ASCII characters (Turkish in your case)
   • Combined with the Record Manager configuration

Here's what you can try:

// 1. First, test without Record Manager
1. Remove the Record Manager configuration completely
2. Try the upsert with your PDF file
3. If it works, you can gradually add back the Record Manager

// 2. If you need to use Record Manager, try this workaround:
1. Split your PDF into smaller chunks first
2. Use the 'Legacy Build' option in the PDF loader settings
   - In Document Store > PDF File settings
   - Enable "Use Legacy Build" under Additional Parameters

This should help because the code shows that the PDF loader has a legacy build option specifically for compatibility issues (from the codebase: packages/components/nodes/documentloaders/Pdf/Pdf.ts).

For finding the upsert logs:

• Check your application logs in the server directory
• The error should be logged under documentStoreServices.insertIntoVectorStore

A few important notes:

1. The PDF processor uses pdf-parse under the hood which can sometimes struggle with certain character encodings
2. The error you're seeing suggests a race condition between the PDF processing and vector store insertion
3. Large PDFs with special characters can cause memory issues during the embedding process

Let me know if you need help finding the logs or if you want to try a different approach!

References:

• PDF Loader implementation: packages/components/nodes/documentloaders/Pdf/Pdf.ts
• Document Store Services: packages/server/src/services/documentstore/index.ts

Hope this helps solve your issue!

[srdrkbs]

I followed the steps shown in the attached images and encountered the same issue every time. However, the error I received this time differs from the one mentioned in the title. Instead of an aggregate error, I am now encountering a null error.

Here are the detailed steps I followed:

1. I created a .txt file containing a simple "Lorem Ipsum" text.
2. I then converted the same content into a PDF file.
3. To test, I first added the .txt file to the document store and performed an upsert operation successfully.
4. My Redis configuration is exactly as shown in the image, and I made no changes to it.
5. Next, I attempted to add the PDF file (containing the same "Lorem Ipsum" text) to the document store using the same setup.
6. When I tried to perform the upsert operation with the PDF file, I immediately received a null error without any further details or explanation.
7. I'm not using record manager.

This is text content ;
`
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec gravida pulvinar nisi et vulputate. Mauris sit amet malesuada orci, at rhoncus lorem. Ut orci nulla, euismod vel leo in, vehicula luctus sem. Fusce tempor suscipit quam, a sagittis felis sodales convallis. Praesent luctus nisl id rhoncus interdum. Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec vitae sapien mattis, mollis orci at, tristique leo. Ut ante eros, tincidunt at pulvinar vitae, euismod eu enim. Aenean sodales, enim at pharetra interdum, leo sem pellentesque diam, at condimentum arcu eros vel mauris. Suspendisse in venenatis tellus. Proin fringilla nisi et sollicitudin cursus. Morbi nec accumsan massa. Aliquam bibendum risus ut ligula auctor, a elementum diam accumsan. Vivamus sagittis, quam eu volutpat feugiat, tellus velit ornare odio, a maximus tortor turpis quis metus. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae;

Vestibulum facilisis luctus nulla quis tristique. Curabitur et hendrerit ipsum. Integer quis viverra ligula. Aliquam fermentum odio justo, sed euismod diam mattis quis. Curabitur nisi est, consequat at consequat eget, malesuada ut sapien. Phasellus vestibulum erat quis neque euismod mollis. Quisque in felis egestas, vestibulum turpis eu, vulputate ligula. Nam semper gravida quam non faucibus. Integer porttitor, felis vel pharetra rhoncus, magna mauris fringilla metus, vel venenatis felis turpis quis nunc. Phasellus ac bibendum ligula, id tincidunt mauris. Nam a libero dapibus, hendrerit arcu sit amet, dignissim elit. In malesuada semper rutrum. Mauris pulvinar dictum augue, cursus auctor erat aliquam vel. Suspendisse bibendum ac turpis ac faucibus. Sed porttitor ante ex, vel lacinia nibh facilisis in. Quisque cursus aliquam molestie.

Pellentesque pellentesque accumsan sem ut porta. Suspendisse vulputate magna non quam vehicula, tempus ultricies enim scelerisque. Aliquam erat volutpat. Proin rutrum nisl purus, ac tincidunt orci posuere eu. Praesent volutpat tincidunt dignissim. Vestibulum congue purus augue, at cursus ante ornare nec. Morbi sed varius dolor, id rutrum arcu. Proin pretium vestibulum nisl, fringilla finibus lorem aliquet non. Integer tortor ligula, suscipit a diam vel, gravida semper mi. Phasellus blandit sapien sodales sapien suscipit faucibus. In hac habitasse platea dictumst. Vivamus a tincidunt justo.

Cras interdum sapien turpis. Etiam tristique egestas quam sit amet ultricies. Donec neque eros, mollis sit amet blandit sed, sollicitudin eget risus. Duis lectus orci, molestie sit amet malesuada ac, bibendum et est. In sodales lacus ac nulla hendrerit elementum. Proin cursus pharetra diam, vitae placerat velit fringilla quis. Donec nec cursus quam, et tempor diam. Nam condimentum augue sit amet massa viverra laoreet. Pellentesque sollicitudin dictum augue, ut tincidunt dolor.

In tempus scelerisque sagittis. Sed ac mi elit. Nullam blandit placerat tellus in ullamcorper. Pellentesque in nisl eget sem accumsan malesuada a quis massa. Nulla tempus elit non nulla maximus, nec tempor nulla porttitor. Sed cursus justo id urna bibendum vehicula. Pellentesque vestibulum ornare eros vel dictum. Vestibulum sed felis mauris. In at tortor eu tortor tincidunt consequat sed eget justo. Sed non leo sed mi lacinia gravida vitae eget justo. Nunc condimentum nisi eget nunc tincidunt, nec scelerisque tellus luctus.
`

[HenryHengZJ]

should be fixed with this PR: #3789

[srdrkbs]

It worked thanks.