[BUG] Memory Leak in Cashier Cat's Rabbit Hole Causes Container and Host Crashes with Large Data Uploads

[dell1503]

Describe the Bug
Memory overflow occurs when handling large amounts of data.

When using the Rabbit Hole with big data, the container's memory is not released properly. This leads to the container or even the host (if no memory limit is set) crashing. My Cashier Cat instance is using an external Qdrant DB.

To Reproduce
Steps to reproduce the behavior:

1. Set the memory limit of the container to, for example, 2 GB.
2. Navigate to Cashier Cat's home page.
3. Upload one document (e.g., a 300MB PDF).
4. Check the logs to ensure everything is functioning correctly.
5. Upload one or multiple documents (uploading four 300MB PDFs was sufficient for me).
6. Monitor the RAM consumption of the container and review the logs.
7. Wait until the process crashes.

Logs When Uploading Multiple Documents and Crashing
Below are the logs showing the latest successful upload before the crash:

2024-10-02 07:23:06 [2024-10-02 05:23:06.526] WARNING cat.rabbit_hole.RabbitHole.store_documents::402 
2024-10-02 07:23:06 "Done bigfile.pdf"
2024-10-02 07:23:38 INFO:     172.18.0.3:47032 - "GET /rabbithole/allowed-mimetypes HTTP/1.0" 200 OK
2024-10-02 07:24:06 INFO:     172.18.0.3:58416 - "POST /rabbithole HTTP/1.0" 307 Temporary Redirect
2024-10-02 07:24:07 INFO:     172.18.0.3:58424 - "POST /rabbithole HTTP/1.0" 307 Temporary Redirect
2024-10-02 07:24:08 INFO:     172.18.0.3:58434 - "POST /rabbithole HTTP/1.0" 307 Temporary Redirect
2024-10-02 07:24:10 INFO:     172.18.0.3:58466 - "POST /rabbithole HTTP/1.0" 307 Temporary Redirect
2024-10-02 07:24:11 INFO:     172.18.0.3:58470 - "POST /rabbithole HTTP/1.0" 307 Temporary Redirect
2024-10-02 07:24:13 INFO:     172.18.0.3:58476 - "POST /rabbithole HTTP/1.0" 307 Temporary Redirect
2024-10-02 07:24:16 [2024-10-02 05:24:16.159] INFO   cat.routes.upload..upload_file::102 
2024-10-02 07:24:16 "Uploaded application/pdf down the rabbit hole"

Expected Behavior
Cashier Cat should manage memory efficiently, releasing it appropriately to prevent memory leaks and crashes.

Additional Context
The issue persists regardless of whether the container's memory limit is set to 2 GB or 12 GB; the crash occurs at a later point. I attempted to optimize rabbit_hole.py, but I'm not sure if the problem originates there. Attached is a graph showing memory consumption over time.

Workaround

1. Split large PDFs into smaller MD files.
2. Upload a batch of files, such as 15 files.
3. Restart Cashier Cat.
4. Repeat the process as needed.

Memory usage over time.