[Bug]: MongoDB does not get all the data from a heavy query

[felix-appsmith]

Is there an existing issue for this?

• I have searched the existing issues

Description

A user has a Mongo collection of 3 documents, each document contains approximately 30 images. The problem is that when executing the find function, only 1 is obtained, but when the count function is executed, it obtains 3
I attach images so that you can observe the case.

Steps To Reproduce

``
1. Create a new database.
2. Create a new collection within the database.
3. Add 3 new documents to the collection using this Python script:

    from pymongo import MongoClient
    from bson.objectid import ObjectId
    from bson.binary import Binary
    
    client = MongoClient("mongodb://localhost:27017")
    db = client["Git_19887_Test"]
    col = db["Test_Upload_Images"]
    
    def read_image(file_path):
        images = []
        for i in range(1, 21):
            with open(file_path, 'rb') as f:
                binary_image = Binary(f.read())
            new_file_path = file_path.replace('.jpg', f'_{i}.jpg')
            images.append((new_file_path, binary_image))
        return images
    
    document1 = {"Inputs": [1, 2, 3], "works": {"work1": "yes", "work2": "no"}, "Features": {"feature1": True, "feature2": False}, "images": [], "images2": [], "_id": ObjectId()}
    document2 = {"Inputs": [4, 5, 6], "works": {"work1": "no", "work2": "yes"}, "Features": {"feature1": False, "feature2": True}, "images": [], "images2": [], "_id": ObjectId()}
    document3 = {"Inputs": [7, 8, 9], "works": {"work1": "yes", "work2": "yes"}, "Features": {"feature1": True, "feature2": True}, "images": [], "images2": [], "_id": ObjectId()}
    
    for document in [document1, document2, document3]:
        images = []
        images2 = []
        for i in range(1, 21):
            image_file_path = "../1.jpg"
            image_list = read_image(image_file_path)
            for new_file_path, binary_image in image_list:
                images.append({new_file_path: binary_image})
            image_file_path = "../1.jpg"
            image_list = read_image(image_file_path)
            for new_file_path, binary_image in image_list:
                images2.append({new_file_path: binary_image})
        document["1"] = images
        document["1"] = images2
    
col.insert_many([document1, document2, document3])

``
4. Use a sample image with a size of 33.3kb, for example, this image.

`
5. Now in Appsmith, if we execute the "find" command, we obtain a single result, but if we execute the "count" command, we get all 3 results.

Public Sample App

No response

Issue video log

No response

Version

Appsmith Community v1.8.9

[barshag]

@sribalajig Why did you lowered the severity to medium?
This is absolutely not an UX problem as the fetch from the DB is not working correctly.
It's maybe even some kind o f "critical" in the sense that this lead to misleading information in some situation where not all the data is presented (and they even not aware of it..).

[sribalajig]

@barshag can you please help me understand whether the images array is a list of URLs or actual binary data representing the images? Also is this issue is happening for only this collection or other collections as well?
In the mean time, I have bumped this back to high priority. Thanks.

[barshag]

It's binary (BSON) objects (not URLS) I checked it only in this collection, but in several instances of this collection. Thanks for the change for HIGH! :) Any approx ETA for solving this issue?

…

On Fri, 20 Jan 2023 at 18:11, Sri Balaji Gopinath ***@***.***> wrote: @barshag <https://github.com/barshag> can you please help me understand whether the images array is a list of URLs or actual binary data representing the images? Also is this issue is happening for only this collection or other collections as well? In the mean time, I have bumped this back to high priority. Thanks. — Reply to this email directly, view it on GitHub <#19887 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO6X5HOCYP7HZ7VUX6V47PLWTK2KXANCNFSM6AAAAAAT7S4TEE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[sribalajig]

@barshag we have prioritised this issue for implementation and it is going to be a part of a larger epic which deals with Appsmith's ability to deal with large data volumes - #18245 (Note : The epic deals with large file uploads, and your problem is with queries. Although not exactly the same, they are related)

Would you be interested in having a discussion with me on your problem? I think it would give a good perspective for dealing with the larger epic as well. You can find my calendar here - https://calendly.com/balaji-gopinath/appsmith-user-interview

In the mean time, if possible, I would suggest storing images in a separate place (like Amazon S3) and referencing the URLs in your Mongo documents. Mongo is not really optimised to store large binary objects.

[nidhi-nair]

@barshag From the steps for reproduction given on this issue, it seems like the result Appsmith is coming back with is limited because to total array of documents exceeds 16 MB which is greater than the allowed batch size for Appsmith. Our Mongo plugin today only allows for the first batch of data to be loaded. We understand that this information is not best represented on the screen, but do you think paginating with this limitation in mind would help your use case?

[barshag]

Not really. It could, but In my case i use GridFS that supports more than 16MB.... Pagination is okay to be limit - but it should support DB with much larger size

…

On Mon, 17 Apr 2023 at 13:00, Nidhi ***@***.***> wrote: @barshag <https://github.com/barshag> From the steps for reproduction given on this issue, it seems like the result Appsmith is coming back with is limited because to total array of documents exceeds 16 MB which is greater than the allowed batch size for Appsmith. Our Mongo plugin today only allows for the first batch of data to be loaded. We understand that this information is not best represented on the screen, but do you think paginating with this limitation in mind would help your use case? — Reply to this email directly, view it on GitHub <#19887 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO6X5HJ4TXDZNQC3XN4BMU3XBUIFFANCNFSM6AAAAAAT7S4TEE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[nidhi-nair]

Do you mind sharing what the size of these documents that you are working with is?

I am also looping in @rohan-arthur and @sribalajig to figure out a more comprehensive solve for this.

[akshayvijayjain]

based on the comments, few points to take care while picking this issue

1. need to store 5 images/media in documents, each around 5 mb, and try to fetch them all
2. are we able to fetch all 25 mb, probably not, then how much we are able to fetch
3. what possible solution can we add to be able to fetch all 5 image, or any number images
4. what is possibility of notifying user, if the fetched data is less than actual existing data