Mongoengine is very slow on large documents compared to native pymongo usage

[baruchoxman]

(See also this StackOverflow question)

I have the following mongoengine model:

class MyModel(Document):
    date = DateTimeField(required = True)
    data_dict_1 = DictField(required = False)
    data_dict_2 = DictField(required = True)

In some cases the document in the DB can be very large (around 5-10MB), and the data_dict fields contain complex nested documents (dict of lists of dicts, etc...).

I have encountered two (possibly related) issues:

1. When I run native pymongo find_one() query, it returns within a second. When I run MyModel.objects.first() it takes 5-10 seconds.
2. When I query a single large document from the DB, and then access its field, it takes 10-20 seconds just to do the following:
   m = MyModel.objects.first()
val = m.data_dict_1.get(some_key)

The data in the object does not contain any references to any other objects, so it is not an issue of objects dereferencing.
I suspect it is related to some inefficiency of the internal data representation of mongoengine, which affects the document object construction as well as fields access.

[touilleMan]

Hi,

Have you profiled the execution with Profile/cProfile ? A graph of it with objgraph should give us a better view of where the trouble is.

[baruchoxman]

Hi,

Please see the following chart: http://i.stack.imgur.com/qAb0t.png (attached in the answer to my StackOverflow question) - it shows that the bottleneck is in "DictField.to_python" (being called 600,000 times).

[amcgregor]

The entire approach of eager conversion is potentially fundamentally flawed. Lazy conversion on first access would defer all conversion overhead to the point where the structure is actually accessed, completely eliminating it in the situation where access is not made. (Beyond such a situation indicating proper use of .only() and related helpers is warranted.)

[amcgregor]

Duplicate of #1137 ?

[lafrech]

@amcgregor, your help is valuable.

Would you like to join @MongoEngine/mongoengine team and/or get write access to the repo?

Even if you don't intend to contribute code right now, at least you'd be able to act on the bugtracker, close/tag issues.

I can't speak on behalf of the team, but I think a new recruit would be appreciated.

[amcgregor]

@lafrech Unfortunately, I'm going to have to decline the offer. Ref: marrow/contentment#12… though I didn't expect Github to ping each referenced ticket like that. Apologies! O_O

[lafrech]

@amcgregor No pb. Thanks for your inputs, anyway.

[apolkosnik]

I've played a bit with a snippet and with some modifications( https://gist.github.com/apolkosnik/e94327e92dd2e0642e2b263efd87d1b1 ), then I ran it against Mongoengine 0.8.8 and 0.11:

Please see the pictures...
On 0.8.8:
mongoengine with dict took 16.95s

On 0.11:
mongoengine with dict took 32.74s

[apolkosnik]

It looks like some change from 0.8.8 to 0.9+ caused the get() in ComplexBaseField class to go on a dereference spree for dicts().