Extract ObjectIds without loading ReferenceField

[twiggy]

I have a reference field to a large object. Using lazy loading is causing performance issues as is select_related. I really need to do an in_bulk type operation and then map/join the data myself.

objectids = slist.scalar("sessionexception__distinctlog")

sessionexception is an EmbeddedDocument while distinctlog is a referencefield.

If I do a str() on the objectids array elements I get "DistinctLog" which is the ReferenceField's document type.

The problem is if I then do something like objectids[0].id it loads the entire document.

I can't for the life of me figure out a way to extrace the Object id from the ReferenceField. Someting like

objectids = slist.scalar("sessionexception__distinctlog__id")

gives me an error that I cannot do a join in mongodb. I'm trying to tell mongoengine that I just want the id to come out of the referencefield object...any help would be appreciated. I might just have to do with a raw objectid field type or in addition.

[twiggy]

Found a workaround

str(e.dict["_data"]["distinctlog"].id)

where e is a document that has a distinctlog = referencefield("DistinctLog");

Obviously isn't safe if you were to change internals.

Looking through the code it seems the first time get sees the dbref object it loads it up and then calls get on it essentially. Would it be cool if I just said something like " is Dbref, but field name is id just return DBRef.id" ?

[ddorian]

Maybe make it like django? Django create a referencefield_id for every reference field containing just the id. Or just magically support without querying the referencefield.pk (if other field is requested than make the query?)

[romanlv]

Or maybe simply add another method to the document class, which always returns id without dereferencing

e.g.

e.val('sessionexception__distinctlog')

[rozza]

Sorry for the delay in 0.7.8+ you can use as_pymongo rather than scalar to stop dereferencing.

In 0.8 - You can now use the no_dereference context manager - see: #82
Will change scalar to stop dereferencing in #68

[BobDickinson]

I'm not sure I consider this bug closed. The specific request to be able to extract an ObjectId from a ReferenceField without triggering a load of the underlying document still hasn't been address as far as I can tell. I have this specific problem now. I have an embedded document (actually a leaf node in a tree of embedded documents) that has a ReferenceField that refers to a fairly large document. I have a case where I just need the ObjectId to pass in to another query, and I can't get it without triggering a load of the referenced document. Reloading and navigating that complex document using "as_pymongo" is not a great workaround (there is some point where if I'm using raw and as_pymongo everywhere to work around performance issues, it's time to just throw out MongoEngine, which I really don't want to do).

I think not having ".id" trigger a dereference and just returning the ObjectId would solve my problem, and would probably stop a lot of accidental poor performance from users who don't understand that this kind of thing would cause a dereferencing. I can't really see any problem that this would cause (if you access any real field in the referenced document, fine, dereference it then).

And to be clear, it's not the case that I don't ever want to dereference when accessing this document reference. If I accessed a field of the referenced document that actually required dereferencing, then I would expect it to work like it does now. I just can't afford to have it dereference when it doesn't need to.

This would also allow me to stop using "as_pymongo" in a bunch of queries, where I'm using it specifically to avoid this issue (as you recommended in your comment above).

[rozza]

@BobDickinson sounds like it could be an improvement - however there are some subtleties if you look into a listfield containing References we auto dereference - as we try to do it efficiently as possible. Getting a trigger to do it once you look at a single reference in the list - is going to be tricky.

I think the context manager will fix this for you but will double check.

[rozza]

Added #298 - its a chunk of work so will schedule for 0.9

[jonmcoe]

@twiggy thanks for the str(e.dict["_data"]["distinctlog"].id) workaround. did the trick!

i found it to be a neat solution to add a property for it. something like:

@property
def distinctlog_id(self):
    return str(self.dict["_data"]["distinctlog"].id)

[orarbel]

@jonmcoe thanks for this.
For me this worked:

@property
def owner_id(self):
        return str(self._data["owner"].id)