Description
openedon Feb 29, 2024
Prerequisites
- I have written a descriptive issue title
- I have searched existing issues to ensure the bug has not already been reported
Mongoose version
8.2.0
Node.js version
20.x
MongoDB server version
Atlas 7.x
Typescript version (if applicable)
5.2.2
Description
SOLVED - problem due to calls to toObject
on complex hydrated documents, not the transactions.
We noticed massive performance issues in PROD on rather trivial bulk updates. I ran some local tests and narrowed it down to virtual properties in combination with transactions/sessions).
This is the bulkSave
we perform to update our entities.
await this.model.bulkSave(batch, { session: transaction?.session });
Our entities have 4 virtual properties, e.g.:
CredentialSchema.virtual("parentAuthorization")
.get(function() {
return (this as any)._parentAuthorization;
})
.set(function(v) {
(this as any)._parentAuthorization = v;
});
For my tests, I ran the bulkSave
- with the virtual properties declared in the schema, and populated (the problem)
- with the virtual properties declared in the schema, but set the field values to
undefined
before running the update - without virtual properties in the schema, and setting field values to
undefined
before running the update.
2 and 3 resulted in the same performance, so I'll just list measures for 1 and 2 going forward.
Test Results
Here's some measurements with a clear outlier:
- Updating 100 entities with populated virtual properties and no transaction: < 150 ms
- Updating 100 entities with populated virtual properties within a transaction: ~ 4 seconds
- Updating 100 entities without populated virtual properties and no transaction: < 150 ms
- Updating 100 entities without populated virtual properties within a transaction: < 150 ms
With bigger arrays, the performance degradation is absolutely fatal, resulting in timeouts etc.
A test with 10K entities crashed my process because the JS heap ran out of memory (V8::FatalProcessOutOfMemory+662
))
This looks like a bug to me, as the virtual properties shouldn't even be touched. My guess is that mongoose iterates through those virtual properties in order to detect changes rather than skipping them, which seems to introduce a massive overhead. That wouldn't explain why it only occurs with a transaction/session in the mix.
Right now, this really is a show stopper for us. Is there a better short-time solution than setting those properties to undefined before every update, and then reassigning them?
Thanks!
Steps to Reproduce
- Create a schema with some virtual properties and create 100 documents.
- Modify your entites by changing a flag
- Then create a session and run a
bulkSave
in order to persist the changes.
Note in case it matters: some the virtual properties in our case are related mongoose entities.
Expected Behavior
No performance degradation at all from virtual properties. Also, virtual properties and transactions should not affect each other.