Repository navigation
Document (or fix?) v8.deserialize' 2gb limitation for input buffer #40059
Description
Activity
After tracking for a while, I found that the constructor of
ValueDeserializercan only acceptsizeargument to be a signed int (on 64-bit unix-like systems, it is 32-bit), or, the construction will be marked ashas_aborted. Meanwhile, constructor ofValueSerializerdoesn't have this limitation. That makesValueSerializercan serialize huge object into a huge buffer larger than 2GB, which cannot be deserialized byValueDeserializer. This is why we getDataCloneDeserializationError, which readsUnable to deserialize cloned data..See line
3273and3281.Lines 3271 to 3283 in 5c1adda
I've tried to simply use
size_t, which ranges from 0 to 264-1 on 64-bit systems, instead ofintas template argument to createValueDeserializerinstance as followingValueDeserializer::ValueDeserializer(Isolate* isolate, const uint8_t* data, size_t size, Delegate* delegate) { - if (base::IsValueInRangeForNumericType<int>(size)) { + if (base::IsValueInRangeForNumericType<size_t>(size)) { private_ = new PrivateData( reinterpret_cast<i::Isolate*>(isolate), - base::Vector<const uint8_t>(data, static_cast<int>(size)), delegate); + base::Vector<const uint8_t>(data, static_cast<size_t>(size)), delegate); } else { private_ = new PrivateData(reinterpret_cast<i::Isolate*>(isolate), base::Vector<const uint8_t>(nullptr, 0), nullptr); private_->has_aborted = true; } }but get
DataCloneDeserializationVersionError, which saysUnable to deserialize cloned data due to invalid or unsupported version..Let me explain why we get this error.
When we try to deserialize a huge buffer, we need to construct a
ValueDeserializerobject first, by calling its constructor. Line 1120 calculates buffer end by addingdata.begin()anddata.length(), and assign the result toValueDeserializer::end_.node/deps/v8/src/objects/value-serializer.cc
Lines 1114 to 1122 in 55379eb
HOWEVER,
data.begin()returns a pointer (64-bit width), whiledata.length()returns an integer (32-bit width). If we have a buffer larger than 2GB,data.begin() + data.length() # known as end_will overflow. This makesdata.begin() # known as position_somehow greater thanend_(line 1143). In this case,version_variable is not set (line 1146), callingGetWireFormatVersion()gets its initial value0.node/deps/v8/src/objects/value-serializer.cc
Lines 1142 to 1153 in 55379eb
Perhaps we can just leave everything original, if we are okay to be unable to deserialize huge objects successfully serialized by v8. This is just a little bit weird, but it is safe.
Or, we can make every object serialized by
ValueSerializerdeserializable.Possible Solution:
- Remove buffer size limit when trying to deserialize buffer. See https://chromium-review.googlesource.com/c/v8/v8/+/3170411
- Document buffer size limit of node.js. See buffer,doc: Throw error instead of assert when buffer too large #40243
You can pull rayw000@d556d61 and #40243 for test. This following code snippet
#!/usr/bin/env node --max-old-space-size=32768 const v8 = require('v8'); const str = `AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA` const obj = {} for (let i = 1; i < 22; i++) { for (let j = 0; j < 2 ** i; j++) { obj[j] = str; } } const buffer = v8.serialize(obj)
will output
node:v8:333 return ser.releaseBuffer(); ^ Error: Cannot create a Buffer larger than 0x100000000 bytes at Object.serialize (node:v8:333:14) at Object.<anonymous> (/Users/rayw000/repo/open-source/test-js/index.js:12:19) at Module._compile (node:internal/modules/cjs/loader:1095:14) at Object.Module._extensions..js (node:internal/modules/cjs/loader:1147:10) at Module.load (node:internal/modules/cjs/loader:975:32) at Function.Module._load (node:internal/modules/cjs/loader:822:12) at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) at node:internal/main/run_main_module:17:47 { code: 'ERR_BUFFER_TOO_LARGE' } Node.js v17.0.0-preReacted by Anna Henningsen and SamliorI'm kinda +1 with @rayw000's idea, we could just remove the limitation and allow the deserialization of all the objects that can be serialized and document it somewhere.
@joyeecheung @addaleax what do you both think?
Reacted by RayI just submitted a patch to Gerrit: https://chromium-review.googlesource.com/c/v8/v8/+/3170411
Reacted by Anna Henningsen- added a commit that references this issue
on Sep 27, 2021 Since #40450 and #40243 are landed, now we can serialize any objects requiring buffer no larger than
kMaxLength. For those requiring buffer larger thankMaxLength, anERR_BUFFER_TOO_LARGEerror will be thrown.
As to deserialization, all buffers do not violate object model can be deserialized, no matter how large they are, if possible.- added 3 commits that reference this issue
on Nov 24, 2021 - added a commit that references this issue
on May 22, 2026
Version
v16.9.0
Platform
Darwin theodore 20.6.0 Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:31 PDT 2021; root:xnu-7195.141.2~5/RELEASE_X86_64 x86_64
Subsystem
v8
What steps will reproduce the bug?
Here is a script that will demonstrate the issue:
And here is the output on my system:
How often does it reproduce? Is there a required condition?
100% of the time if the buffer to deserialize exceeds 2gb.
What is the expected behavior?
Either the documentation is updated to reflect the limit or the limit is removed.
What do you see instead?
Please see the output above.
Additional information
v8's
serializeanddeserializeare fantastic and very performant for large datasets; much faster than the alternatives like e.g.msgpackr. I'd love to continue using them as my dataset grows so that I can put off re-architecting things a bit further, but I realize it's probably a very extreme use case. Documenting the limitation will save the next person who hits it some time, however! I spent a lot of time assuming that my file writing code was broken in some way (which I had to rewrite because it also has a 2gb limitation; I hit both limits at the same time but didn't realizedeserializehad the same limit).In the short term I will probably shard keys across multiple files.