Error when writing huge of content to a file json

[trungducng]

<--- JS stacktrace —->

==== JS stack trace =========================================

Security context: 0x3b902a9266a1 <JS Object>
    1: fromString(aka fromString) [buffer.js:~194] [pc=0x1b9dcb6f3ad0](this=0x2f5cceb02311 <undefined>,string=0x866865dc389 <Very long 
string[14371]>,encoding=0x3b902a92f309 <String[4]: utf8>)
    2: from [buffer.js:~96] [pc=0x1b9dcb6c9936](this=0x2bea42122009 <JS Function Buffer (SharedFunctionInfo 0x3b902a94b481)>,value=0x86
6865dc389 <Very long string[14371]>,encodingOrOffset=0x3b902a92f309 <...

FATAL ERROR: CALL_AND_RETRY_LAST All

Error when writing huge of content to a file json, Can anyone help me solve this propblem? I tried

node --max-old-space-size=8192 index.js

But it still help me about out of memory in heap problem

[vsemozhetbyt]

Could it be related to String size limit? See #9489

What is String length you try to output in JSON?

[trungducng]

@vsemozhetbyt The JSON file is more than 500mb when I carry out writing

[vsemozhetbyt]

Could you provide a small code fragment to reproduce the issue?

[vsemozhetbyt]

cc @nodejs/buffer ?

[addaleax]

Yes, you are either hitting the string length limit (which I thought was 256 MB?), or are going out of memory, neither of which are very unlikely when handling strings this huge.

I agree with @vsemozhetbyt, in order to see if we are even able to help you, you’d need to provide enough code to understand exactly what is going on. You might also want to look into the possibility of using a streaming JSON module to handle your data.

[vsemozhetbyt]

@addaleax

de facto limit for the current v8: 268,435,440 characters (Math.pow(2, 28) - 16), 536,870,880 bytes in UTF16.

(#9489)

These issues repeat themselves. Maybe we should add something about String limit to docs (buffer.md?), with some workarounds like the mentioned streaming module.

[addaleax]

Maybe we should add something about String limit to docs (buffer.md?), with some workarounds like the mentioned streaming module.

Sounds reasonable, yes. Maybe we should also export V8’s String::kMaxLength to JS, like buffer.kMaxLength, what do you think?

[vsemozhetbyt]

Will this allow to increase the String limit for v8? It is hard for me to assess the effects.

cc @nodejs/v8 ?

[addaleax]

@vsemozhetbyt I’m talking about just providing the value to JS scripts, not to make it mutable (I have no idea if that would work). That might be useful for giving programmers a way to tell whether something can even be turned into a string, or whether it’s too large.

[tniessen]

Why is this limit in place? Why doesn't v8 allow arbitrary strings as long as it does not run out of heap space?

[addaleax]

That’s really a question for @nodejs/v8. ;)

[vsemozhetbyt]

Maybe we should also foresee the cases when using this check is not so obvious. For example:

fs.writeFileSync('file.json', JSON.stringify(hugeObject), 'utf16le');

Maybe some checks should be carried out by buffer or fs themselves, with a clear error message.

[addaleax]

@vsemozhetbyt I don’t quite get your example here. By the time JSON.stringify is finished, the string has already been created, and writing the data should work fine as far as Node is concerned. If you are talking about JSON.stringify throwing a certain (better?) error… I don’t know, but that’s not under Node’s control, right?