[xo-web:build] Error: JavaScript heap out of memory

[ismo-conguairta]

Context

• XO origin: the sources
• Versions:
  • Node: v12.16.3
  • xo-web: 5.60.0
  • xo-server: 5.60.0
  • xo-os: Debian GNU/Linux 10 (buster)
  • xo-vm-ram: 2 GB

Expected behavior

Building XO from sources ends successfully.

Current behavior

The build fails with the following error:

<--- Last few GCs --->

[2832:0x3225ab0]   137506 ms: Mark-sweep 954.5 (1000.3) -> 954.3 (1000.6) MB, 1396.0 / 0.0 ms  (average mu = 0.100, current mu = 0.027) allocation failure GC in old space requested
[2832:0x3225ab0]   138967 ms: Mark-sweep 959.7 (1000.6) -> 959.1 (1003.6) MB, 1420.8 / 0.0 ms  (average mu = 0.064, current mu = 0.028) allocation failure scavenge might not succeed

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x13c5b79]
Security context: 0x3afff83408d1 <JSObject>
    1: SourceMapConsumer_parseMappings [0x3666946551e1] [/opt/xen-orchestra/node_modules/vinyl-sourcemaps-apply/node_modules/source-map/lib/source-map-consumer.js:~423] [pc=0x176dbe46afd8](this=0x1e5080ecdf19 <BasicSourceMapConsumer map = 0x1d303755f69>,0x1eb937ac0119 <Very long string[4753623]>,0x1a91b7d001b9 <null>)
    2: get [0x217d940ee99] [/opt/xen-orchestr...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20200616.185202.2832.0.001.json
Node.js report completed
 1: 0xa09830 node::Abort() [gulp build]
 2: 0xa09c55 node::OnFatalError(char const*, char const*) [gulp build]
 3: 0xb7d71e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [gulp build]
 4: 0xb7da99 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [gulp build]
 5: 0xd2a1f5  [gulp build]
 6: 0xd2a886 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [gulp build]
 7: 0xd37105 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [gulp build]
 8: 0xd37fb5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [gulp build]
 9: 0xd3aa6c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [gulp build]
10: 0xd0232e v8::internal::Factory::NewUninitializedFixedArray(int, v8::internal::AllocationType) [gulp build]
11: 0xe71561  [gulp build]
12: 0xe71810  [gulp build]
13: 0x101985b v8::internal::Runtime_GrowArrayElements(int, unsigned long*, v8::internal::Isolate*) [gulp build]
14: 0x13c5b79  [gulp build]
Aborted
* xo-web:build − Error: 134
✖ 1
error Command failed with exit code 1.

The issue occurs when you are using nodejs v12.x and the XO-VM has 2GB RAM (as per XOA VM specifications)

The issues doesn't happen if you have these environments:

1. nodejs v10.x, no matter how much RAM is allocated to VM
2. allocating more than 2GB RAM

Workaround

The issue can be mitigated by adding the environment variable specification NODE_OPTIONS='--max_old_space_size=2048' at line :151 in packages/xo-web/package.json (take a look at the patchfile xo-web_build_heap_limit.patch.zip). The resulting line becomes:

"build": "NODE_ENV=production NODE_OPTIONS='--max-old-space-size=2048' gulp build",

For context, Node 12 has a different heap management strategy which is based on available memory instead of using defaults. More details here, here and here

Properly configure default heap limits

This update will configure the JavaScript heap size based on available memory instead of using defaults that were set by V8 for use with browsers. In previous releases, unless configured, V8 defaulted to limiting the max heap size to 700 MB or 1400MB on 32 and 64-bit platforms respectively. Configuring the heap size based on available memory ensures that Node.js does not try to use more memory than is available and terminating when its memory is exhausted.

This is particularly useful when processing large data-sets. As before, it will still be possible to set  -- max-old-space-size to use a different limit if the default is not appropriate for your application.

[julien-f]

I cannot reproduce with Node 12.18.0 on my machine.

Anyway, I don't want to touch xo-web build system, it will be completely revamped for XO6, but if there are issues, I'm open to work-around 🙂

[ismo-conguairta]

I have just updated to Node 12.18.0. The build error is still there.

[olivierlambert]

You just have not enough RAM during the build.

[ismo-conguairta]

Sure, I know. I just wanted to point out that the build could be fail even if you comply with the minimum resource requirements for the XO VM and with the suggested software environment.
My XO-VM has 2GB RAM, there are nothing else that consumes memory. I also tried to stop the xo-service before starting the build, but the error still occurs.

@julien-f, I understand that you don't want to touch xo-web build system. The workaround could also work by exporting the env variable in the session (export NODE_OPTIONS='--max_old_space_size=2048') before the build process starts. Maybe this could be useful as a note in the documentation, unless I'm the only one having the issue... in that case I can't figure out why.

[julien-f]

If this issue impacts other people, we'll add this flag to the build script 👍

[olivierlambert]

Sorry but I can't find in the documentation where we are talking about this 2GiB RAM as suggested environment to XO built on the sources?

[ismo-conguairta]

@olivierlambert, you're right there is no explicit notes about the suggested environment for XO built from the sources. Indeed, I never mentioned it, while I always referred to XOA VM specificatons (as in the issue details).
My assumption is that if those are the XOA VM specifications, then I'll be able to build and use XO-sources into a machine with the same characteristics. Probably, I misinterpreted the documentation (reading what is not written), in that case... my apologies.

Also, probably I'm the only one with such limited resources on my test lab. 😞

[olivierlambert]

No problem. XOA VM is made to run XO, it's not a dev environment 😉 However, our doc should probably add a warning on source/build section about this potential issue. We'll do that!

[Danp2]

We ran into this same issue a few months ago. Discussion is here --
#4787 (comment)

Wasn't aware of NODE_OPTIONS='--max_old_space_size=2048', so will definitely do some testing with that.

[julien-f]

Closing because will be completely fixed with XO6.