bytecode-only SEA caching?

[kvakil]

Currently SEA is focused on snapshotting. However it can be difficult to make a snapshot because (1) not everything is supported and (2) it can be hard for the developer to disentangle the various stages of startup into "snapshottable" and "not snapshottable" parts.

On the other hand V8 bytecode caching always works. In my testing bytecode caching can speedup initialization of some common Node executables (like yarn, npm) by ~10%. This is a pretty modest improvement but it's nearly free.

I think it would be interesting if SEA could also support "bytecode-only" caching, where:

• the source code is stored into the executable and loaded as a V8 external string,
• the bytecode is stored into the executable and used to speed up compilation.

This also has minor benefits for memory usage as the source code memory can be shared across different executables.

Alternatives:

• bytenode exists, but it's focused on obfuscation use cases, and not faster loading. Because of this it doesn't work in some cases, which makes it difficult to recommend as a general solution.
• v8-compile-cache exists and has been used by various projects. However it is pretty hacky in how it works (by monkey-patching require) and also is slower than it could be (since user-space require is pretty slow).
• (You could also imagine Node.js persisting the cached bytecode to disk automatically like Python does with __pycache__, but that's sort of disjoint from this.)

[RaisinTen]

bytenode exists, but it's focused on obfuscation use cases, and not faster loading. Because of this it doesn't work in some cases, which makes it difficult to recommend as a general solution.

To be clear, the cases you're referring to where bytenode doesn't work are the ones that are listed in https://github.com/bytenode/bytenode#known-issues-and-limitations, right?

[kvakil]

bytenode exists, but it's focused on obfuscation use cases, and not faster loading. Because of this it doesn't work in some cases, which makes it difficult to recommend as a general solution.

To be clear, the cases you're referring to where bytenode doesn't work are the ones that are listed in https://github.com/bytenode/bytenode#known-issues-and-limitations, right?

Yes. I don't know of any other cases where bytenode fails, but it's possible they exist.

[kvakil]

Here's a benchmark for the runtime of yarn help for various compilation strategies:

1. yarn-3.5.1.cjs: cjs bundle of all of yarn's source code
2. cached_yarn.js: uses a Node.js native module which has yarn's cjs bundled source code & a code cache (with kEagerCompile) embedded in the shared library.
3. yarn: non-bundled yarn source code. (This uses v8-compile-cache.)
4. bytenode yarn-3.5.1.jsc: executed the result of bytenode -c on the cjs bundle yarn-3.5.1.cjs

$ hyperfine -N -w 30 -L cmd 'node yarn-3.5.1.cjs help','node cached_yarn.js help','yarn help','bytenode yarn-3.5.1.jsc help' '{cmd}'
Benchmark 1: node yarn-3.5.1.cjs help
  Time (mean ± σ):     144.2 ms ±   1.4 ms    [User: 168.3 ms, System: 12.5 ms]
  Range (min … max):   142.9 ms … 147.5 ms    20 runs
 
Benchmark 2: node cached_yarn.js help
  Time (mean ± σ):      95.8 ms ±   1.3 ms    [User: 123.0 ms, System: 10.8 ms]
  Range (min … max):    93.7 ms …  99.4 ms    31 runs
 
Benchmark 3: yarn help
  Time (mean ± σ):     110.0 ms ±   1.6 ms    [User: 88.9 ms, System: 14.4 ms]
  Range (min … max):   107.8 ms … 114.0 ms    27 runs
 
Benchmark 4: bytenode yarn-3.5.1-patched.jsc help
  Time (mean ± σ):      73.1 ms ±   1.5 ms    [User: 53.0 ms, System: 10.4 ms]
  Range (min … max):    69.5 ms …  77.7 ms    41 runs
 
Summary
  'bytenode yarn-3.5.1-patched.jsc help' ran
    1.31 ± 0.03 times faster than 'node cached_yarn.js help'
    1.50 ± 0.04 times faster than 'yarn help'
    1.97 ± 0.04 times faster than 'node yarn-3.5.1.cjs help'

so bytenode is around 50-97% faster than regular yarn, which is pretty impressive. Replacing the source code with "fake" code avoids a lot of parsing overhead, which means that it's even faster than actually using the source code.

This also made me look more at bytenode, which made me see that it uses the V8 flags --no-lazy / --no-flush-bytecode. This will increase memory usage but that's probably acceptable for many use cases. It also means it will stop working if V8 removes either of those flags, which is not great from a stability perspective.

Anyway, SEA-style bytecode caching would give a solid 13-33% startup improvement, depending on exactly what your baseline is. I may try to see how fast snapshotting would be as a next step, but so far it seems pretty difficult to get yarn into a snapshottable form.

[RaisinTen]

@kvakil thanks for sharing the numbers! I've submitted a PR for this - nodejs/node#48191.