Research Discussion: Faster startup with dynamically generated custom v8 snapshots

If you know about v8 snapshots, and node's 'default' snapshot, you might be able to provide insight into this discussion.

I seem to be using more and more 'commands' that are node based. When I run them, like npm, it's my understanding node has to reload and recompile the entry.js and all its require()d modules every time.

I'm wondering if just before node exited, if it could take a v8 snap and save it to a file named to correlate to the full path of entry.js. (obviously optimized to only snap when something changed however, so not every time node exited)

Then, when node was launched again, it could look for a saved snap based on entry.js and just create a new context from that snap, effectively getting to an executing state much faster.

At a high level I'm wondering if my understanding is correct, that a v8 snap is a heap dump that can be reloaded into an isolate, and that when a new context is created in the isolate it starts with modules already jitted, having come from the snap. But wouldn't there be some state that was 'left over' from the original snap that could potentially 'infect' the new context in some non deterministic way? Like say a module set a flag within itself, then the snap was taken, then when reloaded the flag would still be set?

Is there even a possibility, in some way, to save something from run to run, such that entry.js and all require()'d modules don't have to be parsed/jitted every time?

Thinking some more, I can't see how state from a previous snap couldn't potentially 'infect' a later context. So maybe entry.js would just have to tell node that it's 'agreeing' to be snapped (i.e explicit opt-in to the above described behavior), and will deal with 'resetting' it's global state as necessary. Like maybe it could just call a method vm.snapable() at any point, activating the snap-on-exit behaviour, etc.

Any thoughts?

[mscdex]

Well, vm already has a way to produce/use binary code cache data. Is that what you're after or ?

Are you referring to this vm in node?

There's nothing there that lets you generate a v8 snapshot, nor load one. Am I missing something?

@mscdex Could you also mark as a discussion too? I think I could implement something like this, but looking to others to get their thoughts first.

For example, it could very well speed startup times, but it creates a potential for whats loaded from the snap becoming out of sync with the actual source .js, like if a dependency was updated.

There could/should be a way to 'clear' the snap. Or maybe require calls could be tracked and saved with the snap, and then checked for changed modified times on startup, but things like npm might have hundreds or thousands of dependency modules, so checking each may work against the whole point of the thing.

There also needs to be a way to detect if a new snap must be made on exit, for example npm dynamically require()s modules based on command line args (most likely to also speed startup), which would mean tracking require calls, and overhead, and again working against the whole point of the thing.

I guess this first thing I can try is a POC to see if the time savings on startup are even worth pursuing further... collect some metrics.. Just let it snap on exit every time, add a '--clear-snap' switch. Can try with npm -v, which seems to take quite a long time just to spit out the version.

[mscdex]

@phestermcs It's probably not a V8 snapshot proper. The vm options I was referring to specifically were cachedData and produceCachedData. Also, see this V8 blog post about it.

I see now; didn't catch that!. That would be another way to go as well, maybe even a better, simpler way. In fact I think I can tweak node's module.js module to POC it and see if there's a big difference. It would address the issue of reloading state, and therefore could in theory be applied automatically rather than an entry.js having to opt into snapping.

I'm still curious just how much of a boost a snap would have though.

Thanks for the clarification!

@mscdex Would you mind adding the 'discussion' label to this issue?

I tweaked node and made a POC that creates and loads codeCachedData files for each '.js' file that gets require()d; the first require() will generate an adjacent '.js.bin' file to the '.js' file, and subsequent require()s from separate executions will then load the codeCachedData from the '.js.bin' file.

Unfortunately, and surprisingly, I did not notice any improvement in startup times running things like npm, tsc, a couple others; there might have been some, but it was not detectable to the human eye. (I did discover npm launches itself twice on every run! or more precisely the shell script that launches 'node npm-cli.js' does, once to get the 'prefix' config, and then a second time to re-run npm-cli.js located relative to 'prefix'.. yuck.. but I digress).

Is there anyone out there that might have insight as to why, in node's case, code caching in the above way didn't provide any benefit?

Next experiment is to dynamically generate snapshots, a bit more complex to do.

@rvagg I see on some commits you may have insight into snapshots. This issue wasn't intended to be a feature request, but a research discussion around a potential way of improving start times for node based shell commands. Just wondering if you have any thoughts as to viability, potential issues, etc.

You also made a comment on another issue I had in the Modules subsystem. I posted a PR showing what I was talking about. I'm sure you're incredibly busy, but it would be a big favor if you took a peak. I totally get that the node core team will never consider changes to Modules, so I'd much rather have any technical insight you might offer, without the politics if possible? That would be kindof amazing to me if you did that :)

[bnoordhuis]

At a high level I'm wondering if my understanding is correct, that a v8 snap is a heap dump that can be reloaded into an isolate, and that when a new context is created in the isolate it starts with modules already jitted, having come from the snap.

That's correct. A snapshot is essentially the serialized state of the process at the time the snapshot was taken. It's a bit like emacs's unexec or sbcl's save-lisp-and-die command.

cachedData does something different, it's just a precompile step. It takes source code and compiles it into baseline machine code but it doesn't optimize or run it. Unless you are on a slow system (like raspberry pi 1 slow), the overhead of the baseline compiler is negligible for most applications.

The problem with snapshots is that there is no way to revive external state, like file descriptors, child processes, etc. That makes it useless for quickstarting most applications.

@bnoordhuis Thanks for the insight!.

...the overhead of the baseline compiler is negligible for most applications

That would explain why my simple POC using cachedData had no noticeable effect.

The problem with snapshots is that there is no way to revive external state, like file descriptors, child processes, etc. That makes it useless for quickstarting most applications.

Does this mean then that if an application did not have any state outside the v8 heap, that a snapshot would in fact substantially improve startup time?

If so, then for the use case I'm thinking it might be worth the effort to explore this further. Many node commands I'm using seem to follow the same basic pattern:

1. Some amount of .js gets require()d from just starting up
2. One or more config file(s) are opened, read to set flags, etc., then closed
3. Command line args are processed to set/override flags, etc.
4. Files specified on the command line are opened, read, and transformed into some kind of output.

So what if the application, between steps 1 and 2, could explicitly inform node that it was at a point where a snapshot could be taken, and it would not contain any external state of any importance.

1. Would actually taking the snapshot take some inordinate amount of time, like dozes of seconds or minutes? (I get it depends on how much has been require()d, but just trying to get a 'feel' for overhead of actually taking a snapshot)
2. If the application did get to step 2. from a snapshot, would there be a substantial reduction in startup time on the order of at least 4 times or more (i.e. from 400ms to 100ms or less)?

If startup time could be reduced by at least 4 times or more, it would be compelling to me. I've looked around and others have indicated startup times being reduced by up to 20 times in their particular use case, and the way in which they described using snapshots implied similar gains could be achieved. Would you agree, assuming the issues you mentioned where addressed in some way?

Did some more research. Hoping someone having experience with using v8 snaphots could confirm the following.

But also could first validate that, assuming for the moment all the state issues could be addressed, that if a snapshot could be made of some application, like npm, it would offer a substantial increase not just in startup time, but in overall execution time as well, since everything would have already been jitted. And by substantial I'm thinking at least 4 times improvement, hopefully much more. This first kinda 'eyeball-it' of potential gain is a gate I'd like to get through before researching this issue further, so if you know what kinda of gains are typically possible, please let me know :).

Reviewing the node source, it appears dynamically creating a snapshot that included some portion of a user-land application's compiled .js and pure js instances, wouldn't work when reloaded, because even if the application ensured it had no references to any resources outside the v8 heap (OS handles, Buffers, add-ons), node would still have created, and held references to (via bootstrap_node.js), host instances, like process, that in some way had references/pointers to the native implementation (i.e. FunctionTemplates) that obviously live outside the v8 heap.

Now generally, what's important about creating a usable v8 snapshot is that when the snapshot is created, the v8 heap has no references to things outside. However, this does not mean that everything that was executed to create the heap state, couldn't at some point have held references to the outside. For example, the environment could have exposed a FunctionTemplate instance that just provided access to a single native function that could read a file. While creating the heap state, some pure js could have used that function to, say, load a json file which was then deserialized into a pure js object in the heap, then nulled the reference to the FunctionTemplate instance. The heap would then have no references to outside state, and could be safely reloaded and used.

I know intuitively that is conceptually true, but wondering if the special Isolate you get from SnapshotCreator in someway blocks creation of FunctionTemplates, etc., or if it just doesn't care and leaves it up to the consumer to ensure the state of the v8 heap at the point the snapshot is taken. I'll assume for the moment it doesn't care.

It appears with a normal build of node, that a snapshot is made of v8's core library. So, the startup sequence of things is fundamentally:

1. v8 creates an Isolate, and deserializes the v8 core snapshot into it
2. v8 creates some host types/instances (FunctionTemplate)
3. node creates an Environment that creates several host types/instances (process, fs, etc)
4. node runs bootstrap_node.js that creates references to the host types/instances
5. bootstrap_node runs the entry.js specified on command line

Definitely a challenge, but I think there's a way to make this work. It would require that an application explicitly inform node when it could be snapped, which is easy. The hard part would be rebinding all the host types to host instances, like process, after a snapshot was deserialized.

I think I have a way to make that happen, but still researching a bit. But first, it would be such a huge favor if someone with hands-on experience could kinda vet and validate the above, but even more importantly, could make a statement like "If you can make a snapshot, the time savings would be much more the 4x's and it would totally be worth it".