NODE_V8_COVERAGE can run OOM when compiling a lot of scripts in one process

[joyeecheung]

For more context, see https://bugs.chromium.org/p/v8/issues/detail?id=13183. In summary, in Node.js NODE_V8_COVERAGE is implemented by taking precise coverage, which holds on to all the feedback vectors of every function (including scripts) that has ever been compiled, even after the functions are gone. My understanding that this is by-design for precise coverage - by the time the coverage is collected, it's possible that the functions have already been GC'ed, but V8 still needs to consult the feedback vectors for invocation counts that were recorded when the functions were still alive. As an alternative V8 provides best-effort coverage that does not hold on to feedback vectors but as a result the data from functions that have been GC'ed by the time the coverage is taken can be missing.

Reproduction (h/t @ptomato):

//  Run this with NODE_V8_COVERAGE=/tmp node --max_old_space_size=30 test.mjs
import vm from 'node:vm';
import v8 from 'node:v8';
const setupCode = new vm.Script('Date.now();'.repeat(10000));

for (let count = 1; count < 50000; count++) {
  const context = {};
  vm.createContext(context);
  setupCode.runInContext(context);
  if (count % 100 === 0) {
    console.log(count, v8.getHeapStatistics());
    console.log(process.memoryUsage());
  }
}

I think there are two options to tackle this:

1. When NODE_V8_COVERAGE is enabled, periodically collect coverage, which allows V8 to GC the feedback vectors of functions that are no longer alive. We either keep that data in memory and write it out when the process is about to exit (which is what we do now), or flush the collected coverage periodically to the directory specified by NODE_V8_COVERAGE (personally I like the latter better, I believe the existing coverage tooling can already merge the data from different files anyway?) We could add another environment variable for users to control the frequency, and they can use Infinity to tell Node.js to go back to the current behavior.
2. We can also provide an environment variable for the user to specify that they want best-effort coverage instead of precise coverage.

cc @bcoe

[joyeecheung]

Another idea: instead of periodically collecting the coverage, we collect the coverage when the heap size limit is about to be reached, it's probably easier to implement(?), and would result in fewer coverage files.

[bcoe]

@joyeecheung I seem to recall that "best effort" coverage may break some of the report tooling, leading to confusing results . If we could figure out how to dump coverage periodically that sounds ideal.

[joyeecheung]

I looked into this a bit, and looks like simply collecting the coverage with the current configuration wouldn't result in cleanups of the feedback vectors, because V8 only cleans them up when collecting binary coverage (i.e. when callCount of Profiler.startPreciseCoverage is set to false so that V8 only records if a function is invoked, but not the precise invocation counts). So I have a few questions @bcoe:

• How useful is it in practice to have the actual invocation count? One way to go about it would be to add a configuration option to tell NODE_V8_COVERAGE to collect binary coverage instead, so that the feedback can be freed up after coverage collection.
• Or, we might be able to just add an option to V8 to clean the feedback up even when it's collecting invocation counts if the counts can be merged by the consumer in the end (from what I can tell that's what c8 and probably other v8 coverage tools do).

[bcoe]

How useful is it in practice to have the actual invocation count? One way to go about it would be to add a configuration option to tell NODE_V8_COVERAGE to collect binary coverage instead, so that the feedback can be freed up after coverage collection.

There is value in invocation counts, as it can help in determining hot spots in your application that are worth optimizing.

However, I think for many use cases having a binary value of whether or not a line was hit would be sufficient.

Or, we might be able to just add an option to V8 to clean the feedback up even when it's collecting invocation counts if the counts can be merged by the consumer in the end (from what I can tell that's what c8 and probably other v8 coverage tools do).

This sounds quite reasonable, this is exactly what c8 does.