Node CPU Profiling Roadmap

[yunong]

We’d like to get feedback on the status of the profiling tools landscape in Node.js today. In particular -- we want to get alignment on a roadmap which will provide a free, open source, and cross-platform set of tools that are part of the node/v8 API i.e. maintained across LTS versions that can provide a powerful suite to debug and diagnose Node.js issues in production.

Production Challenges

There are some challenges that are unique to debugging and diagnosing issues in production. Specifically for large critical production deployments. In particular here are some of the constraints due to the production nature of the deployments:

1. Generally, most solutions need to be low impact on the process. Attaching debuggers or tracers is often impractical when the application is taking production traffic. Debuggers will pause the process, causing all inflight requests to hang. Tracers and debuggers often introduce large performance penalties, which can impact customers.
2. Tools need to be safe and not cause kernel panics or process crashes.
3. It’s often hard to reproduce issues seen at production scale -- therefore programmatic access of tooling via APIs can help diagnose problems as they occur. e.g. start a profile based on some condition since it’s often impossible to reproduce

Profiling

One of the most useful methodologies to optimize CPU performance in a running application is by sampling the CPU stack frames (CPU profiling) and then visualizing the samples, typically using a flamegraph. This technique will show hot code paths on CPU -- which gives you the opportunity to optimize the relevant source code.

The approach can be done in production with OS level profilers such as (perf, DTrace, systemtap, eBPF) with very low overhead. The profilers lack the information to resolve the JS frames, resulting in unhelpful memory addresses for the JS frames. V8 solves this problem by dumping a mapping of native frame addresses to JS source and line number.

It’s important to mention here that having access to all stack frames, whether native (v8, libc, syscalls, libuv, native modules) or JS is important. Problems can occur anywhere in the stack, and we want to be able to profile Node with complete stack frames. E.g. We heavily use gRPC -- which is a native module -- so without access to native frames we would not be able to get visibility into this critical part of our application.

There are a few issues with this implementation:

1. perf(1) support is now deprecated in V8 and will not be supported starting in Node 8 -- which effectively means we’re losing the ability to profile JS stacks.
2. Lack of cross-platform tooling. Even if perf support wasn’t deprecated, this solution only works on Linux.

We’d like to contribute and collaborate on a set of comprehensive, cross-platform, and open source CPU profiling tools with the Node and V8 team. The V8 team has advised us that they plan to support the v8 profiler and the v8 cpu profiler API going forward, and we want to unlock CPU profiling capabilities in Node using these supported frameworks.

Roadmap:

1. Determine which tools -- v8 profiler (--prof), cpu profiler or something else -- to support and maintain for profiling Node applications.
2. Currently, these tools do not support native c/c++ frames which includes v8, native modules, libuv or syscall stack frames. Note because these tools are in process, they will never be able to show syscalls as those can only be sampled from within the kernel. Supporting native frames will require implementing a stackwalker in the v8 profiler or the cpu profiler.
3. Determine the overhead of profiling using these tools. With the now deprecated OS level profilers, the overhead was quite low -- and thus suitable to use in a production environment without outsized impact to customers.
4. Dynamically toggle profiling in a production environment. It’s not practical to profile all the time in production due to the performance impact -- being able to dynamically enable and disable profiling via an API would be ideal.
5. Provide an API to programmatically consume profiling data once enabled.
6. Add documentation so these tools can be adopted by users.

We’re looking for feedback and alignment with the community on this subject before proceeding with the design and implementation -- please let us know your thoughts.

[mmarchini]

Great explanation and summary of the current scenario.

Suggestion: We could add "Write tests to those profiling tools" to the roadmap, as I think one of the goals is to have these tools as first-class citizens in the future.

[joyeecheung]

Another suggestion: a better standardized format for the profiles, and tools that convert legacy formats to it. The current v8 CPU profile format is quite limiting considering it's JSON so it cannot be concatenated easily and has to be in the memory as a whole before being serialized.

[davidmarkclements]

In 0x (flamegraph tool) I've re-implemented stack tracing on top of v8 profiler (--prof) and the v8 tick processor (--prof-process --preprocess) – see davidmarkclements/0x#99 - it's currently behind a flag but next major version this will be default with a --kernel-tracing flag for OS level tracing.

Pros

• no need for sudo
• totally cross platform - runs on anything that Node does
• zero config to make it work in vms and containers
• single thing to maintain (if I wasn't keeping kernel tracing support)
• works around the problem with kernel tracing and turbofan - namely that interpreted (functions which do not get optimized) are opaque behind BytecodeHandlers (the trace output will have series of Bytecode related stack frames instead of the actual function the bytecode is handling). V8 prof does the memory address lookup to resolve these stack frames. This is considered a V8 bug https://bugs.chromium.org/p/v8/issues/detail?id=7155 so hopefully that will be fixed in future... but we still have the whole Node 8 line to date without it.

Cons

• no I/O stacks and any third party C – if there's a hot path in libuv or some C/Cpp module it's not going to show
• higher overhead
• more difficult if you want realtime processing - while the isolate logfile is written to incrementally, you can't fully control the name of the file (isolate-ADDR-v8.log where ADDR is determined internally) and it's incomplete JSON until the process finishes.
• the output format isn't great - best way is to use v8 tick processor but thats an all at once operation, again blocking to incremental processing
• platform/configuration/hardware dependent timestamps - the prof output uses TimeTicks::HighResolutionNow which is an internal v8 function which has different output depending on system. In a particular configuration of OS X for instance, the TimeTicks::HighResolutionNow output is based on amount of ticks (it just adds one for each tick) – this makes it incredibly difficult to reliably cross reference stacks capture by --prof to stacks captured by anything else. Not only that but how do you match the sample rate so that you get one-to-one on the samples?

What I would love is the ability to cross reference prof stacks with OS level tracing stacks. If we can do that, then perf being deprecated, and bytecode handler problem (and any other problems) would be resolved with --prof while lower level tracing could be used to compliment --prof.

[davidmarkclements]

In terms of Node internal implementation, absolutely agree on tests.

The way we currently expose --prof-process isn't ideal, the stacks processor is concatenated JavaScript based on some JS files in deps/v8/tools - .. with additional injected code to make log output work with Node (see nodejs/node#14966)

[danielkhan]

We recently added production grade continuous sampling to our product. For that we had to find a sweet spot between sampling frequency and duration. Additionally there was a memory leak in v8::CpuProfiler (nodejs/node#14894) we discovered.
Here is a blog post with screen shots https://www.dynatrace.com/news/blog/introducing-code-level-visibility-for-node-js/

As already mentioned, this approach won't give you native frames - still we considered it to be sufficient for many use cases.

@Hollerberg who implemented it will be at the summit and we are happy to discuss the validity of our approach and be part of an initiative to improve CPU sampling capabilities in node.

[aalexand]

We use the CPU profiler in the profiling agent here that is an experimental way to continuously collect the data from production Node.js apps. Some notes on that experience so far:

• The testing and better overhead guarantees from the runtime would be very much desired. "How much will it cost my prod?" is a common question and it's hard to answer that without the implementation backing that claim and testing protecting it.
• The heap profiling is not less important than CPU profiling, so it would be good to discuss that as well, maybe separately. The heap state is interesting in production for both investigating occasionally OOMs and understanding the causes of any excessive pressure on the heap (which in GC languages turns into wasted CPU cycles).
• As a format, we use the proto-based profile.proto format a lot and in the open-source pprof tool, so that would be one possible solution to consider as the unified format.
• For the native stacks, this would be good, yes. What is also desired is that the profiler is capable of profiling the CPU time of all threads in the process. The profiler we use now only samples the Node thread and when that thread is doing nothing, it's hard to say whether it does nothing because there is no work to do or because it's waiting for something in libuv to finish. Having at least some visibility into what happens in libuv would be highly desired and profiling all threads in the process is one way to go.
• Regarding perf support - OS profilers are nice but there are types of virtualized environments where perf hardware events are not available, so the benefits of OS-level profiling are smaller there. Also things like heap profiling are only available within the process as tight cooperation with the runtime is necessary.
• Regarding being able to turn the profiling on and off, this is actually possible today via the StartProfiling / StopProfiling interface.

[mhdawson]

I believe it is important that Node.js have first-class tooling for problem investigation and CPU profiling is part of that along with tracing, debugging, heap dumps, node-report and being able to introspect core files. Ideally, a base level of core tooling would be available with the runtime without having to install additional components as that can often be an issue in production.

That's a long way of saying that I support your effort and will look to see how I can help in respect to getting ci testing and other supporting pieces in place and also how I might be able to pull in people from IBM to help collaborate.

[mmarchini]

For reference: #150

[hashseed]

Another issue that @bmeurer mentioned to me is that there is no way for Linux perf and any other perf tool outside of V8 to discern inlined functions from its caller, since both occupy a single stack frame. The CpuProfiler doesn't provide this either, but @FranziH is addressing this.