Expand and organize repo

README.md

@@ -1,9 +1,24 @@
-# tracing-wg
+# Diagnostics Working Group  
+The goal of this WG is to ensure Node provides a set of comprehensive, documented, extensible diagnostic protocols, formats, and APIs to enable tool vendors to provide reliable diagnostic tools for Node.
 
-Tracing Working Group
+Work is divided into several domains:
+- [Tracing](./tracing)
+- [Profiling](./profiling)
+- [Heap and Memory Analysis](./heap-memory)
+- [Step Debugging](./debugging)
 
-# Members
+Background, reference documentation, samples, and discussion for each domain is contained within its folder.
 
+Work needed includes:  
+- Collect, understand, and document existing diagnostic capabilities and entry-points throughout Node, V8, and other components.
+- Collect and document projects and products providing diagnostics for Node with brief description of their technical architecture and sponsoring organizations.
+- Identify opportunities and gaps, then propose and implement solutions.
+
+### Logistics
+- Monthly Meetings
+- Biannual F2F
+
+### Members  
 * @bnoordhuis 
 * @sam-github 
 * @othiym23 
@@ -28,3 +43,5 @@ Tracing Working Group
 * @matthewloring
 * @pmuellr
 * @jeffolfert
+* @joshgav
+

debugging/README.md

@@ -0,0 +1,29 @@
+## Debugging API
+- [Chrome Debugging Protocol](https://chromedevtools.github.io/debugger-protocol-viewer/)
+- [V8 Debugging Protocol](https://github.com/v8/v8/wiki/Debugging-Protocol)
+
+Node currently relies on V8's internal Debug API and associated commands and events. The Debug API is published at [v8/include/v8-debug.h](https://github.com/v8/v8/blob/master/include/v8-debug.h), and the message protocol is documented in prose in [the V8 wiki](https://github.com/v8/v8/wiki/Debugging-Protocol) and in code in [v8/src/debug/debug.js#L2341-2372](https://github.com/v8/v8/blob/master/src/debug/debug.js#L2341-L2372). 
+
+Node also provides an agent [node/src/debug-agent.h](https://github.com/blob/master/src/debug-agent.h) which relays requests, responses, and events through a TCP socket, and a command-line debugger [node/lib/_debugger.js](https://github.com/blob/master/lib/_debugger.js).
+
+The Chrome/V8 team has deprecated the internal V8 API and command set and proposed replacing it with a subset of the Chrome Debugging Protocol (CrDP), see https://github.com/nodejs/node/issues/2546. CrDP is documented at [https://chromedevtools.github.io/debugger-protocol-viewer/]() and the backing GitHub repo.
+
+The [cyrus-and/chrome-remote-interface](https://github.com/cyrus-and/chrome-remote-interface) library provides a JS proxy for CrDP.
+
+## Step Debugging Tools  
+Name | Sponsor
+-----|--------
+[node-inspector][] | StrongLoop
+[JetBrains WebStorm][] | JetBrains
+[Visual Studio Code][] | Microsoft
+[Node CLI Debugger][] | Node Foundation
+[Chrome DevTools][] | Google
+[Theseus][] | Adobe Research
+
+[node-inspector]: https://github.com/node-inspector/node-inspector 
+[JetBrains WebStorm]: https://www.jetbrains.com/help/webstorm/2016.1/running-and-debugging-node-js.html
+[Visual Studio Code]: https://github.com/Microsoft/vscode
+[Node CLI Debugger]: https://nodejs.org/api/debugger.html
+[Chrome DevTools]: https://github.com/ChromeDevTools/devtools-frontend
+[Theseus]: https://github.com/adobe-research/theseus
+

heap-memory/README.md

@@ -0,0 +1,10 @@
+## Heap and Memory Analysis  
+The [nodejs/post-mortem](https://github.com/nodejs/post-mortem) WG is responsible for this domain.
+
+### Heap Analysis  
+- [mdb_v8](https://github.com/joyent/mdb_v8)
+- [node-heapdump](https://github.com/bnoordhuis/node-heapdump)
+- [llnode](https://github.com/indutny/llnode)
+- [IBM IDDE](https://www.ibm.com/developerworks/java/jdk/tools/idde/)
+- [Snapshot Utils](https://github.com/bmeck/snapshot-utils)
+

profiling/README.md

@@ -0,0 +1,26 @@
+## Profiling for Node  
+- CPU Profiling
+  - V8 CPU Profiler: [v8/include/v8-profiler.h](https://github.com/v8/blob/master/include/v8-profiler.h)
+    - Accessible globally through `--prof` runtime flag, basic processing and display through `--prof-processor` runtime flag
+  - Intel vTune profiler. Accessible through `--enable-vtune-profiling` compile-time flag.
+
+- Heap Profiling
+  - V8 Heap Profiler: [v8/include/v8-profiler.h](https://github.com/v8/blob/master/include/v8-profiler.h)
+    - Heap memory usage stats are available through built-in 'v8' module.
+
+Name | Sponsor
+-----|--------
+[v8-profiler][] | StrongLoop
+[Chrome DevTools][] | Google
+
+[v8-profiler]: https://github.com/node-inspector/v8-profiler
+[Chrome DevTools]: https://github.com/ChromeDevTools/devtools-frontend
+
+### Stack Analysis  
+- [0x](https://github.com/davidmarkclements/0x)
+- [StackVis](https://github.com/joyent/node-stackvis)
+
+## Docs  
+[guides/simple-profiling](https://nodejs.org/en/docs/guides/simple-profiling/)
+[paulirish/automated-chrome-profiling](https://github.com/paulirish/automated-chrome-profiling)
+

tracing/README.md

@@ -0,0 +1,7 @@
+## Tracing  
+- [AsyncWrap](./AsyncWrap/README.md) - Most native objects are represented indirectly by the [AsyncWrap class](https://github.com/nodejs/node/blob/master/src/async-wrap.h), so hooks have been added to that class to trace lifecycle (init/destroy) and callback activity (pre/post) related to these objects.
+
+- [OS Tracing](./os-tracing/README.md) - LTTng (Linux), SystemTap (Linux), DTrace (OSX), ETW (Windows)
+
+- [VM Tracing](./vm-tracing/README.md) - Tracing native to JS VMs, such as V8's TRACE_EVENT.
+

tracing/os-tracing/README.md

@@ -0,0 +1,26 @@
+## OS Tracing  
+
+* [DTrace](http://dtrace.org/)  
+  Available on IllumOS, Solaris, Mac
+
+  - The [DTrace blog](http://dtrace.org/blogs/blog/category/node-js/) has some articles on using DTrace with Node.js
+
+* [LTTng](http://lttng.org/)  
+  Linux only.  
+
+  - [Tracing Node on Linux with LTTNG](http://nearform.github.io/tracing-node-lttng-nodejsdublin/)
+  - [thekemkid/magic-tracing](https://github.com/thekemkid/magic-tracing) is an example of a Node.js application which implements LTTng tracepoints. A demo is available [here](https://asciicinema.org/a/16785).
+
+* [ETW](https://msdn.microsoft.com/en-us/library/windows/desktop/aa363668.aspx)  
+  Windows only.  
+
+  - Some information was provided by Bill Ticehurst (@billti) in [issue #10](https://github.com/nodejs/tracing-wg/issues/10#issuecomment-137145822) about how he integrated ETW into Node.js for Windows
+
+* [Perf](https://perf.wiki.kernel.org/)  
+  Linux only.  
+
+  - Can be enabled with `--perf_basic_prof` v8 run-time flag.
+
+* [SystemTap](https://sourceware.org/systemtap/)  
+  Linux only.
+

docs/v8-tracing/README.md → tracing/vm-tracing/README.md

@@ -1,13 +1,24 @@
-Node.js tracing - native v8 tracing
-================================================================================
+## VM Native Tracing  
+- See [v8-tracing.md](./v8-tracing.md) for technical details and crude implementation.
 
-Google is restructuring the tracing in V8 to accommodate restructuring of
-tracing in Chrome.
+- All trace events are streamed through the same native component, which can in turn publish as appropriate to its subscribers.
+    - Ideally should be possible to subscribe and receive events in JavaScript too.
+- Requires implementation in Node.js, could learn from Chromium's implementation.
+- For maximum utility, we should add trace events to Node's operations, especially async ones.
+- Possible to handle other trace sources, such as OS (LTTng, ETW) alongside V8 trace source, since they all are event streams.
 
-----
+## Resources  
+- [google/trace-viewer](https://github.com/google/trace-viewer) - `chrome://tracing` UI.
+- [chromium/src/base/trace_event/](https://code.google.com/p/chromium/codesearch#chromium/src/base/trace_event/)
+- [Chromium Speed-Infra](https://www.chromium.org/developers/speed-infra)
+- [Chromium Telemetry](https://catapult.gsrc.io/telemetry)
+- [Trace Platform Explainer](https://docs.google.com/a/chromium.org/document/d/1l0N1B4L4D94andL1BY39Rs_yXU8ktJBrKbt9EvOb-Oc/edit)
+- [Implementations for V8, D8, Chromium](https://docs.google.com/a/chromium.org/document/d/1_4LAnInOB8tM_DLjptWiszRwa4qwiSsDzMkO4tU-Qes/edit#)
+- [Trace Event Format](https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit)
+
+### Background  
 
-Some older-ish comments from @natduca in
-[nodejs/node issue #671](https://github.com/nodejs/node/issues/671#issuecomment-73191538):
+@natduca in [nodejs/node#671_comment](https://github.com/nodejs/node/issues/671#issuecomment-73191538):
 
 I thought I'd provide some notes on the direction we're taking in chrome and indirectly v8 around tracing & stack sampling. Since we both intersect at v8 and do run into some of the same "guhh i need a tracing api but its gotta be low overhead," I hope there's some useful context!
 
@@ -37,8 +48,7 @@ Anyway, thats a lotta words. Hope it makes sense! This was all designed with cli
 
 ----
 
-@ofrobots added some additional detail in
-[issue #21](https://github.com/nodejs/tracing-wg/issues/21#issuecomment-143891637):
+@ofrobots in [nodejs/tracing-wg#21_comment](https://github.com/nodejs/tracing-wg/issues/21#issuecomment-143891637):
 
 Here are the main components, as per my understanding:
 
@@ -54,6 +64,3 @@ Basically this trace-event buffer becomes the 'single pipe' through all trace-ev
 
 Once the above linked CL lands, traces being gathered in V8 will show up there. Node.js can start putting its own traces into the same pipe. I would imagine we would also need an API in Node.js to be exposed to JavaScript to allow JS code to start sending the trace events.
 
-----
-
-If you have more info to provide, please send us a pull request!

tracing/vm-tracing/v8-tracing.md

@@ -0,0 +1,39 @@
+## V8/Chromium Tracing  
+Generic macro definitions intended to be shared by many projects are in `trace_event_common.h`.  
+Source of truth for `trace_event_common.h` is in `chromium:src/base/trace_event/trace_event_common.h`.  
+This file is in Chromium but not in V8. @ofrobots copied it into `nodejs:deps/v8`. Perhaps it should be moved to `nodejs:deps/v8/include` or `nodejs:deps/v8/src/trace_event`.  
+
+Implementation-specific ("INTERNAL_*") macro definitions are in `v8:src/trace_event.h` and `chromium:src/base/trace_event.h`.  
+Chromium: `trace_event.h` macros utilize `v8::base::TraceLog::AddTraceEvent` to write traces.  
+V8: `trace_event.h` macros utilize `v8::Platform::AddTraceEvent` to write traces.  
+Node uses `DefaultPlatform` from `v8:src/libplatform/default-platform.h` which currently simply returns 0.  
+A simplistic implementation which logs all trace events to stdout illustrates the basics:
+
+```
+// from v8:src/libplatform/default-platform.cc
+// add `#include <iostream>` to includes
+
+uint64_t DefaultPlatform::AddTraceEvent(
+    char phase, const uint8_t* category_enabled_flag, const char* name,
+    /* const char* scope, */ uint64_t id, uint64_t bind_id, int num_args,
+    const char** arg_names, const uint8_t* arg_types,
+    const uint64_t* arg_values, unsigned int flags) {
+
+  time_t timestamp = std::time(nullptr);
+
+  std::cout << "phase: " << phase << std::endl;
+  std::cout << "category_enabled_flag: " << category_enabled_flag << std::endl;
+  std::cout << "name: " << name << std::endl;
+  std::cout << "id: " << id << std::endl;
+  std::cout << "epoch_timestamp: " << timestamp << std::endl;
+  std::cout << "human_timestamp: " << std::asctime(std::localtime(&timestamp)) << std::endl;
+
+  return 0;
+}
+
+const uint8_t* DefaultPlatform::GetCategoryGroupEnabled(const char* name) {
+  static uint8_t yes = CategoryGroupEnabledFlags::kEnabledForRecording_CategoryGroupEnabledFlags | 
+    CategoryGroupEnabledFlags::kEnabledForEventCallback_CategoryGroupEnabledFlags;
+  return &yes;
+}
+```