Investigation of post-mortem debugging with Promises.

[uzun]

I am a graduate student at the IBM Centre for Advanced Studies—Atlantic at the University of New Brunswick working with @mhdawson. We are interested in working on post-mortem debugging for promises in Node.js as part of our research and are hoping to get the group’s input as we tackle this problem. I realize from reading the Issues that there isn’t consensus on a solution and although I am not yet a Node expert myself, I am willing to dedicate time to contribute towards a possible fix.

Our current plan would be to:

1. Develop a set of test cases that show the problem of not being able to capture stack traces, core dumps etc. for unhandled exceptions that cause promise rejections.
2. Implement and test a number of different options based on the suggestions in the pull requests so far, as well as new ideas we come up with.
3. Document what can or cannot be captured in each option and the overhead of doing so.
4. Receive input on different options, tradeoffs, and iterate as required.
5. Submit a final pull request to add the most suitable solution from the investigation above.

Some of the ideas we would plan to test would include:

1. Capturing different levels of information (stack trace, core dump etc.) to find a good balance between the overhead and information captured.
2. Using the V8 API to determine the reason for a promise rejection.
3. Adopting a double-run approach. The call stack and relevant metadata for an unhandled rejection are captured on the first pass. Then during the second pass, the rejections are tracked and compared with previous data to determine if a core dump is necessary. We want to analyze when to output or abort and whether it is preferential to do so immediately.
4. Experimenting with when to capture information or abort and the impact this has on existing code using promises.
5. Building up more information over time to help make better decisions on when to capture information.

We plan to keep this issue up to date with pointers to a fork that includes the code for the options we experiment with.

If you have any suggestions on what we should consider looking at or would like to collaborate on this effort just let me know.

[benjamingr]

Hey, have been thinking on this for a while - can we discuss?

[davepacheco]

@misterdjules has done quite a bit of work to identify problematic cases and challenges in addressing them. I don't have the links handy, but perhaps he (or somebody else) does.

[misterdjules]

@misterdjules has done quite a bit of work to identify problematic cases and challenges in addressing them. I don't have the links handy, but perhaps he (or somebody else) does.

I tried to document the challenges of using post-mortem debugging with code that uses promises at https://gist.github.com/misterdjules/2969aa1b5e6440a7e401. I tried to do the same with regards to using post-mortem debugging with code that uses async/await at https://gist.github.com/misterdjules/30db8fc651746c6f917a.

Both documents are works in progress because shortly after I started writing them, the pull request that had been submitted to node core to provide support for promises was closed, and so that work became low priority.

Both documents focused on trying to find a solution so that we could use the exact same post-mortem debugging tools that we use at Joyent (mdb and mdb_v8) to inspect core files generated after an uncaught error is thrown within a promise handler.

The main obstacle to making more progress was, in my opinion, finding out what changes to the behavior of programs using promises would be acceptable to accommodate the needs of post-mortem debugging users. That would have required discussing the design of promises with other people (e.g TC39 members, etc.) and exploring what could be changed about it, but it hasn't been done.

If I understand correctly, this issue introduces an effort that will explore other avenues, so maybe the two gists I linked above and the context I just gave are not that relevant. However, if you have any question, feel free to ask and I'll be happy to try to help!

[mhdawson]

@benjamingr I can facilitate setting up a meeting to discuss for those who are interested. Doodle here: http://doodle.com/poll/mpmrqsxi3tq4msiq. I only suggested a few times, if these don't work out we can find a different set. Anybody who is interested and @uzun please fill out the poll.

[mhdawson]

Starting point for examples discussion with Max today:

Archive.zip

[benjamingr]

So on the 12th, anyone else wants to join the meeting?

[mhdawson]

Just set the time to 12 EST on Monday the 12th of June based on the doodle, I'll post a link to a hangout in this issue just before 12 on Monday. If anybody else is interested pleas join in.

[benjamingr]

@mhdawson ?

[mhdawson]

Link https://hangouts.google.com/hangouts/_/ytl/8DjSM5K6BFZWZG0AEKsTmlc7le8YVbb6ZbMW-E8AF-4=?eid=100598160817214911030

[mhdawson]

@benjamingr @uzun

[mhdawson]

Meeting recording here: https://www.youtube.com/watch?v=WeQAIUPbHOA although we started it a bit late so did not capture all of the discussion.

We also agreed to have a regular meeting 12 EST on Mondays to get started. @benjamingr is going to list a few more test cases we should cover and he also provided a link to the v8 API's that let you register for rejections (max will add link to this issue)

[benjamingr]

In my opinion, the most interesting case is:

function test() {
  var p = Promise.reject(new Error());
  process.nextTick(() => modifyHeap()); 
  // if you remove me, a core dump should return an unmodified heap
  setTimeout(() => { p.catch(function(() => {}); 
}

The promise is handled in a macrotick - but the heap is modified in a microtick. This is the "core" case @misterdjules and I have been discussing. Unless we can get a core dump "fast enough" to take in this case (which we can't if I understand correctly, since it's not viable to only capture the page changed in modifyHeap() and the rest later if it's unhandled).

[rnchamberlain]

It might be possible to have an option/hook to fork the process at the reject point (i.e. before modifyHeap()). The parent and child then run as copy-on-write. Then if/when the decision is made that a core dump is needed, we request the child (which has been idle) to dump core.

Alternatively, we capture a lightweight node-report or similar at the unhandled point - first failure. Then using the information it contains (eg stack trace?) we set a option/hook to trigger a core dump at the reject point on a subsequent run.

[benjamingr]

It might be possible to have an option/hook to fork the process at the reject point (i.e. before modifyHeap()). The parent and child then run as copy-on-write. Then if/when the decision is made that a core dump is needed, we request the child (which has been idle) to dump core.

I was under the impression that this solution is impossible because taking a copy-on-write core dump is too expensive. What if we fork and then just block - and then take the dump only when we're sure?

Is forking 2 times a second without doing any memory touching too expensive?