Promise Performance Meeting

[benjamingr]

I would like to set up a hangout with the V8 team regarding promise performance. I've already talked to @bmeurer the person in charge on V8's side regarding a meeting and he has been gracious enough to agree to set up one.

by all means, if discussion here gets noisy feel free to unsubscribe - we will discuss this in the hangout

Outstanding issues I'd like to discuss

• Any hooks we need in the microtask queue (for example for the VM issue) and how they might impact performance.
• The overhead of creation and how we can make util.promisify faster to a point where native promises win against Bluebird 3.5 in the doxbee benchmark.
• Common use cases we have gathered from the thousand+ questions in the promise tag in StackOverflow and their performance profile.
• Performance cliffs and gotchas that relate to promise code "in the wild" and how we can deliver a better experience to users.

Outstanding issues that are interesting, but not necessarily for the meeting

• The promises/post-mortem issue that's been blocking integration and whether or not we might be able to get help from the V8 side.
• ETAs and stability notes on new promise features (async iterators, promise#finally etc) so we can include those in the relevant APIs (we're mostly prepared though).
• Async iterator performance profiles (basically, async iterators are a little useless in performance based scenarios if the iteration is not inlined and optimized somehow because of the overhead of a promise + iterator object per call).

Participation

In addition to nodejs/promises members - pinging @petkaantonov @nodejs/performance, @addaleax, @caitp and @misterdjules.

I'd like to brainstorm here for a few days and then set up the hangout. I'd like to limit participation to a few people - I'd love to attend to share the cliffs and bluebird experience, I'd love @addaleax and @mcollina to attend from Node's side and @bmeurer obviously :) If anyone would like to attend because they don't feel their view point will be represented please comment here with why you would like to attend.

Optimally most discussion before the meeting should happen here.

[mcollina]

I do not think we should care for the bluebird benchmark at all, mostly because of this: petkaantonov/bluebird#887 (comment).
We probably need new benchmarks that are independent from the ones in bluebird - IMHO they are built to make bluebird shine.

More importantly, I think the best approach is to focus on async/await benchmarking and performance. This is what people will use, and one of the main drivers for adopting native promises in modules and applications.

I'm +1 in joining, typically CEST business hours works well for me, but I can also join later in the evening.

[benjamingr]

We probably need new benchmarks that are independent from the ones in bluebird - IMHO they are built to make bluebird shine.

Actually, the benchmarks predate bluebird (so it's more accurate to say that Bluebird is built to make the benchmarks shine). Bluebird is not at the best maintenance state after 4 years but it is the benchmark that V8 is using internally for promise performance. I didn't suggest them because they are terrific. We can also extract the benchmark from https://github.com/petkaantonov/bluebird/tree/master/benchmark/doxbee-sequential or approach @spion directly about the suite - I assume V8 did the former.

I'm also +1 on writing better applicative promise benchmarks that simulate real world loads. I think we'll discover quickly that our bottleneck is still util.promisify and the fact that V8 schedules then on the microtask queue where bluebird doesn't if it can avoid it (by tagging the code).

I think V8 can definitely win vs. bluebird with TurboFan inlining promise calls and async/await based optimizations (I'm sure TurboFan, unlike Crankshaft is fully capable of that level of optimization after reading the optimization code). Moreover, the V8 team are willing to listen to Node.js performance concerns now (which is awesome). I think async/await can win against callbacks (as a long term goal) but that's a farther point.

[mcollina]

Actually, the benchmarks predate bluebird (so it's more accurate to say that Bluebird is built to make the benchmarks shine). Bluebird is not at the best maintenance state after 4 years but it is the benchmark that V8 is using internally for promise performance. I didn't suggest them because they are terrific. We can also extract the benchmark from https://github.com/petkaantonov/bluebird/tree/master/benchmark/doxbee-sequential or approach @spion directly about the suite - I assume V8 did the former.

I'm ok if we move them/extract them to another repo, the fact that they currently live within bluebird makes them not fair.

I'm also +1 on writing better applicative promise benchmarks that simulate real world loads.

Definitely 👍 .

I think we'll discover quickly that our bottleneck is still util.promisify

util.promisify is only relevant to convert callback APIs. However most of the CPU time in applications is spent in business logic or modules/frameworks. I would expect some of them to be natively written with async/await. How does that compare? What's the different between writing native async/await  code and using util.promisify?

and the fact that V8 schedules then on the microtask queue where bluebird doesn't if it can avoid it (by tagging the code).

But that's it not spec-compliant, right?

I think V8 can definitely win vs. bluebird with TurboFan inlining promise calls and async/await based optimizations (I'm sure TurboFan, unlike Crankshaft is fully capable of that level of optimization after reading the optimization code).

I can confirm this. It's very impressive and it took me some time to understand why some async/await code was faster than the equivalent callback-based.

[benjamingr]

I would expect some of them to be natively written with async/await. How does that compare? What's the different between writing native async/await code and using util.promisify?

Well that's a good point. Today our entire Node.js platform is using callbacks, for people using a promisified Node.js core (with util.promisify) the bottleneck of util.promisify is real. At Peer5 for example we have hundreds of thousands of promises in flight at once very commonly and the overhead of calling every single I/O method being a little bit slower can be significant.

Once we deliver a promisified core API a faster util.promisify won't be a big priority. I think that there is still a lot low hanging fruit that can be picked to make native promises faster than bluebird without discussing promisified core.

But that's it not spec-compliant, right?

Sure it is, I'll give an example:

new Promise(r => setTimeout(r, 1)).then(x => console.log("Got here"))

In this case, bluebird sees that new Promise did not resolve synchronously and thus does not defer the console.log to the next tick (it was already deferred). It also does this optimization when promisifying.

By comparison Node or V8 makes no such assumptions and always defers .then callbacks to the microtick queue even if they can run synchronously (since we know they run after "only platform code" remained on the stack as required).

Scheduling everything on nextTick can have measurable and perhaps even considerable overhead. Although I think promisification is what dominates the performance charactaristics of benchmarks that make hundreds of thousands of calls.

I'd like promises to be a 0 overhead abstraction in Node, Bluebird is close - but I think native promises can win here and I'm rooting for them. With the help of Benedikt I think that's a very reasonable goal.

Other than creation of promises and a few optimization tricks - I think V8 promises can be very competitive. I'm looking for additional pain points to optimize for before the meeting - any input (from you in particular) would be very useful here.

[targos]

Has anyone tried to benchmark the current implementation of util.promisify vs. one written with the functions provided by V8 extras?

[mcollina]

Once we deliver a promisified core API a faster util.promisify won't be a big priority. I think that there is still a lot low hanging fruit that can be picked to make native promises faster than bluebird without discussing promisified core.

I agree.

By comparison Node or V8 makes no such assumptions and always defers .then callbacks to the microtick queue even if they can run synchronously (since we know they run after "only platform code" remained on the stack as required).

Can V8 implement the same behavior?

[benjamingr]

@targos I'm unfamiliar with that approach - if @addaleax chimes in she might have done it. I benchmarked util.promisify vs a userland promisify and it was faster measurably on my computer. The benchmarks are here.

With util.promisify:

 promises-ecmascript6-native.js                  664       60.14

And without util.promisify:

 promises-ecmascript6-native.js                  879       63.53

One nice side effect is that promises beat caolan/async with Node at the moment. Those benchmarks are on older Node. I'll run them again and compare (If you create a fork with a promisify based on another approach I can check that out).

Will run the tests with Node 8.4 and post the results here.

---

@mcollina

Can V8 implement the same behavior?

I don't see why not, they control the promise constructor and the then and can provide a hook for promisify. I'd like to gather a bunch of possible optimizations we do at Bluebird for the call and present them to the V8 team.

By the way, how do you feel about forking Bluebird under the Node org (possibly in nodejs/promises), removing the excess code and making a benchmark here where we can maintain it? I hope you realize Petka didn't close that PR because he was doing anything fishy - he is just a very busy individual without a lot of time to maintain the none-critical parts. We're all Node collaborators (me you and Petka) with the same goal (faster core promises).

[benjamingr]

I created two versions of native promises (one with util.promisify and one without) and removed everything but bluebird and native promises:

file                                      time(ms)  memory(MB)
callbacks-baseline.js                          162       32.80
promises-bluebird-generator.js                 257       40.44
promises-bluebird.js                           316       45.18
promises-ecmascript6-native-promisify.js       538       63.64
promises-ecmascript6-native-base.js            659      109.70
promises-ecmascript6-asyncawait.js             772      135.84

Platform info:
Darwin 16.7.0 x64
Node.JS 8.4.0
V8 6.0.286.52
Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz × 8

I'm trying to figure out what's the best place to upload said benchmark. It simulates doing several IO operations and committing to a database. Going to check out node-v8 on V8 6.2.248 next and see what numbers I get on it.

Same hardware - running the latest canary:

file                                      time(ms)  memory(MB)
callbacks-baseline.js                          152       28.04
promises-bluebird-generator.js                 235       40.29
promises-bluebird.js                           283       48.36
promises-ecmascript6-native-base.js            432       98.27
promises-ecmascript6-native-promisify.js       482       65.26
promises-ecmascript6-asyncawait.js             500      119.54

Platform info:
Darwin 16.7.0 x64
Node.JS 9.0.0-pre
V8 6.2.248
Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz × 8

Looks like everything got faster, but native promisify is actually faster than util.promisify, which is a very surprising result. It looks like the memory consumption of native promises in particular got a lot better.

As a comparison, here is 8.1 on the same hardware (still running TF on the async/await one):

file                                      time(ms)  memory(MB)
callbacks-baseline.js                          138       30.59
promises-bluebird-generator.js                 269       41.82
promises-bluebird.js                           306       47.99
promises-ecmascript6-native-promisify.js       577       68.37
promises-ecmascript6-native-base.js            642       98.47
promises-ecmascript6-asyncawait.js             755      117.48

Platform info:
Darwin 16.7.0 x64
Node.JS 8.1.3
V8 5.8.283.41
Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz × 8

[kyrylkov]

Numbers on Windows:

file                               time(ms)  memory(MB)
callbacks-baseline.js                   149       33.54
promises-bluebird-generator.js          306       41.39
promises-bluebird.js                    378       45.06
async-bluebird.js                       523       73.09
promises-es2015-util.promisify.js       598       68.02
async-es2017-util.promisify.js          604       77.70
promises-es2015-native.js               651      110.50
async-es2017-native.js                  653      109.88

Platform info:
Windows_NT 10.0.15063 x64
Node.JS 8.4.0
V8 6.0.286.52
Intel(R) Core(TM) i7-3667U CPU @ 2.00GHz × 4

Source: https://github.com/kyrylkov/promise-async-performance

[benjamingr]

@kyrylkov thanks! Given I did not share the code the fact we got similar results is reassuring. For comparison and until we decide where to put it - here is the patch I applied https://gist.github.com/benjamingr/a7c7cd7ff2ee69d3a71f0058d286ff9a#file-1-diff-L617-L675

[kyrylkov]

@benjamingr Here is the code https://github.com/kyrylkov/promise-async-performance

[targos]

@benjamingr you can try with this branch: https://github.com/targos/node/commits/expose-v8-extras

I checked with a very simplistic benchmark and the V8 extras seem to bring a non-negligible improvement:

'use strict';

function asyncF(cb) {
  cb();
}

const {promisify} = require('util');
const promiseF = promisify(asyncF);

(async function() {
  for (var i = 0; i < 1e6; i++) await promiseF();
  console.time('test');
  for (var i = 0; i < 1e6; i++) await promiseF();
  console.timeEnd('test');
})();

Results:

• 8.4.0: 804.281ms
• master: 805.989ms
• master with V8 extras: 654.518ms

[benjamingr]

Thanks @targos, that's very interesting. Running those on the real (doxbee) benchmark after building Node from your source at that branch I get:

results for 10000 parallel executions, 1 ms per I/O op

file                                      time(ms)  memory(MB)
callbacks-baseline.js                          178       35.48
promises-bluebird-generator.js                 271       42.20
promises-bluebird.js                           332       47.52
promises-ecmascript6-native-promisify.js       572       69.91
promises-ecmascript6-native-base.js            603       94.12
promises-bluebird-asyncawait.js                645      104.20
promises-ecmascript6-asyncawait.js             677      103.06

Platform info:
Darwin 16.7.0 x64
Node.JS 9.0.0-pre
V8 6.0.286.52
Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz × 8

Looks like on V8 6.0 native promisify from v8-extras wins (PR welcome).

Here are the results from your branch merged into node-v8 canary (so on top of v8 6.2):

results for 10000 parallel executions, 1 ms per I/O op

file                                      time(ms)  memory(MB)
callbacks-baseline.js                          148       27.93
promises-bluebird-generator.js                 236       40.37
promises-bluebird.js                           290       48.64
promises-bluebird-asyncawait.js                416       89.74
promises-ecmascript6-native-base.js            474       94.57
promises-ecmascript6-asyncawait.js             493       96.20
promises-ecmascript6-native-promisify.js       512       73.68

Platform info:
Darwin 16.7.0 x64
Node.JS 9.0.0-pre
V8 6.2.248
Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz × 8

---

@kyrylkov thanks! That's very useful, it's also interesting to benchmark bluebird promises with native async/await (interesting result, faster than native async/await)

[targos]

@benjamingr not sure I'm following. My branch is on V8 6.0, not 6.2.

[benjamingr]

@targos updated the above with benchmarks both with V8 6.0 and 6.2 (6.2 by doing a git pull from your branch on top of node-v8's canary branch).