Add performance benchmark suite

[markerikson]

Update: See this comment for current status and requests for help

Prior and related references: #104 , #398 , #407, #416 , #898 .

I was doing a bit of searching earlier for any links to benchmarks I could find in the issues, and here's the links I found:

https://github.com/jimbolla/react-redux-perf
https://github.com/mweststrate/redux-todomvc
https://github.com/jscriptcoder/stressing-redux
https://github.com/broadsw0rd/react-redux-perf
https://github.com/somebody32/high-performance-redux/tree/gh-pages/assets
https://github.com/coderitual/redux-performance
#686 (comment)
https://github.com/Kalkut/redux-data-frequency-benchmark

I think it would be very valuable if we were to swipe some of these somehow and set up some performance benchmarks. This would be particularly valuable as we evaluate the changes in 5.1 (per #980 ), and the notional future 6.0 and 7.0 releases as laid out in #950 .

[timdorr]

Interestingly, I'm working on a startup (though I haven't given it much attention lately) to provide this kind of tracking of metrics over time. Maybe I should get my ass in gear so we can hook this up and track progress 😄

[markerikson]

Dumping some research notes:

• Perf analysis and capture:
  • https://www.npmjs.com/package/tracealyzer
  • https://github.com/sitespeedio/chrome-trace
  • https://github.com/Rowno/react-benchmark
  • https://github.com/MicrosoftEdge/fps-emitter
  • https://www.npmjs.com/package/performance-mark-metadata
  • https://www.npmjs.com/package/marky
• Performance API:
  • https://developer.mozilla.org/en-US/docs/Web/API/Performance
  • https://www.html5rocks.com/en/tutorials/webperformance/usertiming/
  • https://dev.to/azu/performancemark-with-medata-is-useful-for-real-user-monitoring-54p
  • https://dev.to/hom3chuk/performance-marks-the-missing-manual-part-1-b6
• Other
  • React perf benchmark setup: https://github.com/facebook/react/tree/3b3b7fcbbdc734d7f2a2e509a0b07c0eb053f779/scripts/bench
  • React PR for prod profiling: https://github.com/facebook/react/pull/12886/files

[markerikson]

I've set up an initial standalone perf benchmark experiment repo here:

https://github.com/markerikson/benchmark-react-redux-perf

It builds a CRA version of Jim Bolla's stock ticker benchmark, which has been configured to log FPS changes as performance API events. I can retrieve those from Puppeteer and log them out afterwards.

I have the CRA project modified to define React, Redux, and React-Redux as external libs in a prod build. That lets me easily swap out the React-Redux version by overwriting a script tag and reloading the page (and is sorta similar to what React does in their benchmark setup).

Jim's benchmark is a stock ticker app, and originally had 330 connected stock listing components. I've bumped that up to 1000 for stress testing, and run the test with four versions:

• 5.0.7 (current): steady at 58-60 FPS
• 4.4.9 (old, but still used in a few places): single-digits of 4-5 FPS
• Greg's 5.1 PR branch in Fix getDerivedStateFromProps usage #980 : 35-40 FPS
• My new context POC in React 16 experiment: rewrite React-Redux to use new context API #898 : 58-60 FPS

I haven't done any investigation into why #980 is slower.

My next step is to set up a way to build and run multiple benchmark apps with varying versions of React-Redux and capture the results. Once I've got that figured out, I can move this setup into the React-Redux repo and automate things, then we can start adding more benchmark apps.

After that, we ought to look at instrumenting the codebase for perf timings, probably with a separate "profiling" build or something. I'd be interested in things like how long it takes to run mapState, how long it takes from the start of notifying subscribers to the end, etc.

[markerikson]

Pasting some comments between myself and Greg on Discord:

[8:07 PM] acemarke:oh, wow
[8:08 PM] acemarke: the commits I just pushed are way slower than the prior version
[8:08 PM] acemarke: I think a lot of this has to do with the forwardRef and stuff
[8:08 PM] acemarke: dropped from 24-26 to 17-19
[8:08 PM] acemarke: at 2500 components
[8:08 PM] acemarke: could also be that my implementation isn't very good

[8:10 PM] CelloG:dammit :/
[8:10 PM] CelloG: I'll try removing forwardRef and see what happens
[8:11 PM] CelloG: if that speeds it up we need to add back withRef
[8:20 PM] CelloG: you're right!

[8:27 PM] CelloG: should probably tell the React team
[8:27 PM] CelloG: that's a huge issue, it should be performance-agnostic if it isn't passing an actual ref
[8:48 PM] CelloG: Ok, running the latest version now. This only uses forwardRef if withRef is set to the text "forwardRef" so we can catch folks using withRef set to true and warn them to not rely on getWrappedInstance since it don't exist no more
[8:49 PM] CelloG: And my implementation is now doing better. Let me see how it matches to 5.0.7
[8:51 PM] acemarke: Yeah, I was thinking we'd introduce a new forwardRef option or something
[8:54 PM] CelloG: ok, I just pushed the commit
[9:01 PM] CelloG: this version I get slightly faster than your old one, but still 2 fps slower than 5.0.7
[9:01 PM] CelloG: but we are in the ballpark. I'm going to re-run and see if there is any change
[8:20 PM] CelloG: after disabling forwardRef, mine is faster
[8:20 PM] CelloG: 2 fps
[8:21 PM] CelloG: than 995

[timdorr]

Is there an issue open on the React repo?

[markerikson]

We haven't opened one yet, and I didn't see one when I was looking earlier this evening.

The chat conversation is from Saturday. I just pinged Dan on Twitter, and he said he might be able to take a look.

[markerikson]

Aaaand opened up React issue 13456 referencing this issue.

[sebinsua]

@markerikson I didn't see speedracer in your list of links. :)

[markerikson]

@sebinsua : oh, wow! That looks like exactly what I was sorta trying to build myself :) The biggest difference I immediately see is that my setup will re-run the same benchmark multiple times for multiple versions of React-Redux.

I'm currently using Puppeteer for the headless Chrome - can speedracer use that?

I'd really appreciate any advice you can offer on using speedracer for this scenario.

[markerikson]

Okay, so for anyone visiting this thread who is interested in helping out, here's where things stand at the moment:

• We currently have a passable start on a benchmark harness at https://github.com/markerikson/benchmark-react-redux-perf
• That harness only has one app it can run (a "stock ticker"-type scenario), and really only captures one primary metric (Frames Per Second)
• The harness is in a separate repo, and not part of the React-Redux repo
• The harness currently prints out FPS and Chrome perf trace values for each version of React-Redux it tests against

What I really want is:

• Multiple benchmark apps that represent different scenarios ("a shallow page full of single connected components", "a deep tree with connected components in different places", "an app that actually changes the UI structure over time")
• Hopefully some more metrics captured
• Ability to easily run all this against multiple builds/versions of React-Redux
• Automatic +- change comparisons between versions
• All this integrated into the React-Redux repo as a sub-folder called benchmarks, similar to our existing tests folder

As an immediate goal, I'd appreciate assistance in building out some more benchmark apps, based on both the list of examples I linked at the start of the issue, as well as any other good ideas people have for meaningful example apps.

If you're interested in helping out, please comment in this thread, and we can coordinate from there.

Thanks!

[chirgjn]

I'll start with the single shallow page one, first thing tonight.

Do I need to setup anything or I can just make a react app with CRA?