Chrome 100 results are ... interesting

[krausest]

Results for Chrome 100 look pretty different.
The fastest strategy for clear rows has changed and results for create rows are significantly slower. I think I have to take a closer look before publishing those results as official.
If anyone finds the trick for clear rows please post it here ;-)

https://krausest.github.io/js-framework-benchmark/2022/table_chrome_100_preliminary.html

[krausest]

Looks like #166
Within a RAF call vanillajs clears in ~ 50 msecs.

[krausest]

Vanillajs can be improved significantly by wrapping create, replace and clear in RAF (left with RAF, right without). Not sure if I should commit this as it'll certainly spreads into benchmark implementations.

[fabiospampinato]

Personally I only see a bug in Chrome in here. Simply wrapping clearing in a raf is probably a bug even, now if you clear and append fast enough you won't see the appended rows. I don't see any reason to push frameworks toward this workaround for something that should probably be fixed in Chrome.

[ryansolid]

Yeah RAF has always been slightly faster and it has varied at different times. We made a mark for it because it clearly was gaming it. That being said there are a couple of frameworks that have lower scores on clear and I'm not sure in all cases it is just RAF. I looked at doohtml and didn't see an obvious RAF call.

Also worth noting is slight hit on replace rows. Most frameworks have taken a slight hit. sifr interestingly seems to be good on both these so I might try to see what the difference is.

[krausest]

Well, doohtml's implementation is particularly clear:

js-framework-benchmark/frameworks/keyed/doohtml/js/Main.class.js

Line 121 in 1e01af1

this.tbody.textContent = ''

It's actually what vanillajs does

js-framework-benchmark/frameworks/keyed/vanillajs/src/Main.js

Line 228 in 1e01af1

this.tbody.textContent = "";

Home come doohtml is 24 msecs faster?
I'll try to profile this weekend...

[krausest]

I actually have doubts that the webdriver measurement is correct.
First I removed the CPU slowdown. The webdriver results still show that doohtml is faster than vanillajs:

result vanillajs-keyed_09_clear1k_x8.json min 8.897 max 11.889 mean 10.4755 median 10.78 stddev 0.9645715514039265
result doohtml-keyed_09_clear1k_x8.json min 6.334 max 7.056 mean 6.6656 median 6.6165 stddev 0.3000385901105982

I prepared a branch for puppeteer. How do results look there (measurement isn't strictly comparable since it closes the browser after every benchmark loop):
result vanillajs-keyed_09_clear1k_x8.json min 7.516 max 10.225 mean 8.3141 median 8.1485 stddev 0.84825709546104
result doohtml-keyed_09_clear1k_x8.json min 6.162 max 11.239 mean 7.912800000000002 median 6.6165 stddev 2.207555299420606
doohtml is still a bit faster, but way closer than with webdriver.

If I profile with chrome I can't see any difference between both. I chose a 4x CPU slowdown:
vanillajs:

doohtml

And I think that's how it should be. Both clear tbody.textContent. I changed vanillajs to clear textContent only and skip all other logic, but it didn't change the results.

[krausest]

This was nastier than I thought. I couldn't find the reason for the numbers reported by chromedriver and chrome's profiler showed a different picture.

Some time ago I prepared an alternative benchmark runner that uses puppeteer, but I didn't finalize the switch. But I'm planning to do it now.
The biggest advantage is that it produces traces that can be loaded in the chrome profiler. This makes it much easier to check the results.

The first candidate can be seen here: puppeteer results

And here's an excerpt (left new puppeteer results, right old webdriver results):

One large difference can be seen for doothml and create benchmark. Let's take a look at a trace from puppeteer:

This trace shows (at least together with the others) that a median of 86 msecs appears to be plausible.

But there was another important change to get there. Some frameworks got worse after switching to puppeteer. The traces looked like that:

Here we see that after performing the create operation a timer fires which causes a redraw.
The old logic was to compute the duration from the start of the click event to the last paint event. And this would result in a value of 180 msecs for doohtml! And I guess a similar issue happened for the old results.

How can we measure the duration better? I propose the following: For all benchmarks except select row we compute the duration like that: from the start of the click event to the end of the first paint after the last layout event (usually there's just one layout event).
Cases with multiple layout events happen e.g. for aurelia (replace rows):

There's a first layout event after the click event and a second after the actual dom update happened.
Or for ember (create rows):

There's a first layout event right after the click before a timer event that appears to cause the real redraw.

For select row no layout event should happen.
There are cases with multiple paint event e.g. for fre:

The first paint event happens about 10 msecs after the click but the second one (after 417 msecs) is the correct paint event for our duration.
Or glimmer:

Or marko - which is interesting, since it happens only sometimes for marko:

There are even frameworks that happen to have a layout event (and three paint events) for select rows like reflex-dom:

So for select rows I'm staying with the old logic. The duration is from the starting of the click event to the end of the last paint event. Please note that this leaves the issue open that we solved above for the other benchmarks, but I have no other idea how to measure the duration better if there's no layout event.

@hman61 Sorry, I'm afraid to say that it currently seems like I measured doohtml incorrectly!
Do you have an idea what's causing the second paint in traces like these? Is it the test driver or are there any timers fired in doothml? I think I've never seen them in the chrome profile so it may be the test driver.

[krausest]

I ported the logic back to the webdriver to get a better understanding of the differences between webdriver and puppeteer results.

A few questions arise when comparing puppeteer and webdriver.

1. Why is doohtml much faster for create rows with puppeteer?
2. Why is svelte much faster for replace rows with puppeteer?
3. What's the reason for most frameworks to be faster for replace rows with puppeteer (vanilla, solid, vue)?
4. Is puppeteer slower for select row for react?
5. Why are vanillajs and solid much faster for clear rows with pupeteer?

[krausest]

1.Why is doohtml much faster for create rows with puppeteer?

For svelte the new logic returns the same result for create rows, since there's just a single paint event.

For doohtml there's a difference:

more than one paint event found
paints event {
  type: 'paint',
  ts: 181465296692,
  dur: 629,
  end: 181465297321,
  evt: '{"method":"Tracing.dataCollected","params":{"args":{"data":{"clip":[-16777215,-16777215,16777216,-16777215,16777216,16777216,-16777215,16777216],"frame":"F318F3FB0BBBC36CD591D24FEF975201","layerId":0,"nodeId":212}},"cat":"devtools.timeline,rail","dur":629,"name":"Paint","ph":"X","pid":1938853,"tdur":622,"tid":1,"ts":181465296692,"tts":880680}}'
} 88.744
paints event {
  type: 'paint',
  ts: 181465333324,
  dur: 2633,
  end: 181465335957,
  evt: '{"method":"Tracing.dataCollected","params":{"args":{"data":{"clip":[-16777215,-16777215,16777216,-16777215,16777216,16777216,-16777215,16777216],"frame":"F318F3FB0BBBC36CD591D24FEF975201","layerId":0,"nodeId":212}},"cat":"devtools.timeline,rail","dur":2633,"name":"Paint","ph":"X","pid":1938853,"tdur":2622,"tid":1,"ts":181465333324,"tts":907592}}'
} 127.38
WARNING: New and old measurement code return different values. New  88.744 old 127.38

The mean for doohtml with the new logic is 86.45, i.e. comparable to the puppeteer result.
Seems like the old idea "measure from click till last paint" didn't work right for this framework, since there's a two paint events.

[ryansolid]

Yeah it's interesting DooHTML does very well in the webdriver in areas you wouldn't expect. On average Svelte is a tiny bit better on puppeteer in relation to others but most things do stay in line so to speak other than those 2 (and React's slow select row).

As for 5. I think other than DooHTML which is a bit of an enigma, VanillaJS and SolidJS are the only 2 doing el.textContent = "" to clear. Everything stays in line on clear but I think the difference is more emphasized in puppeteer.

[krausest]

2. Why is svelte much faster for replace rows with puppeteer?

That's somehow different. The old and the new logic return the same result for svelte for replace rows since there's just a single paint event and that event takes about 124 msecs.

The puppeteer trace also looks completely unsuspicious:

I have currently no idea where that large difference comes from.

When I perform replace rows manually and profile the 5th create rows I'm getting that profile in chrome:

Those numbers look closer to the webdriver results.

So I came back to the last resort: Adding some client side hack for measuring the duration ( 89b546f ).
The result is [105, 99.30, 94.30, 97.60, 95.40, 115.10, 111.7, 102.10, 94.80, 87.10, 94.60, 97.9] for puppeteer and [115.80, 145,10, 132, 141.10, 145.30, 133.60, 132.9, 131.9, 138.40, 134.20, 119.70, 139.30] for webdriver. This confirms that the two benchmark drivers observe different performance.
Running manually in chrome gives me numbers close to webdriver.

[Updated 4/9/22]
I'm getting the fast results for a very small list of categories like 'devtools.timeline,disabled-by-default-devtools.timeline'. Those categories are enough to compute the duration from the timeline and show event names.
If I enable full tracing and enable many categories (and it seem like disabled-by-default-devtools.timeline.stack is the one that causes the slow down) I'm getting much slower results, closer to webdriver-ts (but this is not an explanation since the webdriver benchmark driver itself also uses just the small category list).
What's interesting to see what the client hack reveals:

BROWSER: JSHandle:run took 94.79999999701977
BROWSER: JSHandle:run took 96.70000000298023
BROWSER: JSHandle:run took 106.59999999403954
BROWSER: JSHandle:run took 84.90000000596046
BROWSER: JSHandle:run took 93.30000001192093
after initialized  02_replace1k 0 svelte
runBenchmark
BROWSER: JSHandle:run took 111.6000000089407

The replace row operation invokes create rows and then 5 warm up create rows (which replace the content) before the 6th create row is called. The duration of this 6th operation is counted. All 5 replace row invocations are all faster than the final one and I suppose this is due to chrome's profiler which is active only for that last call. With the small list of enabled categories the last run is in the same range as the others:

BROWSER: JSHandle:run took 95
BROWSER: JSHandle:run took 99.5
BROWSER: JSHandle:run took 93
BROWSER: JSHandle:run took 95.30000001192093
BROWSER: JSHandle:run took 99.29999999701977
after initialized  02_replace1k 0 svelte
runBenchmark
BROWSER: JSHandle:run took 97.20000000298023

It seems like the impact of full profiling is much less for the other frameworks. Here's vanillajs and there's hardly a difference for the final run:

BROWSER: JSHandle:run took 85.40000000596046
BROWSER: JSHandle:run took 87.79999999701977
BROWSER: JSHandle:run took 79.59999999403954
BROWSER: JSHandle:run took 96.79999999701977
BROWSER: JSHandle:run took 79.90000000596046
after initialized  02_replace1k 0 vanillajs
runBenchmark
BROWSER: JSHandle:run took 85.5

or react hooks:

BROWSER: JSHandle:RUN took 107.6000000089407
BROWSER: JSHandle:RUN took 133
BROWSER: JSHandle:RUN took 93.20000000298023
BROWSER: JSHandle:RUN took 96.20000000298023
BROWSER: JSHandle:RUN took 99.29999999701977
after initialized  02_replace1k 0 react-hooks
runBenchmark
BROWSER: JSHandle:RUN took 95.09999999403954

=> So if using puppeteer we should use the smallest list of enabled trace categories.

[krausest]

[Updated 4/9/22]
3. What's the reason for most frameworks to be faster for replace rows with puppeteer (vanilla, solid, vue)?

When I first looked into it I ran into the usual trap: I used my default chrome with extension (1password, ublock) enabled.
I really should remember to always use incognito mode for all profiling!

Vanilllajs then gives me results that look close to the puppeteer results:

Most replace rows tasks were all close to 83 msecs.

=> Seems like puppeteer results are ok.

[krausest]

[Updated 4/9/22]
4.Is puppeteer slower for select row for react?

It seems like this was also caused by enabling many tracing categories. With the minimal trace category list it looks pretty similar.

webdriver react-hooks
mean 54.8273 median 55.2395
mean 53.99490000000001 median 53.8345

puppeteer react-hooks
mean 52.32250000000001 median 52.4035 stddev 2.6654604459434195
mean 53.0296 median 52.593500000000006 stddev 2.7626347810257763

[krausest]

5. Why are vanillajs and solid much faster for clear rows with pupeteer?

As for clear rows there is just 1 paint event for vanillajs and solid, so the new logic can't help.
The client side hack reports also numbers in the 70 msecs range.