automated tests should be able to test html5 client

[totaam]

Issue migrated from trac ticket # 2231

component: tests | priority: critical | resolution: fixed

2019-03-21 01:04:35: antoine created the issue

---

And record the client type so we can compare python vs html5.

[totaam]

2019-03-21 01:08:58: antoine changed priority from major to critical

[totaam]

2019-03-21 01:08:58: antoine changed status from new to assigned

[totaam]

2019-03-21 01:08:58: antoine commented

---

Blocker for #2232.

[totaam]

2019-04-04 03:58:01: antoine commented

---

Blocked by #2251

[totaam]

2019-04-23 11:04:21: antoine changed status from assigned to new

[totaam]

2019-04-23 11:04:21: antoine changed owner from antoine to encodedEntropy

[totaam]

2019-04-23 11:04:21: antoine commented

---

Working OK now that #2251 is fixed + minor fixes from r22517.

With one important caveat: the browser session must not be resumed from an earlier one as this would interfere with the testing, so one has to use a profile specifically setup for this testing.
Either a brand new user, or forcing a new profile:

• firefox now uses firefox -P Test: so you must create a "Test" profile
• google-chrome now uses google-chrome --user-data-dir=~/Downloads/TEMP, so you must run mkdir -p ~/Downloads/TEMP once.

Notes:

• more browsers could be added, @encodedEntropy: do you have a standalone one I could use for testing?
• we could also add a setting for testing via https vs http, but again this would require manual steps for accepting the certificates - probably not worth the hassle.
• when running the tests with a browser as client, it doesn't make sense to have XPRA_TEST_ENCODINGS and many of the other options, so r shortcuts that out. Which means that there are fewer browser tests than python client tests sets. Adjust the test options accordingly.
• do we also want to set the window size so that the test setup is more reproducible? Firefox has an option, google-chrome does not. (so it would require an xdotool hack)
• I have seen firefox occasionally failing to startup and getting an error page instead of the html5 client - not sure why that is yet. It means the corresponding test data is garbage.. Another time, it came up with a "do you want to refresh" dialog... PITA

@encodedEntropy: I think there's enough there to hand over to you, let's get some data and iterate.

[totaam]

2019-08-01 12:59:57: smo changed owner from encodedEntropy to smo

[totaam]

2019-08-07 12:19:24: smo commented

---

Browsers are working great with automated tests. The only feedback I have is what you have said already is that we may want to set the window size but in my tests my window always starts the same size.

Firefox window seems to be a good size but the chrome one sometimes cuts off the window of the application being tested.

Not sure if this matters all that much since it is the same size every time.

[totaam]

2019-08-07 12:19:55: smo changed owner from smo to Antoine Martin

[totaam]

2019-08-08 06:37:37: antoine changed owner from Antoine Martin to smo

[totaam]

2019-08-08 06:37:37: antoine commented

---

Can you post some data?

[totaam]

2019-08-08 17:14:37: smo changed owner from smo to Antoine Martin

[totaam]

2019-08-08 17:14:37: smo commented

---

I'm attaching some perf tests comparing firefox and chrome. I wanted to compare them with the python2 client as well but the data doesn't seem to line up.

[totaam]

2019-08-08 17:14:57: smo uploaded file comparebrowser.tar.gz (141.4 KiB)

Compare firefox and chrome with charts

[totaam]

2019-08-11 07:19:40: antoine changed owner from Antoine Martin to smo

[totaam]

2019-08-11 07:19:40: antoine commented

---

Interesting data points:

• memscroller is killing the latency (#2382): maximum goes up to 7 seconds with chrome!
• memscroller aside, Firefox is doing better than chrome in almost all key metrics: batch delay, damage latency, quality, decoding speed, regions and pixels per second, but it does use quite a bit more cpu client side

Can you post some comparison of python vs html5? On auto, so we can see the gap in performance.
(then I think we can close this ticket)

[totaam]

2019-08-14 01:22:40: smo uploaded file compareclients.tar.gz (142.2 KiB)

compares all clients

[totaam]

2019-08-14 01:23:58: smo commented

---

Attached is a comparison of all the 3 clients. I had the data before but I had to fix the script for the charts to compare them.

Also changed the css a bit to be able to read things.

[totaam]

2019-08-14 01:24:10: smo changed owner from smo to Antoine Martin

[totaam]

2019-08-14 07:31:21: antoine uploaded file avg-batch-delay.png (28.5 KiB)

average batch delay

[totaam]

2019-08-14 07:31:34: antoine uploaded file regions-per-second.png (29.2 KiB)

regions per second

[totaam]

2019-08-14 07:33:30: antoine changed status from new to closed

[totaam]

2019-08-14 07:33:30: antoine set resolution to fixed

[totaam]

2019-08-14 07:33:30: antoine commented

---

Also changed the css a bit to be able to read things.
Nice, thanks!

First thing to mention is that the decoding-pixels-per-second values for the html5 client are wrong: javascript is not decoding 50GPixels per second! (nothing can - I will try to fix it)

As expected, the python client is way ahead of the html5 client:

• best network send speed of 20MB/s vs 3MB/s
• max damage latency: ~50ms vs 7000! (because of memscroller: #2382)
• max batch delay at 22ms, vs over 300.. (memscroller again - but overall quite bad with Chrome)
• the key metrics: more pixels, much better batch delay, more regions-per-second:
  [[Image(avg-batch-delay.png)]]
  [[Image(regions-per-second.png)]]

That said, the html5 client does better than I thought it would do. (it is sacrificing framerate and quality in the pure-video glxgears test - but managing OK)
Also interesting to see that Firefox does quite a bit better than Chrome overall.

[totaam]

2019-08-21 08:59:45: antoine commented

---

Fix for the decoding latency reported by the html5 client: r23551.

[totaam]

2020-05-30 05:58:12: antoine commented

---

r23551 was completely wrong: both Performance.now and Date.now return time values in milliseconds!
r26526 reverts it.