Run e2e load test to compare to playwright JS client

[ftes]

ℹ️ If you have feedback, consider commenting on ftes/phoenix_test_playwright#86

---

Run load test to compare phoenix_test_playwright (ex) performance to official playwright javascript (js) client.
Measure total test execution time for combinations of concurrency C and parallelizable tests T.
Also partially evaluate experimental browser pool ftes/phoenix_test_playwright#86 ex_pool.

Chrome only.

Findings

ex performance is on par js in most considered constellations.
For js and ex, C=number of CPU cores is optimal.
For ex, the ExUnit default --max-cases = 2x cores is nearly as good.
js performance scales slightly better for large number of tests by reusing browsers.
For C=16/T=256 there was an 8% difference (134.0s vs 145.1s).
ex_pool scales better than ex, but worse than js.

Concurrency C

• js: --workers
  • each worker starts a persistent browser instance, which is reused across test suites
• ex: --max-cases
  • phoenix_test_playwright does not reuse browser instances
  • a new browser instance is launched per test suite
• ex: --max-cases and browser_pool_size
  • ExUnit concurrency and browser pool size in lockstep. This is not required, they could be changed independently. But since results for ex with C=16 and C=32 are similar, there would likely be little difference.

Tests T

Number of test execution units that can be run concurrently.

• js: number of tests (fullyParallel = true used, so that they are run concurrently even when in a single test suite)
• ex and ex_pool: number of test suites (parameterized), with 1 test each

For both js and ex/ex_pool, each execution unit contains 1 long-running test (register flow repeated 10 times).

Result details

concurrency C	tests T	time js (s)	time ex (s)	time ex_pool (s)	throughput js (tests/s)	throughput ex (tests/s)	throughput ex_pool (tests/s)
1	1	6.8	6.3		0.15	0.16
1	2	12.2	11.7		0.16	0.17
1	4	23.5	22.7		0.17	0.18
1	8	47.5	46.1		0.17	0.17
2	2	6.7	6.6		0.30	0.30
2	4	12.6	12.3		0.32	0.33
2	8	23.2	22.7		0.34	0.35
2	16	43.7	43.3		0.37	0.37
4	4	7.1	6.8		0.56	0.59
4	8	13.5	13.6		0.59	0.59
4	16	24.9	24.8		0.64	0.65
4	32	44.7	46.0		0.72	0.70
8	8	7.3	7.1		1.10	1.13
8	16	13.6	13.7		1.18	1.17
8	32	23.8	24.8		1.34	1.29
8	64	47.0	47.4		1.36	1.35
16	16	9.5	9.4	8.6	1.68	1.70	1.86
16	32	14.7	16.7	15.4	2.18	1.92	2.08
16	64	29.0	30.1	28.9	2.21	2.13	2.21
16	128	59.0	63.4	60.1	2.17	2.02	2.13
16	256	134.0	145.1	141.1	1.91	1.76	1.81
32	32	18.0	16.5		1.78	1.94
32	64	35.5	30.2		1.80	2.12
32	128	81.8	65.0	61.6	1.56	1.97	2.08

Notes

• end to end tests for better comparison: run tests against standalone MIX_ENV=prod mix phx.server
  • no ecto sandboxing
• hardware: Apple M4 Pro (14 cores = 10 regular + 4 efficiency)
• drop database between runs (observed slowdown otherwise)
• ex_pool code see Run e2e load test to compare browser pooling to playwright JS client  #3

Test script (roughly)

mix phx.server&

mix ecto.reset
time (cd assets; npx playwright test --workers $C)
mix ecto.reset
time mix test test/features/register_test.exs --max-cases $C