Feature request: Limit number of concurrent e2e tests

[Valian]

Hello 👋 Big thank you for your library, it made it possible to easily test my live_vue project.

Just one small question: I enjoy running all my tests with mix test, no matter if they're e2e or not. Just, starting up to 20 browsers (default --max-case) times out on my machine. I can limit it to --max-cases 4 or similar and then it works, but then it also limits normal unit tests parallelism.

Would it be hard to introduce a config option to reduce number of simultaneously active headless browsers?

[ftes]

Thanks for the question!
As you've noticed, the current implementation is very basic: Launch one browser per test module.
That was easiest to implement, and uses default ExUnit concurrency control.
Pro: Easy to implement. Allow browser config per test module (headless, slow_mo).
Con: Overhead per test module (browser instance not reused). No custom concurrency control.

For comparsion, Playwright (JS client) uses a fixed number of workers (default: 1 worker for every 2 CPU cores). Each worker has one browser instance. So there is a fixed number of browser instances. And these are recycled between test 'modules'. Much like what you describe.

I think that's a good idea. Both for existing users (speed up: don't launch new browser for every module). And also for your slightly more exotic use case. I would like to look into it further. But I think it will require substantial changes.

[ftes]

Naive thought: Use ExUnit group option?

use PhoenixTest.Playwright.Case, async: true, group: :playwright

That would limit concurrency of e2e tests to 1.

[Valian]

That seems like a good workaround. Btw, I'm quite surprised but there is very little difference in test time between running --max-cases 1 and --max-cases 4, like maybe 20% improvement?

Maybe it's because of an overhead to start individual browser instances. Do you think we could reuse browsers across modules? It should speed things up and make it possible to add custom numbers of workers.

[ftes]

there is very little difference in test time between running --max-cases 1 and --max-cases 4, like maybe 20% improvement?

I can't generally confirm that.
Running the tests in this repo, I see nearly 100% difference between --max-cases 1 and --max-cases 4 if I duplicate PhoenixTest.PlaywrightTest.

Tests within a module always run sequentially. The exact numbers will depend on how you distribute your feature tests between modules. But in general there should already be async speed-up potential.

That being said, I think your suggestion is a good one and we should look into constraining the maximum number of browsers launched to far less than number of cores.

I'd be grateful for any pointers/precedents on how to achieve this kind of 'resource sharing' in ExUnit tests.
Browser worker pool, each test module waits until browser is available in setup_all. Unsure how to deal with timeouts - I'd have to bump timeout pretty high.

[Valian]

Yes, I think high timeouts would be necessary, but there seems to be no way around that? Still, it would be only a slow setup, tests itself should be executed equally quickly.

There are some writeups around poolboy - I think it's (was?) used by eg. Ecto

https://elixirforum.com/t/how-to-use-poolboy-in-elixir/1894/6
https://github.com/thestonefox/elixir_poolboy_example

But in general, probably pulling in a good polling library would be fine - it's only for test environment anyway.

[ftes]

Could you check if #86 improves performance for you?
My test results indicate it doesn't help much.
The implementation is mostly AI-generated, I didn't check it too deeply yet.
But I load tested it with varying pool sizes and verified the pooling works.

If results are more promising for you than they were for me I'm happy to invest more time to get it into a maintainable state.

[ftes]

Also see load tests results here:
ftes/phoenix_test_playwright_example#2

The JS client performs slightly better with increasing number of tests.
But non-pooled elixir is reasonably good (8% slower for 256 tests), and pooled elixir is inbetween.

@Valian You could help me further analyze by

• checking if the test assumptions are representative of your requirements (number and complexity of tests)
• verifying the test results by running on your machine

[Valian]

@ftes thank you for that!

I've also used Claude to run that branch and give me an overview. Done 3 rounds of testing for both versions. Results surprised me - it's a bit slower. On the other hand, it shouldn't make that much of a performance difference: it's rather useful for running alongside other tests, without limiting concurrency. I'll play with it a bit more!

[Valian]

So:

1. With higher concurrency it's a bit faster (19.2s or so).
2. Even if not about performance, this change will allow to adjust memory usage to user preferences and run full test suite (unit test + E2E) at the same time without limiting max number of tests running in paralell.

WDYT?

[ftes]

Note: I've only tested chrome so far, no other browsers.

[ftes]

Even if not about performance, this change will allow to adjust memory usage to user preferences and run full test suite (unit test + E2E) at the same time without limiting max number of tests running in paralell.

Good point. And looks like you're not alone:
#86 (comment)

[Valian]

Amazing! I will be testing it soon <3

[Valian]

Ok, it seems to be working nicely! I'm starting it without supervisor in test_helpers (it doesn't seem to be useful in case of a single pool). Performance is slightly better, but the biggest win is lower memory pressure on the system. @s3cur3 probably you'll be happy too 😉

{:ok, _} =
  PhoenixTest.Playwright.BrowserPool.start_link(
    name: :chromium,
    size: 4,
    browser: :chromium
  )

Thank you @ftes !