Proof of concept: playwright e2e tests

[SteffenDE]

This PR implements a proof of concept of using Playwright to run end to end browser tests that actually run the whole LiveView pipeline from JavaScript to Elixir, no weird mocking necessary.

I'm happy to discuss how to move this forward.

[vanderhoop]

A lot of devs hate end-to-end/feature tests because flakiness and speed, but front-end <-> back-end integration is uniquely crucial in LiveView, so I'd love to see the core happy paths tested end-to-end at the lib level.

I've got a lot of experience de-flaking and optimizing e2e builds, so I'd be down to collaborate/contribute.

Regarding tooling, we already have wallaby in the elixir ecosystem, which allows us to use ExUnit and test_helper.exs for test setup and building out a FeatureCase, so I'd think that would make sense. It would also keep the local and CI dependencies from ballooning too much.

[SteffenDE]

@vanderhoop what makes Playwright special is that it supports all browser engines. Wallaby seems to only support Chromium and Firefox. There is a playwright library for Elixir, but it is not yet in a state comparable to the official libraries (TS, JS, Python, Java, .NET). As this is just about telling the browsers what to do I don't think that it is crucial that the end to end tests are written in Elixir. It would definitely be nice though!

I am using Playwright for testing a big LiveView app at work now for ~2 years and they do a really good job at preventing flakiness.

[SteffenDE]

Adjusting CI pipelines is always fun...

I added some more tests for changes in other recent PRs on mine, but in general this is ready for review (probably @chrismccord).

I know that the style I used in the e2e test files is different than the assets js files, because that's the style I usually use when writing javascript (semicolons, spaces between curly braces, ...) and the assets eslint config does not apply to this folder. A project-wide linter or formatter for the JS code would be nice. If needed, I'll adjust the style accordingly.

I also thought about a way to define the LiveView code that is necessary for the tests in the test files themselves. At work, we have an HTTP endpoint that Code.eval_strings what we send from playwright that is quite handy. That would need adding live routes to the router at runtime though and I'm not sure if that's possible. Basically something like this:

const { elixirEval } = require("../does_not_exist_yet");

test("my test 123", async ({ page, request }) => {
  await elixirEval(request, `
    defmodule LiveView.LiveViewTest.E2E.MyLiveViewTest123 do
      ...

      def render(assigns) do
        ...
      end
    end

    LiveView.LiveViewTest.E2E.Router.somehow_define_live_route_at_runtime("/my-liveview-test-123", LiveView.LiveViewTest.E2E.MyLiveViewTest123)
  `);

  await page.goto("/my-liveview-test-123");
  ...
});

This would be especially nice for self-contained regression tests.

This has the drawback of not getting any compilation errors before starting the tests.

[chrismccord]

@SteffenDE This looks amazing! The dynamic code eval could work by compiling entire routers for each test case, and dispatching from the elixir script endpoint to a "dynamic" router, but I am not sure yet about the tradeoffs. For now, it's probably best for us to simply start with the elixir script server serving up LV's that smoke test all our core features like you've started here.

JS Style wise, I am not overly concerned and can touch up myself if needed.

I'm happy to merge this as is at this point and we can revisit collocated elixir snippets if things get unruly in the future. Is the move-to-first spec blocked by a potential issue on the LV side or just where you left off? Thank you!!!

[SteffenDE]

Awesome, very happy to hear that!

I think the problem with the skipped test is that LiveViewTest and the JS client handle prepend differently, because you recently changed the JS code not to move elements on prepend if they already exist. So actually the problem is probably on the LiveViewTest side. The fix would be to change move-to-first to use stream_delete followed by stream_insert. Maybe I'll find the time to look at this more in-depth tomorrow.

[chrismccord]

Makes sense. The LV test should be using stream_delete + stream_insert anyway. I'll fix it up and unfail the test. Thank you for this!

[chrismccord]

❤️❤️❤️🐥🔥