Development: Playwright E2E client coverage

[muradium]

Checklist

General

• Language: I followed the guidelines for inclusive, diversity-sensitive, and appreciative language.
• I chose a title conforming to the naming conventions for pull requests.

Client

• I strictly followed the client coding and design guidelines.
• I documented the TypeScript code using JSDoc style.

Motivation and Context

It's not possible to understand how many percentage of the system is covered by E2E tests.

Description

This PR adds client code coverage for Playwright E2E tests. It's implemented using Playwright's own coverage feature and monocart-reporter for creating lcov reports.
Coverage information is generated automatically after test execution is finished. On CI, we zip and export the lcov coverage data as .zip archive.

Steps for Testing

• Code Review: Ensure that code quality is good and does not break the existing system.
• Review Coverage Report:

1. Check out artifacts section of build runs: Bamboo Artifacts
2. Run E2E tests and check if coverage report is generated correctly. Coverage reports should be generated on src/test/playwright/test-reportsfolder after test run is finished if running on headless mode or after UI is quit if running on UI mode.

Steps for running the tests:

1. Navigate to src/test/playwright
2. Configure Playwright using playwright.env file based on your local setup. Current configuration should work for default Artemis setup.
3. Run npm install && npm run playwright:setup
4. Run the tests using one of the methods:

• Run npm run playwright:test to run all tests
• Run npm run playwright:open to open the Playwright UI and selectively run few tests

Testserver States

Note

These badges show the state of the test servers.
Green = Currently available, Red = Currently locked

Review Progress

Code Review

• Code Review 1
• Code Review 2

Manual Tests

• Test 1
• Test 2

[egekocabas]

Just a quick note: since Bamboo will soon be removed, we’re actively migrating our E2E test runs to GitHub Actions. In a couple of weeks, the tests should be fully running here. After that, Helios will parse the final test results (results.xml). So, IMO there’s not much value in adding any new Bamboo-specific functionality.

From what I can see, you’re still generating results.xml as the final artifact in JUnit format, and that logic hasn’t changed—please correct me if I’m wrong.

[muradium]

Yes, test results are still in JUnit format, no changes there.
This PR is for coverage report. The part you commented on is only for preparing artifact for Bamboo. On GitHub actions, we can just upload the lcov report directory as an artifact. Do you think it makes sense to skip the CI part for now until we transition to new CI pipeline?

[SimonEntholzer]

Although there is a coverage report, its value is a bit limited, when only the coverage percentage of the whole file is visible, but no information on which lines of the file:

I'd assume that's not the intended behaviour

[muradium]

@SimonEntholzer Locally, reports contain line information as well. On CI it runs on a .war build of Artemis, code sources don't exist. That's why lines can't be mapped back to sources. I'm not sure yet if it makes sense to inline the entire source code into Artemis.war builds to achieve this

[muradium]

We decided to postpone E2E coverage PRs until GitHub actions integration is completed. fyi @egekocabas