Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Development: Playwright E2E client coverage #10466

Draft
wants to merge 35 commits into
base: develop
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
3d95c6f
Use monocart-reporter and auto-fixture for collecting Playwright cove…
muradium Jan 6, 2025
03c2621
Add script for archiving coverage reports for artifact extraction
muradium Jan 6, 2025
b42ed3e
Add script for merging coverage reports
muradium Jan 6, 2025
00e0592
Zip coverage results after merging, disable css coverage, disable raw…
muradium Jan 9, 2025
1d14b3d
Extra logs
muradium Jan 9, 2025
9784e14
Change merge coverage script into mjs script
muradium Jan 9, 2025
ee0dc3e
Fix the names of cleaned-up reports on compose command
muradium Jan 9, 2025
6c7900a
Use istanbul to efficiently merge json coverage reports
muradium Jan 9, 2025
67c12a9
Upload only the coverage of parallel tests as artifact to check if th…
muradium Jan 10, 2025
4421b5a
Merge branch 'develop' into feature/playwright/test-coverage
muradium Jan 10, 2025
db814e2
Try using artemis-app instead of nginx as the base url of tests
muradium Jan 10, 2025
759b0f2
Revert "Try using artemis-app instead of nginx as the base url of tests"
muradium Jan 10, 2025
12bcab1
Enable inlineSources to make sure webapp sources are still present in…
muradium Jan 10, 2025
c15657c
Revert "Upload only the coverage of parallel tests as artifact to che…
muradium Jan 12, 2025
0f5046e
Revert "Enable inlineSources to make sure webapp sources are still pr…
muradium Jan 12, 2025
5967396
Execute only a small portion of tests to avoid heap memory limits
muradium Jan 12, 2025
4e68100
Enable inlineSources to make sure webapp sources are still present in…
muradium Jan 10, 2025
7b63c9b
Try increased heap size for the whole test set
muradium Jan 19, 2025
5e29c1d
Remove raw report remains
muradium Jan 19, 2025
6a31344
Archive the original unmerged report
muradium Jan 23, 2025
55b5b85
Use artemis-app files for coverage generation
muradium Jan 26, 2025
f113623
Fix base url by using the value from env
muradium Jan 27, 2025
923c394
Enable all tests back to test coverage performance
muradium Jan 27, 2025
a39c735
Revert to archiving the whole coverage report
muradium Jan 27, 2025
991692f
Zip all client coverage reports separately
muradium Jan 31, 2025
0b2baf6
Archive the whole report
muradium Jan 31, 2025
3e7bffe
Improve the merge coverage script
muradium Feb 2, 2025
0189964
Try coverage without inlining sources
Feb 10, 2025
f298e61
Minor improvements
Feb 10, 2025
dcfed29
Merge branch 'develop' into feature/playwright/test-coverage
muradium Mar 2, 2025
8d36858
Merge branch 'develop' into feature/playwright/test-coverage
muradium Mar 9, 2025
1c9d3a3
Filtering coverage after report generation and mapping sources
muradium Mar 9, 2025
f87438b
Add config file extension on import
muradium Mar 10, 2025
58d7bdc
Move coverage filters to merge script
muradium Mar 10, 2025
5063b26
Refactor coverage filtering logic in merge-coverage-reports
muradium Mar 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docker/playwright-E2E-tests-mysql-localci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ services:
depends_on:
mysql:
condition: service_healthy
ports:
- "8080:8080"
env_file:
- ./artemis/config/playwright.env
- ./artemis/config/playwright-local.env
Expand Down
2 changes: 1 addition & 1 deletion docker/playwright.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ services:
npm ci &&
npm run playwright:setup &&
npm run playwright:test;
rm ./test-reports/results-fast.xml ./test-reports/results-slow.xml
rm ./test-reports/results-parallel.xml ./test-reports/results-sequential.xml
'
volumes:
- ..:/app/artemis
Expand Down
105 changes: 105 additions & 0 deletions src/test/playwright/merge-coverage-reports.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
// merge-coverage-reports.mjs
import path from 'path';
import { fileURLToPath } from 'url';
import archiver from 'archiver';
import coverage from 'istanbul-lib-coverage';
import reports from 'istanbul-reports';
import libReport from 'istanbul-lib-report';
import fsAsync from 'fs/promises';
import fs from 'fs';

/**
* Represents a configuration object used for filtering file paths during coverage report generation.
*
* @property {string[]} includePaths - An array of file path patterns to include in the coverage analysis.
* @property {string[]} excludePaths - An array of file path patterns to exclude from the coverage analysis.
*/
const coverageFilters = {
includePaths: ['src/main/webapp'],
excludePaths: [],
};

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

const coverageParallelDir = path.join(__dirname, 'test-reports/monocart-report-parallel');
const coverageSequentialDir = path.join(__dirname, 'test-reports/monocart-report-sequential');
const coverageDir = path.join(__dirname, 'test-reports/client-coverage');
const lcovDir = path.join(coverageDir, 'lcov-report');

console.log(`Merging coverage reports`);

Check failure on line 30 in src/test/playwright/merge-coverage-reports.mjs

View workflow job for this annotation

GitHub Actions / client-style

'console' is not defined

const coverageParallel = JSON.parse(fs.readFileSync(path.join(coverageParallelDir, '/coverage/coverage-final.json'), 'utf8'));
const coverageSequential = JSON.parse(fs.readFileSync(path.join(coverageSequentialDir, '/coverage/coverage-final.json'), 'utf8'));

// Apply filters to coverage data before merging
const filteredCoverageParallel = filterCoverageData(coverageParallel, coverageFilters);
const filteredCoverageSequential = filterCoverageData(coverageSequential, coverageFilters);

const mapA = coverage.createCoverageMap(filteredCoverageParallel);
const mapB = coverage.createCoverageMap(filteredCoverageSequential);
mapA.merge(mapB);

const context = libReport.createContext({
dir: lcovDir,
coverageMap: mapA,
});

const htmlReport = reports.create('html');
const lcovReport = reports.create('lcovonly', { file: 'lcov.info' });

htmlReport.execute(context);
lcovReport.execute(context);

console.log(`Merged coverage reports successfully`);

Check failure on line 54 in src/test/playwright/merge-coverage-reports.mjs

View workflow job for this annotation

GitHub Actions / client-style

'console' is not defined

await fsAsync.rm(coverageParallelDir, { recursive: true, force: true });
await fsAsync.rm(coverageSequentialDir, { recursive: true, force: true });

// Bamboo can upload only files as an artifact, not directories
// That's why we archive the lcov coverage directory on CI to prepare it as an artifact
Comment on lines +59 to +60
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a quick note: since Bamboo will soon be removed, we’re actively migrating our E2E test runs to GitHub Actions. In a couple of weeks, the tests should be fully running here. After that, Helios will parse the final test results (results.xml). So, IMO there’s not much value in adding any new Bamboo-specific functionality.

From what I can see, you’re still generating results.xml as the final artifact in JUnit format, and that logic hasn’t changed—please correct me if I’m wrong.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, test results are still in JUnit format, no changes there.
This PR is for coverage report. The part you commented on is only for preparing artifact for Bamboo. On GitHub actions, we can just upload the lcov report directory as an artifact. Do you think it makes sense to skip the CI part for now until we transition to new CI pipeline?

if (process.env.CI === 'true') {

Check failure on line 61 in src/test/playwright/merge-coverage-reports.mjs

View workflow job for this annotation

GitHub Actions / client-style

'process' is not defined
try {
await createArchive(path.join(coverageDir, 'e2e-client-coverage.zip'), lcovDir);
} catch (err) {
console.error('Error while creating archives:', err);

Check failure on line 65 in src/test/playwright/merge-coverage-reports.mjs

View workflow job for this annotation

GitHub Actions / client-style

'console' is not defined
}
}

// Archives the directory
async function createArchive(outputPath, inputDirectory) {
const output = await fs.createWriteStream(outputPath);
const archive = archiver('zip', { zlib: { level: 9 } });

output.on('close', () => {
console.log(`Coverage report archived on: ${outputPath}`);

Check failure on line 75 in src/test/playwright/merge-coverage-reports.mjs

View workflow job for this annotation

GitHub Actions / client-style

'console' is not defined
});

archive.on('error', (err) => {
throw err;
});

archive.pipe(output);
archive.directory(inputDirectory, '', (entry) => entry);
await archive.finalize();
}

/**
* Filter coverage data based on include and exclude path filters
* @param {Object} coverageData - The coverage data to filter
* @param {Object} filters - The filters to apply
* @returns {Object} - The filtered coverage data
*/
function filterCoverageData(coverageData, filters) {
Object.keys(coverageData).forEach(filePath => {
const shouldInclude = filters.includePaths.length === 0 ||
filters.includePaths.some(includePath => filePath.includes(includePath));
const shouldExclude = filters.excludePaths.some(excludePath => filePath.includes(excludePath));

if (!(shouldInclude && !shouldExclude)) {
delete coverageData[filePath];
}
});

return coverageData;
}
Loading
Loading