Add exhaustive Playwright E2E tests for debrief UI by deacon-mp · Pull Request #80 · mitre/debrief

deacon-mp · 2026-03-16T14:53:41Z

Summary

Adds 32 Playwright E2E tests covering debrief plugin page load, operation selection, report tabs (stats, agents, steps, tactics, facts), PDF download, JSON export, D3 graph rendering, graph settings modal, and error states
Tests run against a full Caldera instance using CALDERA_URL env var (default http://localhost:8888)
Includes Playwright config, package.json, and test spec file

Test plan

Install dependencies with npm install && npx playwright install
Start a Caldera instance with debrief plugin enabled
Run CALDERA_URL=http://localhost:8888 npx playwright test
Verify all 32 tests pass against a live instance
Verify PDF/JSON export mocks trigger correct API calls
Verify error states handle API failures gracefully

Covers plugin page load, operation selection, report tabs (stats, agents, steps, tactics, facts), PDF download, JSON export, D3 graph rendering, graph settings modal, and error states. Tests run against a full Caldera instance via CALDERA_URL env var.

Copilot

Pull request overview

Adds a Playwright E2E suite to validate the Debrief UI against a running Caldera instance, including core flows (operation selection, tabs, graphs) and export/error behaviors.

Changes:

Added a comprehensive Playwright spec covering debrief page load, operation selection, report tabs, exports (PDF/JSON), graph UI, and error states.
Introduced Playwright runner configuration (single-worker, auth credentials, reporter, baseURL).
Added a minimal Node package setup to install and run the E2E tests.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File	Description
tests/e2e/debrief.spec.js	Adds Debrief E2E tests with a mix of live calls and mocked API routes.
playwright.config.js	Defines Playwright configuration (baseURL, credentials, retries, reporter, chromium project).
package.json	Adds Playwright as a dev dependency and scripts to run E2E tests.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

deacon-mp · 2026-03-16T22:03:57Z

tests/e2e/debrief.spec.js

+// Helper: navigate to the debrief plugin page inside magma
+// ---------------------------------------------------------------------------
+async function navigateToDebrief(page) {
+  await page.goto(`${CALDERA_URL}${PLUGIN_ROUTE}`, { waitUntil: 'networkidle' });


Fixed — replaced with domcontentloaded + explicit waitFor on the Debrief heading element.

deacon-mp · 2026-03-16T22:03:59Z

tests/e2e/debrief.spec.js

+    await select.selectOption({ index: 1 });
+    await page.waitForTimeout(2000);


Fixed — all waitForTimeout calls replaced with waitForResponse or waitFor on UI locators.

deacon-mp · 2026-03-16T22:04:01Z

tests/e2e/debrief.spec.js

+  test('should fetch operations from the API', async ({ page }) => {
+    const response = await page.request.get(`${CALDERA_URL}/api/v2/operations`);
+    expect(response.ok()).toBeTruthy();
+    const ops = await response.json();
+    expect(Array.isArray(ops)).toBeTruthy();
+  });


Fixed — removed the unmocked live API test entirely.

deacon-mp · 2026-03-16T22:04:03Z

tests/e2e/debrief.spec.js

+    await navigateToDebrief(page);
+    // Playback buttons: fast-backward, backward, play/pause, forward, fast-forward
+    const buttons = page.locator('#debrief-graph .buttons button');
+    await expect(buttons).toHaveCount(6, { timeout: 15_000 });


Fixed — comment updated to account for the legend toggle button (6 total).

deacon-mp · 2026-03-16T22:04:05Z

tests/e2e/debrief.spec.js

+    await page.route('**/api/v2/operations', (route) =>
+      route.fulfill({
+        status: 200,
+        contentType: 'application/json',
+        body: JSON.stringify(MOCK_OPERATIONS),
+      })
+    );
+    await page.route('**/plugin/debrief/report', (route) =>
+      route.fulfill({
+        status: 200,
+        contentType: 'application/json',
+        body: JSON.stringify(MOCK_REPORT),
+      })
+    );
+    await page.route('**/plugin/debrief/graph**', (route) =>
+      route.fulfill({
+        status: 200,
+        contentType: 'application/json',
+        body: JSON.stringify({ nodes: [], links: [] }),
+      })
+    );
+    await page.route('**/plugin/debrief/sections', (route) =>
+      route.fulfill({
+        status: 200,
+        contentType: 'application/json',
+        body: JSON.stringify(MOCK_SECTIONS),
+      })
+    );
+    await page.route('**/plugin/debrief/logos', (route) =>
+      route.fulfill({
+        status: 200,
+        contentType: 'application/json',
+        body: JSON.stringify({ logos: [] }),
+      })
+    );


Fixed — extracted mockDebriefRoutes(page, overrides?) helper that eliminates ~400 lines of copy-pasted route mocking.

deacon-mp · 2026-03-16T22:04:08Z

playwright.config.js

+    httpCredentials: {
+      username: process.env.CALDERA_USER || 'admin',
+      password: process.env.CALDERA_PASS || 'admin',
+    },


Fixed — CI now throws if CALDERA_USER/CALDERA_PASS env vars are missing. Defaults only apply locally.

deacon-mp · 2026-03-16T22:04:10Z

tests/e2e/debrief.spec.js

+// Helper: navigate to the debrief plugin page inside magma
+// ---------------------------------------------------------------------------
+async function navigateToDebrief(page) {
+  await page.goto(`${CALDERA_URL}${PLUGIN_ROUTE}`, { waitUntil: 'networkidle' });


Fixed — removed duplicate CALDERA_URL from spec file, now uses Playwright's baseURL config.

…e bugs - Extract mockDebriefRoutes() helper to eliminate ~400 lines of copy-pasted route mocking across 15 tests - Replace all waitForTimeout() calls with deterministic waits (waitForResponse, waitFor on locators) - Replace networkidle with domcontentloaded + explicit heading wait - Remove unmocked live API test that would fail without running server - Fix playback button count comment (6 buttons including legend toggle) - Use baseURL from playwright config instead of duplicating CALDERA_URL - Fix file handle leak in c_story.py adjust_icon_svgs - Add viewBox null check in c_story.py - Fix stale op_id bug in debrief_svc.py build_steps_d3

In CI (process.env.CI), CALDERA_USER and CALDERA_PASS must be set explicitly via env vars — config throws if missing. Locally, defaults to admin/admin for developer convenience.

Copilot

Pull request overview

Adds a Playwright-based E2E test suite intended to validate the Debrief UI against a running Caldera instance (configurable via CALDERA_URL).

Changes:

Introduces a large Playwright spec covering Debrief page load, operation selection, report tabs, exports, graphs, and error states.
Adds Playwright runner configuration (single-worker, HTML + list reporters, baseURL, credentials).
Adds a minimal package.json to install/run Playwright tests.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 15 comments.

File	Description
tests/e2e/debrief.spec.js	New Debrief E2E spec with extensive UI coverage plus request stubbing.
playwright.config.js	Playwright configuration for running the E2E suite against a Caldera instance.
package.json	Adds Playwright dependency and npm scripts to execute the E2E suite.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/e2e/debrief.spec.js

+    await select.selectOption({ index: 1 });
+    await page.waitForTimeout(1000);


tests/e2e/debrief.spec.js

+    const select = page.locator('select').first();
+    await select.selectOption({ index: 1 });
+    await page.waitForTimeout(2000);
+    // Page should not crash
+    await expect(page.locator('h2', { hasText: 'Debrief' })).toBeVisible();


tests/e2e/debrief.spec.js

+    const response = await page.request.get(`${CALDERA_URL}/api/v2/operations`);
+    expect(response.ok()).toBeTruthy();
+    const ops = await response.json();
+    expect(Array.isArray(ops)).toBeTruthy();


tests/e2e/debrief.spec.js

+  test('should show tabs when an operation is selected (mocked)', async ({ page }) => {
+    await page.route('**/api/v2/operations', (route) =>
+      route.fulfill({
+        status: 200,
+        contentType: 'application/json',
+        body: JSON.stringify(MOCK_OPERATIONS),
+      })
+    );
+    await page.route('**/plugin/debrief/report', (route) =>
+      route.fulfill({
+        status: 200,
+        contentType: 'application/json',
+        body: JSON.stringify(MOCK_REPORT),
+      })
+    );
+    await page.route('**/plugin/debrief/graph**', (route) =>
+      route.fulfill({
+        status: 200,
+        contentType: 'application/json',
+        body: JSON.stringify({ nodes: [], links: [] }),
+      })
+    );
+    await page.route('**/plugin/debrief/sections', (route) =>
+      route.fulfill({
+        status: 200,
+        contentType: 'application/json',
+        body: JSON.stringify(MOCK_SECTIONS),
+      })
+    );
+    await page.route('**/plugin/debrief/logos', (route) =>
+      route.fulfill({
+        status: 200,
+        contentType: 'application/json',
+        body: JSON.stringify({ logos: [] }),
+      })
+    );


tests/e2e/debrief.spec.js

+// Helper: navigate to the debrief plugin page inside magma
+// ---------------------------------------------------------------------------
+async function navigateToDebrief(page) {
+  await page.goto(`${CALDERA_URL}${PLUGIN_ROUTE}`, { waitUntil: 'networkidle' });


tests/e2e/debrief.spec.js

+const CALDERA_URL = process.env.CALDERA_URL || 'http://localhost:8888';
+const PLUGIN_ROUTE = '/#/plugins/debrief';
+
+// ---------------------------------------------------------------------------
+// Helper: navigate to the debrief plugin page inside magma
+// ---------------------------------------------------------------------------
+async function navigateToDebrief(page) {
+  await page.goto(`${CALDERA_URL}${PLUGIN_ROUTE}`, { waitUntil: 'networkidle' });


tests/e2e/debrief.spec.js

+    await page.waitForTimeout(1000);
+
+    await page.locator('button', { hasText: 'Download PDF Report' }).click();
+    await page.waitForTimeout(500);
+    // Click the Download button inside the modal
+    const downloadBtn = page.locator('.modal.is-active button', { hasText: 'Download' });
+    await downloadBtn.click();
+    await page.waitForTimeout(2000);


tests/e2e/debrief.spec.js

+    await page.waitForTimeout(2000);
+    expect(reportRequested).toBeTruthy();
+  });
+});


tests/e2e/debrief.spec.js

+    const select = page.locator('select').first();
+    await select.selectOption({ index: 1 });
+    await page.waitForTimeout(1000);
+


playwright.config.js

+    httpCredentials: {
+      username: process.env.CALDERA_USER || 'admin',
+      password: process.env.CALDERA_PASS || 'admin',
+    },


Copilot

Pull request overview

Adds a Playwright-based E2E test suite targeting the Debrief UI, plus a couple of small backend robustness fixes that support Debrief graph/PDF functionality.

Changes:

Add Playwright configuration + npm scripts/dependency for running E2E tests.
Add a Debrief-focused Playwright spec with route-mocking helpers and coverage of key UI flows + error scenarios.
Fix Debrief backend issues: safer SVG processing and correct operation IDs in steps D3 graph output.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/e2e/debrief.spec.js	New Playwright spec covering Debrief UI flows with shared route mocks and error-state scenarios
playwright.config.js	Playwright runner config using `CALDERA_URL` baseURL and CI credential requirements
package.json	Adds Playwright dev dependency + convenience scripts
app/objects/c_story.py	Avoid crash on SVGs missing `viewBox`; use context manager for writing SVG output
app/debrief_svc.py	Fix incorrect `op_id` usage in steps D3 graph generation (use `operation.id`)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/e2e/debrief.spec.js

+      '**/api/v2/operations': (route) => route.abort('timedout'),
+    });
+    await navigateToDebrief(page);
+    await expect(page.locator('h2', { hasText: 'Debrief' })).toBeVisible();


tests/e2e/debrief.spec.js

+  // Wait for the report API response to arrive
+  await page.waitForResponse((resp) => resp.url().includes('/plugin/debrief/report'), { timeout: 10_000 }).catch(() => {});


tests/e2e/debrief.spec.js

+        route.fulfill({ status: 500, body: 'Server Error' }),
+    });
+    await navigateToDebrief(page);
+    await expect(page.locator('h2', { hasText: 'Debrief' })).toBeVisible();


tests/e2e/debrief.spec.js

+        route.fulfill({ status: 500, body: 'Report generation failed' }),
+    });
+    await navigateToDebrief(page);
+    await selectFirstOperation(page);


tests/e2e/debrief.spec.js

+  test('should handle PDF download failure gracefully', async ({ page }) => {
+    await mockDebriefRoutes(page, {
+      '**/plugin/debrief/pdf': (route) =>
+        route.fulfill({ status: 500, body: 'PDF generation failed' }),
+    });
+    await navigateToDebrief(page);
+    await selectFirstOperation(page);
+
+    await page.locator('button', { hasText: 'Download PDF Report' }).click();
+    await page.locator('.modal.is-active').waitFor({ state: 'visible', timeout: 5_000 });
+    const downloadBtn = page.locator('.modal.is-active button', { hasText: 'Download' });
+    await downloadBtn.click();
+    await page.waitForResponse((resp) => resp.url().includes('/plugin/debrief/pdf'), { timeout: 10_000 }).catch(() => {});
+    await expect(page.locator('h2', { hasText: 'Debrief' })).toBeVisible();
+  });


tests/e2e/debrief.spec.js

+    await selectFirstOperation(page);
+
+    await page.locator('button', { hasText: 'Download Operation JSON' }).click();
+    await page.waitForResponse((resp) => resp.url().includes('/plugin/debrief/json'), { timeout: 10_000 }).catch(() => {});


Timeout error test now verifies the operations dropdown has no options loaded, not just that the page heading is still visible.

Copilot

Pull request overview

Adds a Playwright-based E2E suite to validate the Debrief plugin UI against a running Caldera instance, while also including a couple of small backend robustness fixes that support Debrief rendering/export behavior.

Changes:

Introduces Playwright configuration and a new Debrief-focused E2E spec covering page load, operation selection, tabs, exports, graphs, and error states.
Adds a minimal Node package.json for installing/running Playwright tests.
Fixes Debrief D3 steps graph generation to use the correct operation id, and hardens SVG processing/writing in report generation.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/e2e/debrief.spec.js	Adds Debrief Playwright E2E tests plus shared route-mocking helpers.
playwright.config.js	Configures Playwright (baseURL, credentials, reporters, retries/workers).
package.json	Adds Playwright test dependency and run scripts.
app/objects/c_story.py	Skips SVGs without a viewBox and uses a safe file handle when writing.
app/debrief_svc.py	Fixes steps D3 graph node/link IDs to reference the correct operation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/e2e/debrief.spec.js

+  await select.selectOption({ index: 1 });
+  // Wait for the report API response to arrive
+  await page.waitForResponse((resp) => resp.url().includes('/plugin/debrief/report'), { timeout: 10_000 }).catch(() => {});


tests/e2e/debrief.spec.js

+  // Wait for the report API response to arrive
+  await page.waitForResponse((resp) => resp.url().includes('/plugin/debrief/report'), { timeout: 10_000 }).catch(() => {});


tests/e2e/debrief.spec.js

+    const downloadBtn = page.locator('.modal.is-active button', { hasText: 'Download' });
+    await downloadBtn.click();
+    await page.waitForResponse((resp) => resp.url().includes('/plugin/debrief/pdf'), { timeout: 10_000 }).catch(() => {});
+    expect(pdfRequested).toBeTruthy();
+  });


tests/e2e/debrief.spec.js

+    await selectFirstOperation(page);
+
+    await page.locator('button', { hasText: 'Download Operation JSON' }).click();
+    await page.waitForResponse((resp) => resp.url().includes('/plugin/debrief/json'), { timeout: 10_000 }).catch(() => {});


tests/e2e/debrief.spec.js

+  test('should show tabs when an operation is selected', async ({ page }) => {
+    await mockDebriefRoutes(page);
+    await navigateToDebrief(page);
+    await selectFirstOperation(page);


deacon-mp requested a review from Copilot March 16, 2026 14:53

Copilot started reviewing on behalf of deacon-mp March 16, 2026 14:56 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

deacon-mp mentioned this pull request Mar 16, 2026

Refactored Playwright E2E tests deacon-mp/debrief#3

Open

Address Copilot review: require credentials in CI, default only locally

8667bed

In CI (process.env.CI), CALDERA_USER and CALDERA_PASS must be set explicitly via env vars — config throws if missing. Locally, defaults to admin/admin for developer convenience.

deacon-mp closed this Mar 16, 2026

deacon-mp reopened this Mar 16, 2026

deacon-mp requested a review from Copilot March 16, 2026 22:13

Copilot started reviewing on behalf of deacon-mp March 16, 2026 22:14 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

deacon-mp requested a review from Copilot March 16, 2026 22:21

Copilot started reviewing on behalf of deacon-mp March 16, 2026 22:21 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

Address Copilot review: assert empty dropdown on network timeout

ec3101c

Timeout error test now verifies the operations dropdown has no options loaded, not just that the page heading is still visible.

deacon-mp requested a review from Copilot March 16, 2026 22:32

Copilot started reviewing on behalf of deacon-mp March 16, 2026 22:33 View session

Copilot AI reviewed Mar 16, 2026

View reviewed changes

deacon-mp merged commit 502d509 into master Mar 16, 2026
5 checks passed

deacon-mp deleted the test/exhaustive-ui-e2e-tests branch March 16, 2026 23:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add exhaustive Playwright E2E tests for debrief UI#80

Add exhaustive Playwright E2E tests for debrief UI#80
deacon-mp merged 4 commits intomasterfrom
test/exhaustive-ui-e2e-tests

deacon-mp commented Mar 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

deacon-mp Mar 16, 2026

Uh oh!

deacon-mp Mar 16, 2026

Uh oh!

deacon-mp Mar 16, 2026

Uh oh!

deacon-mp Mar 16, 2026

Uh oh!

deacon-mp Mar 16, 2026

Uh oh!

deacon-mp Mar 16, 2026

Uh oh!

deacon-mp Mar 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		await select.selectOption({ index: 1 });
		await page.waitForTimeout(2000);

		await select.selectOption({ index: 1 });
		await page.waitForTimeout(1000);

		// Wait for the report API response to arrive
		await page.waitForResponse((resp) => resp.url().includes('/plugin/debrief/report'), { timeout: 10_000 }).catch(() => {});

Conversation

deacon-mp commented Mar 16, 2026

Summary

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

deacon-mp Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

deacon-mp Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

deacon-mp Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

deacon-mp Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

deacon-mp Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

deacon-mp Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

deacon-mp Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants