test: exhaustive Playwright E2E tests for compass UI#47
Open
test: exhaustive Playwright E2E tests for compass UI#47
Conversation
Add comprehensive E2E test suite covering: - Page load and plugin accessibility (8 tests) - Layer generation and download for all/specific adversaries (4 tests) - Adversary import from ATT&CK Navigator layer files (6 tests) - Error states: API failures, network timeouts, invalid files (6 tests) Tests run against a live Caldera instance via CALDERA_URL env var (default http://localhost:8888) with Chromium and Firefox projects.
There was a problem hiding this comment.
Pull request overview
Adds a comprehensive Playwright E2E test suite for the Compass plugin UI, covering page load, layer generation/download, adversary import from layer files, and error/edge-case handling. Includes shared auth fixture and Playwright config for Chromium + Firefox.
Changes:
- New Playwright config, package.json, and auth fixture for E2E test infrastructure
- 4 spec files testing Compass page load, layer generation, adversary import, and error states
- Support for
CALDERA_URL,CALDERA_USER,CALDERA_PASSenv vars
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/e2e/playwright.config.ts | Playwright config with Chromium/Firefox projects |
| tests/e2e/package.json | Package manifest with test scripts |
| tests/e2e/fixtures/caldera-auth.ts | Shared auth fixture for Caldera login |
| tests/e2e/specs/compass-page-load.spec.ts | Tests for plugin nav, heading, controls, iframe |
| tests/e2e/specs/compass-layer-generation.spec.ts | Tests for layer download and JSON schema validation |
| tests/e2e/specs/compass-adversary-import.spec.ts | Tests for file upload, modal, unmatched techniques |
| tests/e2e/specs/compass-error-states.spec.ts | Tests for API failures, invalid files, UI resilience |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1
to
+138
| import { test, expect } from "../fixtures/caldera-auth"; | ||
|
|
||
| test.describe("Compass plugin — layer generation", () => { | ||
| test.beforeEach(async ({ authenticatedPage: page }) => { | ||
| await page.goto("/#/plugins/compass"); | ||
| await page.waitForLoadState("networkidle"); | ||
| // Wait for the compass UI to be ready | ||
| const heading = page.locator("h2:has-text('Compass')"); | ||
| await expect(heading).toBeVisible({ timeout: 15_000 }); | ||
| }); | ||
|
|
||
| test("should generate a layer for all adversaries (default selection)", async ({ | ||
| authenticatedPage: page, | ||
| }) => { | ||
| // Ensure default "Select an Adversary (All)" is selected | ||
| const select = page.locator( | ||
| 'select#layer-selection-adversary, select:has(option:has-text("Select an Adversary"))' | ||
| ); | ||
| await expect(select.first()).toBeVisible(); | ||
| const selectedValue = await select.first().inputValue(); | ||
| expect(selectedValue).toBe(""); | ||
|
|
||
| // Click Generate Layer and expect a download | ||
| const downloadPromise = page.waitForEvent("download", { timeout: 30_000 }); | ||
| const generateBtn = page.locator( | ||
| 'button#generateLayer, button:has-text("Generate Layer")' | ||
| ); | ||
| await generateBtn.first().click(); | ||
| const download = await downloadPromise; | ||
|
|
||
| expect(download.suggestedFilename()).toBe("layer.json"); | ||
| }); | ||
|
|
||
| test("downloaded layer file should contain valid JSON with ATT&CK layer schema fields", async ({ | ||
| authenticatedPage: page, | ||
| }) => { | ||
| const downloadPromise = page.waitForEvent("download", { timeout: 30_000 }); | ||
| const generateBtn = page.locator( | ||
| 'button#generateLayer, button:has-text("Generate Layer")' | ||
| ); | ||
| await generateBtn.first().click(); | ||
| const download = await downloadPromise; | ||
|
|
||
| // Read and parse the downloaded file | ||
| const path = await download.path(); | ||
| expect(path).toBeTruthy(); | ||
|
|
||
| const fs = await import("fs"); | ||
| const content = fs.readFileSync(path!, "utf-8"); | ||
| const layer = JSON.parse(content); | ||
|
|
||
| // ATT&CK Navigator layer schema fields | ||
| expect(layer).toHaveProperty("name"); | ||
| expect(layer).toHaveProperty("versions"); | ||
| expect(layer).toHaveProperty("domain"); | ||
| expect(layer).toHaveProperty("techniques"); | ||
| expect(Array.isArray(layer.techniques)).toBe(true); | ||
| }); | ||
|
|
||
| test("should populate the adversary dropdown with available adversaries", async ({ | ||
| authenticatedPage: page, | ||
| }) => { | ||
| const select = page.locator( | ||
| 'select#layer-selection-adversary, select:has(option:has-text("Select an Adversary"))' | ||
| ); | ||
| await expect(select.first()).toBeVisible(); | ||
|
|
||
| // Count options — at minimum the default "All" option | ||
| const options = select.first().locator("option"); | ||
| const count = await options.count(); | ||
| expect(count).toBeGreaterThanOrEqual(1); | ||
| }); | ||
|
|
||
| test("should generate a layer for a specific adversary when selected", async ({ | ||
| authenticatedPage: page, | ||
| }) => { | ||
| const select = page.locator( | ||
| 'select#layer-selection-adversary, select:has(option:has-text("Select an Adversary"))' | ||
| ); | ||
| await expect(select.first()).toBeVisible(); | ||
|
|
||
| // Get all options | ||
| const options = select.first().locator("option"); | ||
| const count = await options.count(); | ||
|
|
||
| // Skip if only the default option exists (no adversaries loaded) | ||
| test.skip(count < 2, "No adversaries available to select"); | ||
|
|
||
| // Select the first non-default adversary | ||
| const secondOption = options.nth(1); | ||
| const value = await secondOption.getAttribute("value"); | ||
| await select.first().selectOption(value!); | ||
|
|
||
| // Generate layer | ||
| const downloadPromise = page.waitForEvent("download", { timeout: 30_000 }); | ||
| const generateBtn = page.locator( | ||
| 'button#generateLayer, button:has-text("Generate Layer")' | ||
| ); | ||
| await generateBtn.first().click(); | ||
| const download = await downloadPromise; | ||
|
|
||
| expect(download.suggestedFilename()).toBe("layer.json"); | ||
| }); | ||
|
|
||
| test("layer generated for specific adversary should contain technique entries", async ({ | ||
| authenticatedPage: page, | ||
| }) => { | ||
| const select = page.locator( | ||
| 'select#layer-selection-adversary, select:has(option:has-text("Select an Adversary"))' | ||
| ); | ||
| const options = select.first().locator("option"); | ||
| const count = await options.count(); | ||
|
|
||
| test.skip(count < 2, "No adversaries available to select"); | ||
|
|
||
| const secondOption = options.nth(1); | ||
| const value = await secondOption.getAttribute("value"); | ||
| await select.first().selectOption(value!); | ||
|
|
||
| const downloadPromise = page.waitForEvent("download", { timeout: 30_000 }); | ||
| const generateBtn = page.locator( | ||
| 'button#generateLayer, button:has-text("Generate Layer")' | ||
| ); | ||
| await generateBtn.first().click(); | ||
| const download = await downloadPromise; | ||
|
|
||
| const path = await download.path(); | ||
| const fs = await import("fs"); | ||
| const content = fs.readFileSync(path!, "utf-8"); | ||
| const layer = JSON.parse(content); | ||
|
|
||
| expect(layer.techniques.length).toBeGreaterThan(0); | ||
| // Each technique should have a techniqueID | ||
| for (const tech of layer.techniques) { | ||
| expect(tech).toHaveProperty("techniqueID"); | ||
| } | ||
| }); | ||
| }); |
Comment on lines
+40
to
+94
| await page.waitForTimeout(2_000); | ||
| expect(downloadTriggered).toBe(false); | ||
| }); | ||
|
|
||
| test("should handle adversary upload API failure gracefully", async ({ | ||
| authenticatedPage: page, | ||
| }) => { | ||
| // Intercept the adversary upload API and force a 500 error | ||
| await page.route("**/plugin/compass/adversary", (route) => | ||
| route.fulfill({ | ||
| status: 500, | ||
| contentType: "application/json", | ||
| body: JSON.stringify({ error: "Internal Server Error" }), | ||
| }) | ||
| ); | ||
|
|
||
| // Create a minimal layer file to upload | ||
| const tmpDir = os.tmpdir(); | ||
| const filePath = path.join(tmpDir, "e2e-error-test-layer.json"); | ||
| fs.writeFileSync( | ||
| filePath, | ||
| JSON.stringify({ | ||
| name: "Error Test", | ||
| techniques: [{ techniqueID: "T1059", tactic: "execution" }], | ||
| }) | ||
| ); | ||
|
|
||
| const fileInput = page.locator('input#generateAdversary[type="file"]'); | ||
| await fileInput.setInputFiles(filePath); | ||
|
|
||
| // The page should remain functional (no unhandled crash) | ||
| await page.waitForTimeout(3_000); | ||
| const heading = page.locator("h2:has-text('Compass')"); | ||
| await expect(heading).toBeVisible(); | ||
|
|
||
| // No modal should appear | ||
| const modal = page.locator(".modal.is-active"); | ||
| await expect(modal).not.toBeVisible(); | ||
|
|
||
| fs.unlinkSync(filePath); | ||
| }); | ||
|
|
||
| test("should handle uploading an invalid (non-JSON) file gracefully", async ({ | ||
| authenticatedPage: page, | ||
| }) => { | ||
| // Create a non-JSON file | ||
| const tmpDir = os.tmpdir(); | ||
| const filePath = path.join(tmpDir, "e2e-invalid-file.txt"); | ||
| fs.writeFileSync(filePath, "This is not valid JSON layer data"); | ||
|
|
||
| const fileInput = page.locator('input#generateAdversary[type="file"]'); | ||
| await fileInput.setInputFiles(filePath); | ||
|
|
||
| // Page should remain stable | ||
| await page.waitForTimeout(3_000); |
Comment on lines
+36
to
+41
| let downloadTriggered = false; | ||
| page.on("download", () => { | ||
| downloadTriggered = true; | ||
| }); | ||
| await page.waitForTimeout(2_000); | ||
| expect(downloadTriggered).toBe(false); |
Comment on lines
+47
to
+49
| const generateAdversary = page.locator( | ||
| "text=Generate Adversary, text=Create Operation" | ||
| ); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CALDERA_URLenv var support (defaulthttp://localhost:8888)Test breakdown
compass-page-load.spec.tscompass-layer-generation.spec.tscompass-adversary-import.spec.tscompass-error-states.spec.tsTest plan
cd tests/e2e && npm install && npx playwright install --with-deps chromium firefoxCALDERA_URL=http://localhost:8888 npx playwright test