Add exhaustive pytest test suite (203 tests) by deacon-mp · Pull Request #78 · mitre/debrief

deacon-mp · 2026-03-16T14:02:31Z

Summary

Adds 203 pytest tests covering all Python modules in the debrief plugin
Creates .caldera-shim/ with lightweight Caldera framework stubs enabling isolated testing without a full Caldera installation
Adds pytest.ini and tox.ini for test configuration

Modules Covered

Test File	Module(s) Tested	Test Count
`test_debrief_svc.py`	`app/debrief_svc.py` (DebriefService)	30
`test_debrief_gui.py`	`app/debrief_gui.py` (DebriefGui, template markers)	22
`test_hook.py`	`hook.py` (enable, cache init, metadata)	7
`test_story.py`	`app/objects/c_story.py` (Story)	14
`test_base_report_section.py`	`app/utility/base_report_section.py`	13
`test_sections.py`	All 11 debrief-sections modules	46
`test_attack_mapper_extended.py`	`attack_mapper.py` (Attack18Map, index_bundle, fetch)	46
`conftest.py`	Shared fixtures (operations, agents, links, facts)	-
Existing `test_attack_mapper.py`	(unchanged)	17
Existing `test_detections_table.py`	(unchanged)	8

Coverage Areas

Report generation: all section modules (agents, graphs, tables, TTPs, detections)
Attack mapping: Attack18Map API, STIX bundle indexing, cache fetch, utility functions
D3 graph builders: steps, attackpath, fact, tactic, technique
Static helpers: TTP generation, timestamp formatting, filename sanitization
Hook lifecycle: route registration, config application, ATT&CK cache warmup

Test plan

All 203 tests pass locally (pytest tests/ -v -- 203 passed, 0 failed)
CI pipeline validates on push

- tests/conftest.py: shared fixtures with mock builders for operations, agents, links, facts - tests/test_debrief_svc.py: DebriefService (generate_ttps, d3 graph builders, static helpers) - tests/test_debrief_gui.py: DebriefGui (sanitize, template markers, runtime agents, pretty name) - tests/test_hook.py: plugin hook (enable routes, cache init, metadata) - tests/test_story.py: Story object (append, page break, table objects, header logo) - tests/test_base_report_section.py: BaseReportSection (status names, table gen, grouping) - tests/test_sections.py: all 11 debrief-sections modules (agents, graphs, tables, TTPs) - tests/test_attack_mapper_extended.py: Attack18Map, index_bundle, fetch_and_cache, utilities - .caldera-shim/: lightweight Caldera framework stubs for isolated testing - pytest.ini, tox.ini: test configuration

Copilot

Pull request overview

Adds an isolated pytest-based test harness for the Debrief plugin, including Caldera stubs, to enable running a broad unit test suite without a full Caldera install.

Changes:

Added a large pytest suite covering Debrief service/gui, hook lifecycle, report sections, and ATT&CK mapping.
Introduced .caldera-shim/ Caldera framework stubs to make tests runnable in isolation.
Added pytest.ini and tox.ini to standardize test execution/config.

Reviewed changes

Copilot reviewed 21 out of 27 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tox.ini	Adds tox env for running pytest with required deps
pytest.ini	Configures pytest discovery, asyncio mode, and shim pythonpath
tests/conftest.py	Adds fixtures/builders to avoid depending on Caldera runtime objects
tests/test_debrief_svc.py	Adds unit tests for DebriefService behaviors and graph builders
tests/test_debrief_gui.py	Adds unit tests for DebriefGui helpers and template switching
tests/test_hook.py	Adds unit tests for plugin enable + ATT&CK cache warmup
tests/test_sections.py	Adds tests for all report-section modules
tests/test_attack_mapper_extended.py	Expands test coverage for ATT&CK mapping, cache load/fetch
tests/test_story.py	Adds tests for Story helper class
tests/test_base_report_section.py	Adds tests for BaseReportSection utilities
.caldera-shim/plugins/debrief	Adds a link/redirect for importing the plugin under the shim path
.caldera-shim/app/**	Adds minimal stub modules/classes required for imports in plugin code

Comments suppressed due to low confidence (1)

.caldera-shim/plugins/debrief:1

This appears to be a committed symlink (or link file) pointing to an absolute, machine-local path under /tmp. That will break in CI and for any other developer environment. Prefer a repo-relative symlink target (e.g., pointing to the plugin path within the repo) or replace this with a small shim package/module that adjusts sys.path at test time.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

deacon-mp · 2026-03-16T22:03:10Z

tests/test_sections.py

+agents_mod = importlib.import_module('plugins.debrief.app.debrief-sections.agents')
+attackpath_mod = importlib.import_module('plugins.debrief.app.debrief-sections.attackpath_graph')
+fact_graph_mod = importlib.import_module('plugins.debrief.app.debrief-sections.fact_graph')
+facts_table_mod = importlib.import_module('plugins.debrief.app.debrief-sections.facts_table')
+main_summary_mod = importlib.import_module('plugins.debrief.app.debrief-sections.main_summary')
+statistics_mod = importlib.import_module('plugins.debrief.app.debrief-sections.statistics')
+steps_graph_mod = importlib.import_module('plugins.debrief.app.debrief-sections.steps_graph')
+steps_table_mod = importlib.import_module('plugins.debrief.app.debrief-sections.steps_table')
+tactic_graph_mod = importlib.import_module('plugins.debrief.app.debrief-sections.tactic_graph')
+tactic_technique_mod = importlib.import_module('plugins.debrief.app.debrief-sections.tactic_technique_table')
+technique_graph_mod = importlib.import_module('plugins.debrief.app.debrief-sections.technique_graph')


Intentional — importlib.import_module() is used throughout the debrief codebase to import from the debrief-sections directory. This matches the existing pattern in debrief_gui.py.

deacon-mp · 2026-03-16T22:03:12Z

tests/test_story.py

+
+    def test_header_logo_path_default(self):
+        s = Story()
+        assert Story._header_logo_path is not None or Story._header_logo_path is None


Fixed — replaced with assert Story._header_logo_path is None after resetting to known state.

deacon-mp · 2026-03-16T22:03:14Z

tests/test_story.py

+        with pytest.raises(AttributeError):
+            s.get_description('test')


Fixed — replaced with explicit hasattr checks that verify the delegation pattern without relying on exception type.

deacon-mp · 2026-03-16T22:03:16Z

tests/test_debrief_gui.py

+        svg_content = b'<svg></svg>'
+        encoded = base64.b64encode(svg_content).decode()
+        save_dir = str(tmp_path) + '/'
+
+        with patch('builtins.open', MagicMock()) as mock_open:
+            # Monkey-patch save location
+            svgs = {'test_graph': encoded}
+            with patch('plugins.debrief.app.debrief_gui.DebriefGui._save_svgs') as _:
+                # Test the static method directly
+                import base64 as b64mod
+                for filename, svg_bytes in svgs.items():
+                    decoded = b64mod.b64decode(svg_bytes)
+                    assert decoded == svg_content
+


Fixed — test now calls the real _save_svgs method and asserts mock_fh.write was called with decoded content.

deacon-mp · 2026-03-16T22:03:18Z

tox.ini

+    pytest
+    pytest-asyncio
+    aiohttp
+    reportlab
+    svglib
+    lxml


Fixed — tox deps now have version bounds (e.g., pytest>=7.0,<9.0, pytest-asyncio>=0.21,<1.0).

- Replace hardcoded /tmp/debrief-pytest symlink with relative path ../../ - Fix tautological assertion in test_header_logo_path_default (was always true) - Strengthen test_strategy_det_id_normalization to verify DET0012 exists - Rewrite test_save_svgs to actually call _save_svgs and verify file write - Tighten test_value_truncation to assert truncation directly - Tighten test_generate_ttp_detection_info_with_links to assert href - Fix file handle leak in c_story.py adjust_icon_svgs - Add viewBox null check in c_story.py - Fix stale op_id bug in debrief_svc.py build_steps_d3

- Pin tox dependency versions to avoid breakage from major version bumps - Replace brittle AttributeError test with explicit hasattr checks for get_description delegation pattern

Copilot

Pull request overview

Adds an isolated pytest-based test harness for the Debrief plugin (including Caldera stubs) and introduces test runner configuration via pytest.ini/tox.ini.

Changes:

Introduces .caldera-shim/ stubs to run unit tests without a full Caldera install
Adds a large pytest suite covering Debrief service/gui/sections/hook and ATT&CK mapper logic
Adds pytest.ini and tox.ini for consistent local and tox execution

Reviewed changes

Copilot reviewed 21 out of 27 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tox.ini	Adds tox environment to run pytest with required deps
pytest.ini	Configures pytest discovery, asyncio mode, and shim pythonpath
.caldera-shim/plugins/debrief	Adds a shim mapping for the debrief plugin import path (currently via symlink target)
.caldera-shim/app/utility/base_world.py	Stubs BaseWorld APIs used by the plugin
.caldera-shim/app/utility/base_service.py	Stubs BaseService config access
.caldera-shim/app/utility/base_object.py	Stubs BaseObject constants
.caldera-shim/app/service/auth_svc.py	Stubs auth decorators used by GUI routes
.caldera-shim/app/objects/**	Stubs Caldera object models used by the plugin
tests/conftest.py	Provides lightweight fixtures for operations/agents/links/facts
tests/test_debrief_svc.py	Adds unit tests for DebriefService (TTPS + D3 builders + helpers)
tests/test_debrief_gui.py	Adds unit tests for DebriefGui utilities and flowables
tests/test_hook.py	Adds unit tests for plugin hook enable/cache warmup
tests/test_story.py	Adds unit tests for Story helper object
tests/test_base_report_section.py	Adds unit tests for BaseReportSection helpers
tests/test_sections.py	Adds unit tests for all debrief report section modules
tests/test_attack_mapper_extended.py	Adds extended unit tests for ATT&CK v18 mapping/indexing/caching

Comments suppressed due to low confidence (1)

.caldera-shim/plugins/debrief:1

This appears to be a committed symlink whose target points to an absolute, machine-specific path (/tmp/debrief-pytest). That will break checkouts in CI/other dev machines and makes the test harness non-reproducible. Replace this with a repo-relative symlink target (preferred) or remove the symlink and instead add a small shim package/module under .caldera-shim/plugins/debrief/ that imports/extends the real plugin code via relative paths.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/test_story.py

+    def test_header_logo_path_default(self):
+        s = Story()
+        assert Story._header_logo_path is not None or Story._header_logo_path is None
+
+


tests/test_story.py

+        # _descriptions doesn't exist as a method; this tests the proxy
+        with pytest.raises(AttributeError):
+            s.get_description('test')


tests/test_debrief_gui.py

+        save_dir = str(tmp_path) + '/'
+
+        with patch('builtins.open', MagicMock()) as mock_open:
+            # Monkey-patch save location
+            svgs = {'test_graph': encoded}
+            with patch('plugins.debrief.app.debrief_gui.DebriefGui._save_svgs') as _:
+                # Test the static method directly
+                import base64 as b64mod
+                for filename, svg_bytes in svgs.items():
+                    decoded = b64mod.b64decode(svg_bytes)
+                    assert decoded == svg_content
+


tests/test_hook.py

+
+    def test_access(self):
+        # Access should be RED
+        assert hook.access is not None


tests/test_hook.py

+            # Check routes registered
+            assert mock_router.add_route.call_count >= 7
+            assert mock_router.add_static.call_count >= 2
+


Copilot

Pull request overview

Adds an isolated pytest-based test harness for the debrief plugin (including Caldera shims) and fixes a couple of production-code issues discovered during test authoring.

Changes:

Added .caldera-shim/ Caldera stubs + pytest.ini/tox.ini to run tests without a full Caldera install
Added a large pytest suite covering debrief service/gui/sections/attack mapping/hook behavior
Fixed build_steps_d3 to use operation.id (not the last loop’s op_id) and hardened SVG viewbox handling/writing

Reviewed changes

Copilot reviewed 23 out of 29 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tox.ini	Adds tox env + dependencies to run pytest suite
pytest.ini	Configures pytest discovery + asyncio + shim pythonpath
app/debrief_svc.py	Fixes D3 steps graph node/link IDs to use `operation.id`
app/objects/c_story.py	Guards missing `viewBox` and uses context-managed file write
tests/conftest.py	Adds shared fixtures/builders for isolated tests
tests/test_debrief_svc.py	Adds DebriefService unit tests (D3 builders, helpers, TTP generation)
tests/test_debrief_gui.py	Adds DebriefGui unit tests (markers, filename sanitize, svg save/cleanup, helpers)
tests/test_hook.py	Adds hook enable/cache init metadata tests
tests/test_story.py	Adds Story unit tests
tests/test_base_report_section.py	Adds BaseReportSection unit tests
tests/test_sections.py	Adds tests for all debrief section modules
tests/test_attack_mapper_extended.py	Adds extended tests for ATT&CK mapping/cache functions
.caldera-shim/plugins/debrief	Maps imports to the plugin root for `plugins.debrief.*` imports
.caldera-shim/app/utility/base_world.py	Stub for BaseWorld used by plugin code/tests
.caldera-shim/app/utility/base_service.py	Stub for BaseService
.caldera-shim/app/utility/base_object.py	Stub for BaseObject constants
.caldera-shim/app/service/auth_svc.py	Stub decorators/authorization helpers
.caldera-shim/app/objects/c_agent.py	Stub Agent model
.caldera-shim/app/objects/c_operation.py	Stub Operation model
.caldera-shim/app/objects/c_adversary.py	Stub Adversary model
.caldera-shim/app/objects/c_ability.py	Stub Ability model
.caldera-shim/app/objects/secondclass/c_link.py	Stub Link model + decode helper
.caldera-shim/app/objects/secondclass/c_executor.py	Stub Executor model

Comments suppressed due to low confidence (1)

.caldera-shim/plugins/debrief:1

This looks like a symlink target (intended to map plugins.debrief to the repo root). If it’s not actually committed as a symlink (mode 120000), Python imports will break because this is just a plain file without a .py extension or package directory. To make this portable (especially on Windows/CI where symlinks can be restricted), consider replacing this with a real package directory .caldera-shim/plugins/debrief/__init__.py that adjusts __path__/sys.path to point at the plugin root, or restructure the shim to avoid requiring symlinks.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/test_sections.py

+agents_mod = importlib.import_module('plugins.debrief.app.debrief-sections.agents')
+attackpath_mod = importlib.import_module('plugins.debrief.app.debrief-sections.attackpath_graph')
+fact_graph_mod = importlib.import_module('plugins.debrief.app.debrief-sections.fact_graph')
+facts_table_mod = importlib.import_module('plugins.debrief.app.debrief-sections.facts_table')
+main_summary_mod = importlib.import_module('plugins.debrief.app.debrief-sections.main_summary')
+statistics_mod = importlib.import_module('plugins.debrief.app.debrief-sections.statistics')
+steps_graph_mod = importlib.import_module('plugins.debrief.app.debrief-sections.steps_graph')
+steps_table_mod = importlib.import_module('plugins.debrief.app.debrief-sections.steps_table')
+tactic_graph_mod = importlib.import_module('plugins.debrief.app.debrief-sections.tactic_graph')
+tactic_technique_mod = importlib.import_module('plugins.debrief.app.debrief-sections.tactic_technique_table')
+technique_graph_mod = importlib.import_module('plugins.debrief.app.debrief-sections.technique_graph')


tests/test_story.py

+        """get_description() calls self._descriptions() which is not defined
+        on the base Story class. Verify the delegation attempt occurs."""
+        s = Story()
+        assert hasattr(s, 'get_description')
+        # _descriptions is not implemented on Story — subclasses would provide it
+        assert not hasattr(s, '_descriptions')


tests/test_hook.py

+
+    def test_access(self):
+        # Access should be RED
+        assert hook.access is not None


app/objects/c_story.py

+            viewbox = [int(float(val)) for val in viewbox_attr.split()]
            aspect = viewbox[2] / viewbox[3]
            icon_svg.set('width', str(round(float(icon_svg.get('height')) * aspect)))


deacon-mp requested a review from Copilot March 16, 2026 14:02

Copilot AI reviewed Mar 16, 2026

View reviewed changes

deacon-mp mentioned this pull request Mar 16, 2026

Add exhaustive pytest test suite with fixes deacon-mp/debrief#2

Open

Address Copilot review: pin tox deps, fix get_description test

02775d5

- Pin tox dependency versions to avoid breakage from major version bumps - Replace brittle AttributeError test with explicit hasattr checks for get_description delegation pattern

deacon-mp closed this Mar 16, 2026

deacon-mp reopened this Mar 16, 2026

deacon-mp requested a review from Copilot March 16, 2026 22:13

Copilot AI reviewed Mar 16, 2026

View reviewed changes

deacon-mp requested a review from Copilot March 16, 2026 22:21

Copilot AI reviewed Mar 16, 2026

View reviewed changes

deacon-mp merged commit 31715c4 into master Mar 16, 2026
4 of 5 checks passed

deacon-mp deleted the test/exhaustive-pytest-coverage branch March 16, 2026 23:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add exhaustive pytest test suite (203 tests)#78

Add exhaustive pytest test suite (203 tests)#78
deacon-mp merged 3 commits intomasterfrom
test/exhaustive-pytest-coverage

deacon-mp commented Mar 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

deacon-mp Mar 16, 2026

Uh oh!

deacon-mp Mar 16, 2026

Uh oh!

deacon-mp Mar 16, 2026

Uh oh!

deacon-mp Mar 16, 2026

Uh oh!

deacon-mp Mar 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		with pytest.raises(AttributeError):
		s.get_description('test')

Conversation

deacon-mp commented Mar 16, 2026

Summary

Modules Covered

Coverage Areas

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

deacon-mp Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

deacon-mp Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

deacon-mp Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

deacon-mp Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

deacon-mp Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants