FEAT: Attack analytics SDK

[Roman Lutz (romanlutz)]

Description

Part 3 of the CoPyRIT attack-results analytics roadmap, following the merged contracts (#2742) and storage (#2792). This makes saved-result analytics available to Python callers without introducing analytics REST endpoints or the GUI explorer.

• Add async AttackResultAnalytics.query_async, results_async, and facets_async, plus close_async and an async context manager. Reports include a coherent initial lightweight result page; subsequent pages and facets do not recalculate reports or hydrate conversations, scores, or media.
• Share one loop-bound controller per memory object, with separate report and quick-query capacity. Defaults are five report slots, two page/facet slots, ten queued calls per lane, a one-second queue wait, and five-/one-second execution budgets. Cancellation and response expiry retain capacity until the actual operation/session cleanup finishes. Shutdown drains before another controller can replace it. These are overload safeguards, not latency guarantees.
• Aggregate reader-approved SQLite profiles with Unicode str.lower semantics, deduplicated memberships, typed missing/no-converter values, stable labels, and parity with the complete SQL fallback. Overflow falls back without sampling or partial totals. Exact appended drill-down predicates preserve existing filters and the 16-predicate/500-value budget.
• Consolidate attack and scenario calculations in compute_outcome_statistics / combine_outcome_statistics. Both expose decided-only success_rate_decided and all-outcome success_rate_all; success_rate remains compatible. Raw analytics still counts every saved result ID, while scenario statistics still selects the latest attempt per execution unit. Historical retry/error counts remain separate from selected outcome counts.
• Reject contradictory rich statistics and revalidate mutable inputs before combining them. Add include_outcome_statistics=True to the maintained analyze_results and compute_technique_stats_async APIs while retaining the original six-field AttackStats defaults. Scenario progress and JSON output carry the shared statistics; existing displayed percentages and deprecation schedules remain unchanged.

The shared-calculation follow-up builds on #2820 and preserves the explicit result roles from #2997 without silently excluding any role. Existing lazy exports and import boundaries are retained. Requests are snapshotted and revalidated before admission. There is no persistent result cache or in-flight coalescing.

No schema/migration changes, shared/live database access, History selection changes, new endpoints, frontend changes, or backend lifecycle wiring. REST/History integration, the HTTP performance gate, and the explorer remain roadmap parts 4 and 5. The profile caps bound transferred profiles and SDK work, not SQL scans or peak network bytes.

Tests and Documentation

Added SDK, execution-lifetime, SQL/profile parity, shared-outcome, consistency/serialization, and compatibility coverage. Regressions include all outcomes and empty denominators, exact drill-down boundaries, Unicode and legacy converter metadata, profile caps/overflow, request snapshots, independent native async sessions, task-start failures, Python 3.11 admission-cancellation races, and deterministic shutdown. Scenario/backend/JSON parity and maintained-versus-deprecated API behavior are also covered.

Updated the Python analytics guide, framework responsibilities, and documentation navigation. The guide explains both denominator policies, resource ownership, compatibility, and the limitations of the bounded profile path.

Current head, Windows / Python 3.14.4:

uv run --no-sync pytest -q -n 4 --dist=loadfile tests\unit\analytics tests\unit\models\test_analytics.py tests\unit\models\test_scenario_progress.py tests\unit\models\test_import_boundary.py tests\unit\output\scenario_result tests\unit\output\test_derivation.py --timeout=90

Result: 620 passed.

• uv run --no-sync ty check pyrit: passed.
• uv run --no-sync python -m build_scripts.validate_docs: passed, including 165 navigation references and the orphaned-file check.
• git --no-pager diff --check: passed.
• All applicable normal commit hooks passed, including Ruff formatting/lint, async naming, documentation, and typing. No hooks were skipped.

Additional cross-version and broader compatibility evidence

Python 3.11.15, using a temporary uv-managed environment inside this worktree:

$env:UV_PROJECT_ENVIRONMENT = '.venv-py311'
uv run --no-sync --python 3.11 pytest -q tests\unit\models\test_analytics.py tests\unit\analytics\test_outcome_statistics.py tests\unit\analytics\test_scenario_statistics.py tests\unit\analytics\test_result_analysis.py tests\unit\analytics\test_technique_analysis.py tests\unit\analytics\test_attack_result_analytics.py tests\unit\analytics\test_scenario_statistics_parity.py tests\unit\output\scenario_result\test_json.py --timeout=90

Result: 369 passed, with an existing third-party confusables invalid-escape warning. The temporary environment was removed afterward.

The broader Python 3.14.4 compatibility run included scenario services/routes, adaptive selectors, output formats, and lazy imports:

uv run --no-sync pytest -q -n 4 --dist=loadfile tests\unit\analytics tests\unit\models\test_analytics.py tests\unit\models\test_scenario_progress.py tests\unit\models\test_scenario_result.py tests\unit\models\test_import_boundary.py tests\unit\output\test_derivation.py tests\unit\output\scenario_result tests\unit\backend\test_scenario_progress_read_model.py tests\unit\backend\test_scenario_run_service.py tests\unit\backend\test_scenario_run_routes.py tests\unit\common\test_lazy_package_imports.py tests\unit\scenario\scenarios\adaptive\test_epsilon_greedy.py --timeout=90

Result: 1,062 passed; one scenario alias-import subprocess hit its existing 30-second timeout. That unchanged test and the new technique lazy-export test both passed in isolation:

uv run --no-sync pytest -q tests\unit\common\test_lazy_package_imports.py::test_scenario_short_imports_preserve_canonical_identity 'tests\unit\common\test_lazy_package_imports.py::test_lazy_import_spot_check[pyrit.analytics0]' --timeout=90

Result: 2 passed in 9.31 seconds. No test timeout or import guard was weakened.

Full-repository, live Azure SQL, and HTTP/100,000-row performance validation were not run. SQLite coverage uses isolated test databases. Weighted SDK profile coverage is not a production latency claim.

JupyText was not run: this changeset adds Markdown documentation, not paired Python/notebook examples.