feat(redteam): add detection-to-action smoke benchmark

[Kerberosmansour (kerberosmansour)]

Summary

Adds the first accepted phase of the evidence-led red-team benchmark proposed in #3349:

• a strict, metadata-only benchmark contract under tests/redteam;
• 24 deterministic smoke scenarios covering six trap classes, with one canonical attack, evasion attack, hard-benign case, and near miss per class;
• a side-effect-free L2 mock behavioural harness that records attempted actions, execution outcomes, containment points, blocking controls, and audit evidence;
• a joint detection-to-action matrix that keeps detected/executed and undetected/contained visible;
• a separate, path-scoped CI job, additive to the existing agt red-team path.

The smoke gate fails on executed attacks, benign false blocks, benign detector flags, contract violations, evidence-level overclaims, raw/secret-bearing artifacts, or certification claims. It does not produce a single aggregate resilience score.

Issue and accepted scope

Refs #3349.

This implements the initial scope supported in the maintainer feedback:
#3349 (comment)

The public CLI, agent-sre live execution support, 240-scenario measurement suite, external 54k corpus, and 250-record live tier remain deferred for separate review.

Follow-up hardening

Cross-platform and evidence-integrity review of the reference implementation led to a small hardening pass in this PR:

• contract regressions now cover missing and malformed scenario files, live-URL hygiene, duplicate IDs, unknown trap classes, and trap/control mismatches;
• JSON Schema properties and required fields are checked bidirectionally against the four Python record-field constants, preventing contract drift;
• the benchmark README clarifies that detection_verdict is declared scenario metadata in this phase and real detector output is phase-two work;
• the Bash-backed Python test skips cleanly when Bash is unavailable and uses a working-directory-relative path suitable for Git Bash on Windows;
• the CI job is time-bounded and checkout credentials are not persisted.

The corresponding reference implementation fixes are in kerberosmansour/AGT-Embeddings-Experiment#42. They distinguish an observed model refusal from incomplete live trace capture, remove import-time fixture loading, and document/guard generated corpus tests.

This AGT contribution remains intentionally L2-only. It does not port the live or corpus tiers.

Validation

• python3 -m unittest discover -s tests/redteam -p 'test_benchmark_contract.py' -v — 19 tests passed, including the four schema-definition drift subtests, both executed-attack matrix subtests, and the fail-closed input/hygiene regressions.
• bash tests/redteam/benchmark/run-smoke.sh — 24 scenarios validated; matrix 6 detected/contained, 0 detected/executed, 6 undetected/contained, 0 undetected/executed; 12/12 benign cases allowed; hygiene passed.
• python3 -m ruff format --check ... — passed.
• python3 -m ruff check ... --select E,F,W --ignore E501 — passed.
• python3 -m py_compile ... — passed.
• actionlint .github/workflows/redteam-benchmark.yml — passed.
• shellcheck tests/redteam/benchmark/run-smoke.sh — passed.
• License-header, spelling, Markdown-link, JSON parsing, secret-pattern, and diff checks — passed.

The existing tests/redteam/test_asi.py remains unchanged. Its 84 tests collect successfully; executing them on current main is independently blocked by their existing reference to the absent templates/policies/starters/* fixture path.

Security and evidence boundary

• Python standard library only; no new dependency.
• No network, socket, subprocess, live model, external corpus, or target-filesystem execution.
• Generated artifacts contain metadata only and are rejected if they include raw payload fields, secret-shaped values, live URLs, or certification language.
• Every report is fixed to L2_mock_behavioural and certification_claim: false.
• executed means a benign dry-run completed inside the mock harness; every trace records side_effects: none.

Compatibility

This is additive test and CI coverage. It does not change packages, runtime governance behavior, existing playbooks, existing red-team tests, or the public CLI.

Prior art

The contract, evidence levels, and attempted-versus-executed trace semantics are adapted from the MIT-licensed reference implementation:

• https://github.com/kerberosmansour/AGT-Embeddings-Experiment
• AGT Red Team Benchmark — Overview and Usage Guide kerberosmansour/AGT-Embeddings-Experiment#41
• Complete AGT red-team benchmark runbook kerberosmansour/AGT-Embeddings-Experiment#40
• fix(redteam): fail closed on incomplete live traces kerberosmansour/AGT-Embeddings-Experiment#42

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

🤖 AI Agent: code-reviewer — Action items:

AI-generated review output. Treat it as untrusted analysis and verify before acting.

TL;DR: 0 blockers, 2 warnings. The PR introduces a well-structured and security-conscious red-team smoke benchmark, but there are minor areas for improvement.

#	Sev	Issue	Where
1	Warn	The CI job timeout of 5 minutes may be insufficient for future scaling.	.github/workflows/redteam-benchmark.yml
2	Warn	The _SECRET_RE regex may not cover all potential secret patterns.	tests/redteam/benchmark/benchmark.py

Action items:

1. None.

Warnings (fine as follow-up PRs):

#	Issue	Where
1	Consider increasing the CI job timeout to accommodate future scaling.	.github/workflows/redteam-benchmark.yml
2	Review and expand _SECRET_RE to ensure it captures a broader range of secret patterns.	tests/redteam/benchmark/benchmark.py

[github-actions]

🤖 AI Agent: security-scanner — View details

AI-generated review output. Treat it as untrusted analysis and verify before acting.

No security issues found.

[github-actions]

🤖 AI Agent: docs-sync-checker — Docs Sync

AI-generated review output. Treat it as untrusted analysis and verify before acting.

Docs Sync

Documentation is in sync.

[github-actions]

🤖 AI Agent: test-generator — `tests/redteam/benchmark/benchmark.py`

AI-generated review output. Treat it as untrusted analysis and verify before acting.

tests/redteam/benchmark/benchmark.py

• test_load_scenarios_invalid_json -- Validate behavior when the scenario file contains invalid JSON.
• test_load_scenarios_missing_file -- Ensure proper error handling when the scenario file is missing.
• test_ensure_hygiene_detects_secrets -- Verify that secret-shaped values are correctly identified and rejected.
• test_ensure_hygiene_detects_live_urls -- Confirm that live URLs are flagged as hygiene violations.
• test_ensure_hygiene_detects_certification_language -- Check that certification-related language is rejected as per hygiene rules.

[github-actions]

🤖 AI Agent: breaking-change-detector — API Compatibility

AI-generated review output. Treat it as untrusted analysis and verify before acting.

API Compatibility

No breaking changes detected.

[github-actions]

🟡 Contributor Check: MEDIUM

Check	Result
Profile	MEDIUM
Credential	MEDIUM
Overall	MEDIUM

Automated check by AGT Contributor Check.

[github-actions]

PR Review Summary

Check	Status	Details
🔍 Code Review	⚠️ Missing	No current-run comment
🛡️ Security Scan	⚠️ Missing	No current-run comment
🔄 Breaking Changes	⚠️ Missing	No current-run comment
📝 Docs Sync	⚠️ Missing	No current-run comment
🧪 Test Coverage	⚠️ Missing	No current-run comment

Verdict: ⚠️ AI review incomplete; ready for human review

AI review comments are untrusted advisory output. The summary reports workflow-generated completion status only, not model-authored pass/fail claims.

[MohammadHaroonAbuomar]

• contract.schema.json <-> benchmark.py drift test still missing: the existing schema test only checks $defs names, additionalProperties:false, and the certification const; nothing cross-checks the schema's required/properties against SCENARIO_FIELDS/TRACE_FIELDS/RESULT_FIELDS/REPORT_FIELDS, so the two sources of truth can drift silently.
• one-line phase-2 note in tests/redteam/benchmark/README.md that detection_verdict is declared scenario metadata (live detection = phase 2); a real detector catching an evasion would currently be a contract violation.
• (process) MAINTAINER: approve the fork workflow runs; the PR's own redteam-benchmark.yml has never executed in CI (local equivalent passes: 18 tests + 4 subtests incl e2e smoke).

[Kerberosmansour (kerberosmansour)]

Addressed in dd01cf98:

• Added test_schema_fields_match_python_contract_constants, which checks both JSON Schema properties and required against SCENARIO_FIELDS, TRACE_FIELDS, RESULT_FIELDS, and REPORT_FIELDS. Equality is checked in both directions for all four record definitions.
• Added the phase-two note to tests/redteam/benchmark/README.md: detection_verdict is declared scenario metadata in this phase, while real detector output—including catches on evasion rows—is deferred.

Local validation now passes with 19 contract tests, including the four schema-definition subtests, plus the end-to-end smoke run, Ruff formatting/lint, Python compilation, and diff checks.

The fork workflow approval remains a maintainer-only process step, as noted in the review.

[Kerberosmansour (kerberosmansour)]

MohammadHaroonAbuomar can you review the changes please?

[MohammadHaroonAbuomar]

Minor:

• the set hash is order-sensitive (json.dumps of the list); sort by id before hashing or rename.
• simulate_trace containment is metadata-derived (documented phase-2 detector); workflow pins should adopt #3365's baseline after it merges.

[MohammadHaroonAbuomar]

Minor:

• the set hash is order-sensitive (json.dumps of the list); sort by id before hashing or rename.
• simulate_trace containment is metadata-derived (documented phase-2 detector); workflow pins should adopt #3365's baseline after it merges.

Commits dd01cf9 (drift test, README note) and 96604da (hash recompute, scenario_id cross-check, id-sorted hash) address each request. At head 29f963d I ran the contract tests (21 pass), run-smoke.sh (OK), and probes: the drift test fails when a field is added to either side; a forged report with 24 copies of one benign result and a bogus hash exits 1 through the hygiene CLI, as does one with a genuine hash but duplicated ids; reversed and shuffled fixtures hash identically.

[MohammadHaroonAbuomar]

Verified at 29f963d: the schema-versus-constants drift test fails when a field is added to either side, a forged report with a bogus scenario_set_hash or foreign scenario_ids exits 1 through the hygiene CLI, the set hash is id-sorted, and the README carries the phase-two note. 21 contract tests pass and run-smoke.sh is green.

[MohammadHaroonAbuomar]

The only CI run on this head is from mid-August and its required checks have expired, so the merge gate cannot see them. Please rebase onto current main and force-push to trigger a fresh run. The content is already reviewed and approved; nothing else is needed.

[MohammadHaroonAbuomar]

The branch is behind main again after today's merges, and the update-branch action no longer has permission on your fork (maintainer edits look switched off), so please rebase onto current main and force-push. The content is approved; nothing else is needed.