FEAT: Improve adversarial benchmark dataset and scoring by hannahwestra25 · Pull Request #2551 · microsoft/PyRIT

hannahwestra25 · 2026-09-03T17:16:28Z

Description

Improves the adversarial benchmark end to end: a balanced 120-objective dataset (adversarial_benchmark_v1), better objective scoring, and Role Play / Crescendo / TAP technique defaults.

The ADO pipeline gains quick and full profiles and reuses compatible results from a same-branch SQLite artifact, so a re-run only pays for objectives it has not already measured.

Attacks that never reached the objective target are no longer recorded as measured failures. When the adversarial model produces no attacker turn, the objective target is never probed, so there is no verdict to record. Those attacks now return AttackOutcome.UNDETERMINED and carry a typed marker (pyrit/executor/attack/core/attack_preparation.py) shared by RedTeamingAttack, PromptSendingAttack, and generate_simulated_conversation_async. That keeps them out of the result cache and out of resume-completion.

The typed marker exists so the producers and consumers of that signal agree on one contract without depending on each other's modules, and so the failure reason is guaranteed non-empty — previously the consumer read a magic metadata key imported from red_teaming and could end up with a None reason that silently fell through.

Provider blocks vs. adversarial-model refusals

An SDK-reported refusal and a provider content filter both surface as response_error == "blocked", so they were previously indistinguishable. AdversarialChatRefusedException now subclasses AdversarialChatResponseBlockedException, and _raise_for_adversarial_error branches on the message piece's structured refusal. The two are recorded as distinct failure kinds (ADVERSARIAL_CHAT_BLOCKED vs ADVERSARIAL_CHAT_REFUSED).

Both stay UNDETERMINED: in either case nothing was sent to the objective target, so scoring them as a defensive win would put an unsent attack in the ASR denominator. They are recorded separately so the distinction is auditable. A plain-text refusal from the adversarial model is unaffected — it carries no error, is still sent to the target, and remains a genuine measured FAILURE.

Notable behavior change

Scenario._get_completed_objective_hashes_for_attack now leaves a row pending only when it is an ERROR or carries a preparation-failure marker. This applies to all scenarios, not just this benchmark.

UNDETERMINED alone is deliberately not enough to re-run a row. A row that reached the objective target and ended UNDETERMINED — for example because no objective scorer was configured, or the scorer abstained — recorded the best verdict its configuration allows. Re-running it would re-send the objective on every resume without ever converging. Rows that reached a real verdict are unaffected.

Cache reuse break for TAP consumers

TreeOfAttacksWithPruningAttack now includes its search configuration (tree_width, tree_depth, branching_factor, on_topic_checking_enabled, desired_response_prefix) in its identifier. These parameters change what the attack actually does, so results produced under different settings were previously indistinguishable and could be reused for one another.

This changes TAP's eval hash for every consumer, not just this benchmark — including FoundryScenario, where TAP is one of the available techniques. Existing TAP results will no longer match for cache reuse and will be re-run once.

FoundryScenario.VERSION is deliberately left at 1. A changed eval hash already fails safe: it produces a cache miss, so stale results are ignored rather than incorrectly reused. Bumping the scenario version would additionally invalidate reuse for every other Foundry technique whose identifier did not change, turning a narrow and correct invalidation into a broad one.

Other changes

Generic technique overrides. The scenario exposes a single typed --technique-args parameter taking <technique>.<argument>=<value> entries, dispatched through AttackTechniqueFactory.with_attack_kwargs. This replaces the TAP-specific tap_tree_width / tap_tree_depth / tap_branching_factor / tap_batch_size parameters introduced earlier in this same PR, so the scenario no longer hardcodes knowledge of one technique. Those parameters never shipped, so this is not a breaking change. Unknown technique names, unknown kwargs, and malformed entries all fail with a clear message. The pipeline keeps its operator-facing tapTreeWidth-style inputs and its quick/full tuning; they are now marshalled into --technique-args.
Scorer block policy reaches the scorers that call an LLM. RAISE_IF_DEFAULT_SCORER_BLOCKS now also applies on the registry-default-scorer branch. The registered default (scale_and_refusal) is a TrueFalseCompositeScorer wrapping a FloatScaleThresholdScorer and a TrueFalseInverterScorer, so a policy applied only at the root never reached the SelfAsk* leaves that actually decide whether a blocked scoring response raises. Scorer.with_scorer_block_policy — mirroring the existing get_chat_target delegation convention — is overridden by those three wrappers so each level forwards to its own children and copies only when something changed, leaving the shared registry instance unmutated.
Cache lookup degrades safely. If the analytics lookup against a restored cache fails partway through, the scenario warns, discards any partially collected reuse state, and continues as a cold run rather than pruning objectives from incomplete data. Identifier-construction and programming errors still propagate.

Tests and Documentation

Unit tests cover each behavior above rather than its implementation:

A structured refusal and a provider content filter are raised and recorded as distinct kinds; both stay UNDETERMINED and round-trip their typed marker through memory. A blocked result with no outcome reason still yields a usable reason.
Resume leaves preparation-failure rows pending while counting both measured failures and target-reached UNDETERMINED rows as complete.
--technique-args parsing, including a non-TAP technique, unregistered technique names, unknown kwargs, malformed entries, and conflicting repeats.
The registry-default scorer test builds the real scale_and_refusal shape and asserts the policy lands on both LLM-backed leaves while the shared instance and its leaves stay unmutated. Each wrapper also has a focused delegation test, including that it returns itself when already compliant. The fallback scorer path is covered separately.
Cache lookup failure after partial success produces no partial pruning or reuse.

The pipeline test now verifies structurally that every benchmark variable consumed by a run step's env: block is emitted by the profile-resolution step, and that every declared env var is actually read. It no longer restates literal profile values or template strings, so profile drift is caught without the test needing an edit for every tuning change.

No documentation or notebook changes, and no documented public API surface changes, so JupyText was not run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

This reverts commit cd131ef.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Use task-achievement evaluation so benchmark objectives outside Azure Content Safety categories are scored correctly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

…ataset

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

…ataset

Balance capped objective selection across harm categories and classify Azure cyber policy rejections as blocked scorer responses so benchmark attacks finish undetermined instead of erroring. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

…ataset

Treat blocked adversarial-chat responses as completed attack failures, including simulated-conversation preparation, while preserving unrelated error propagation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

…ataset

- New build_scripts/import_adversarial_benchmark_snapshot.py upserts an externally-produced technique-metrics.json (e.g. downloaded from an Azure DevOps pipeline artifact) into the committed benchmark metrics store, reusing the same upsert_benchmark_metrics() as the local exporter. - Supports --default-objective-target/--default-objective-scorer/ --default-dataset to backfill identity fields on rows produced by an exporter version that predates them, without overwriting a field a row already has. - Ingested the 6 real technique-metrics rows from Azure DevOps build 16150 (a preview run of PR microsoft#2551's not-yet-merged dataset/pipeline updates), backfilling objective_target=openai_chat and dataset=adversarial_benchmark_v1 per the run's actual pipeline parameters; objective_scorer is <unknown> since it wasn't captured by that pipeline version. - Documented the provenance of these rows and the new ingestion path in the dashboard docs, and regenerated the benchmark leaderboard HTML fragment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add stable objective identity fields, committed-store reuse, exporter upserts, and the rerunnable benchmark wrapper while preserving this branch's exact-result cache and balanced sampling behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Run the benchmark exporter with committed-store updates enabled and cover the pipeline command contract. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Match the persisted identifier shape, query both planned and inner attack hashes, and recover scorer identity for scoreless terminal refusals. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Gate committed-store updates behind an opt-in parameter and use a secret GitHub token to publish changes on an automation branch. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Combines PR #2551 benchmark-store/pipeline automation with PR #2667 metrics dashboard for an end-to-end hackathon demo run.

Adds a one-line diagnostic to the 'Publish benchmark store update as a PR' step that prints only the character count of GITHUB_PAT (never the value) before the existing not-configured guard. To be reverted once diagnosed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Azure DevOps macro substitution ($(var)) rewrites a literal $(GITHUB_PAT) reference anywhere it textually appears in a task's script body, not just in the env: mapping. The guard's "unresolved macro" fallback compared $GITHUB_PAT against that same literal text written directly in the script, so once the secret was actually configured, ADO substituted the real secret into both sides of the comparison -- making it always match and incorrectly report "GITHUB_PAT is not configured" even when it was correctly set (confirmed live: a genuine 93-char PAT still tripped the guard). Build the comparison sentinel from separate bash-only parts (dollar='$'; unresolved_macro="${dollar}(GITHUB_PAT)") so ADO's compile-time scanner never sees a literal $(GITHUB_PAT) substring to rewrite in the script; it's reassembled only once bash runs. Verified locally (bash -n, and running the extracted guard with bash) that this now correctly falls through when GITHUB_PAT holds a real value, and still reports "not configured" both when the variable is empty and when its macro is genuinely left unresolved. Also removes the temporary debug diagnostic added while investigating this, and updates the pipeline unit test to assert the corrected guard construction plus the absence of the vulnerable literal comparison. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: PyRIT Benchmark Bot <pyrit-benchmark-bot@users.noreply.github.com>

Resolves an import-list conflict in adversarial.py: main added ScenarioRunSizeEstimateCondition/Status/Factor while this branch added ScorerEvaluationIdentifier. Both sides are kept. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Distinguish adversarial-model refusals from provider blocks, narrow the resume gate so deterministic UNDETERMINED rows converge, and make the default-scorer block policy reach the scorers that actually call an LLM. - Add AdversarialChatRefusedException as a subclass of AdversarialChatResponseBlockedException. An SDK `refusal` field and a provider content filter both surface as response_error == "blocked", so the manager now branches on the structured refusal to report the adversarial model's own decision separately from a deployment filter. - Add AttackPreparationFailureKind.ADVERSARIAL_CHAT_REFUSED and from_exception() so producers map the raised failure to a kind instead of hardcoding ADVERSARIAL_CHAT_BLOCKED. - Carry the typed AttackPreparationFailure through SimulatedConversationResult and PromptSendingAttackParameters rather than a bare reason string, so the kind survives to AttackResult.metadata. - Narrow the resume gate: only ERROR rows and rows carrying a preparation failure stay pending. An UNDETERMINED row that reached the objective target recorded the best verdict its configuration allows, so counting it as incomplete re-sent it on every resume without ever converging. - Add Scorer.with_scorer_block_policy, mirroring get_chat_target, and override it on the composite, inverter, and threshold wrappers. The previous isinstance(scorer, MessageScorer) guard bailed at the root of the registered scale_and_refusal default, so the block policy never reached the SelfAsk leaves. Each level copies only when something changed, leaving the shared registry instance unmutated. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Three hooks failed on the merge commit, all from changes in this branch: - check-no-rest-roles: AdversarialChatRefusedException used a Sphinx :class: role. PyRIT renders docstrings with MyST, so the role would appear as literal text. Use plain double backticks; gen_api_md links known symbols at render time. - ty redundant-condition-strict: AttackPreparationFailureKind has two members, so ty narrowed the second identity check to always-true and the trailing fallback to unreachable. Replace the if-chain with a module-level lookup table, which also makes a future member without a description fail loudly instead of inheriting generic text. - ty unsound-return-statement: params.get("use_cached") is Any, so the ternary inferred bool | (Any & ~None), assignable to but not a subtype of the declared bool return. Coerce the runtime branch explicitly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Resolves conflicts with #2494, which added benchmark-owned adversarial guidance and touched the same regions of AdversarialBenchmark. - VERSION: both branches independently bumped 4 -> 5 for different reasons. Main's v5 (shared benchmark guidance) is already published, and this branch's scoring change is additive on top, so the merged scenario is neither v5 and becomes v6 with both rationale lines retained. Keeping 5 would let runs with different scoring resume and reuse each other's results. - Main's note that the behavioral cache stays keyed across scenario versions is dropped: this branch deliberately made cache reuse version-gated, so that sentence no longer describes the code. - _build_atomic_attacks_async: main wraps each factory with the guidance prefix, this branch resolves generic technique_args overrides. Composed so overrides are resolved first and the prefix is applied to the resulting factories. Applying the prefix only to registry factories would silently drop guidance from overridden techniques and make their ASR non-comparable, which is the whole point of the benchmark. - Test target mocks keep TargetIdentifier (EVAL_ROOT for objective-target eval hashing, needed by this branch's cache identity) while taking main's added objective-scorer identifier mock. Adds a regression test proving an overridden factory still receives the guidance prefix; verified it fails when the prefix is applied before overrides are merged in. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Resolves conflicts with #2738 (Seed Expectation, scorer phase 7), which replaced the scorer condition API in the same files that gained with_scorer_block_policy. Scorer conflicts (message_scorer, float_scale_threshold, true_false_inverter, true_false_composite): kept with_scorer_block_policy and adopted main's _get_child_scorers()/_get_condition_type(), dropping matched_conditions, required_conditions and the wrapper _validate_expectation overrides. Those were pre-existing code from #2400 that #2738 deliberately removed, not part of this branch. Keeping them was not optional: Scorer.__init_subclass__ now raises TypeError when a subclass declares matched_conditions or required_conditions, so retaining them would break import pyrit.score. Executor conflicts (red_teaming, prompt_sending): import-only; unioned the attack_preparation imports with main's score_attack_response_async. Verified with_scorer_block_policy still propagates through a three-level composite, leaves shared registry scorers unmutated, and returns self when the policy is unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Mid-run adversarial blocks are no longer marked as preparation failures. The preparation marker means the objective target was never reached, which resume uses to re-run the objective. The catch sits inside the turn loop, so a block on turn 2+ stamped the marker even though an earlier turn had already probed the target, causing resume to re-run the objective under fresh conversation ids and orphan the turns that did land. The marker is now only attached when executed_turns is 0; a later block returns UNDETERMINED with a truncation reason instead. Outcome is unchanged either way, so a provider block still never counts as a defensive win. Cached-result buckets are re-sorted newest-first after merging. Each lookup returns newest-first on its own, but an objective present under both the technique hash and the inner-attack hash was merged from a set, so the concatenation order was arbitrary and the reuse pass keeps the first row it sees per objective. That could retain an older result than the cache holds. The parent-scenario read now degrades like the cached-result read. It runs inside _collect_reusable_cached_results, which _build_atomic_attacks_async calls with no guard, so a corrupt or schema-drifted cache aborted the whole run instead of giving up on reuse and proceeding cold. Test result stand-ins now carry a concrete timestamp, matching persisted results, which always have one. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Resolves conflicts in scenario resume-progress logic after #2857 ("Prevent duplicate attacks when scenario resume history cannot be read"). - pyrit/scenario/core/scenario.py: adopt main's single-query _get_completed_objective_hashes_by_attack() returning a dict keyed by (parent_collection, parent_eval_hash). Deliberately do NOT reintroduce the try/except swallow that main removed: treating a failed history read as empty progress would re-execute completed objectives. Re-apply this branch's AttackPreparationFailure skip so provider-blocked rows that never reached the objective target are not counted as completed progress, and document that exclusion in the docstring. - tests/unit/scenario/core/test_scenario_retry.py: keep main's _row helper signature plus the metadata parameter needed by the preparation-failure tests, drop the now-obsolete _make_atomic helper, and migrate this branch's two tests to the dict-returning API. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-09-28T16:01:38Z

Pytest (3.12, dev)

1 files ± 0 1 suites ±0 7m 45s ⏱️ + 1m 31s
20 874 tests +91 20 733 ✅ +91 141 💤 ±0 0 ❌ ±0
20 941 runs +91 20 800 ✅ +91 141 💤 ±0 0 ❌ ±0

Results for commit cccc43e. ± Comparison against base commit 82143b7.

This pull request removes 9 and adds 100 tests. Note that renamed tests count towards both.

tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_can_be_set_true
tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_defaults_to_false
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_analytics_lookup_exception_is_swallowed_per_hash
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_identifier_construction_failure_falls_back_to_empty
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_display_group_map_updated_for_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_no_injection_when_no_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_precomputed_results_injected_into_attack_results
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_filters_results_by_parent_collection
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_populates_precomputed_maps_for_skipped

tests.unit.analytics.test_result_analysis ‑ test_objective_target_eval_hash_for_legacy_direct_target
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_is_enabled_and_passed_to_scenario
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_restores_same_branch_state_including_failed_runs
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_database_is_published_even_after_failure
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_defaults_to_quick_profile_with_full_profile_available
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_passes_tap_tuning_through_the_generic_technique_args_flag
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Collect benchmark diagnostics]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Run benchmark and capture result snapshot]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_run_step_reads_every_environment_variable_it_declares
tests.unit.datasets.test_adversarial_benchmark_v1_dataset ‑ test_adversarial_benchmark_v1_resolves_by_name_async
…

♻️ This comment has been updated with latest results.

github-actions · 2026-09-28T16:03:22Z

Pytest (3.11, dev)

1 files ± 0 1 suites ±0 4m 19s ⏱️ - 2m 5s
20 874 tests +91 20 733 ✅ +91 141 💤 ±0 0 ❌ ±0
20 941 runs +91 20 800 ✅ +91 141 💤 ±0 0 ❌ ±0

Results for commit cccc43e. ± Comparison against base commit 82143b7.

This pull request removes 9 and adds 100 tests. Note that renamed tests count towards both.

tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_can_be_set_true
tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_defaults_to_false
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_analytics_lookup_exception_is_swallowed_per_hash
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_identifier_construction_failure_falls_back_to_empty
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_display_group_map_updated_for_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_no_injection_when_no_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_precomputed_results_injected_into_attack_results
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_filters_results_by_parent_collection
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_populates_precomputed_maps_for_skipped

tests.unit.analytics.test_result_analysis ‑ test_objective_target_eval_hash_for_legacy_direct_target
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_is_enabled_and_passed_to_scenario
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_restores_same_branch_state_including_failed_runs
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_database_is_published_even_after_failure
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_defaults_to_quick_profile_with_full_profile_available
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_passes_tap_tuning_through_the_generic_technique_args_flag
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Collect benchmark diagnostics]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Run benchmark and capture result snapshot]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_run_step_reads_every_environment_variable_it_declares
tests.unit.datasets.test_adversarial_benchmark_v1_dataset ‑ test_adversarial_benchmark_v1_resolves_by_name_async
…

♻️ This comment has been updated with latest results.

github-actions · 2026-09-28T16:03:43Z

Pytest (3.13, dev)

1 files ± 0 1 suites ±0 7m 3s ⏱️ +23s
20 874 tests +91 20 733 ✅ +91 141 💤 ±0 0 ❌ ±0
20 941 runs +91 20 800 ✅ +91 141 💤 ±0 0 ❌ ±0

Results for commit cccc43e. ± Comparison against base commit 82143b7.

This pull request removes 9 and adds 100 tests. Note that renamed tests count towards both.

tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_can_be_set_true
tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_defaults_to_false
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_analytics_lookup_exception_is_swallowed_per_hash
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_identifier_construction_failure_falls_back_to_empty
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_display_group_map_updated_for_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_no_injection_when_no_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_precomputed_results_injected_into_attack_results
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_filters_results_by_parent_collection
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_populates_precomputed_maps_for_skipped

tests.unit.analytics.test_result_analysis ‑ test_objective_target_eval_hash_for_legacy_direct_target
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_is_enabled_and_passed_to_scenario
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_restores_same_branch_state_including_failed_runs
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_database_is_published_even_after_failure
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_defaults_to_quick_profile_with_full_profile_available
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_passes_tap_tuning_through_the_generic_technique_args_flag
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Collect benchmark diagnostics]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Run benchmark and capture result snapshot]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_run_step_reads_every_environment_variable_it_declares
tests.unit.datasets.test_adversarial_benchmark_v1_dataset ‑ test_adversarial_benchmark_v1_resolves_by_name_async
…

♻️ This comment has been updated with latest results.

github-actions · 2026-09-28T16:15:37Z

Pytest (3.14, dev)

1 files ± 0 1 suites ±0 5m 14s ⏱️ - 1m 35s
20 874 tests +91 20 733 ✅ +91 141 💤 ±0 0 ❌ ±0
20 941 runs +91 20 800 ✅ +91 141 💤 ±0 0 ❌ ±0

Results for commit cccc43e. ± Comparison against base commit 82143b7.

This pull request removes 9 and adds 100 tests. Note that renamed tests count towards both.

tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_can_be_set_true
tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_defaults_to_false
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_analytics_lookup_exception_is_swallowed_per_hash
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_identifier_construction_failure_falls_back_to_empty
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_display_group_map_updated_for_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_no_injection_when_no_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_precomputed_results_injected_into_attack_results
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_filters_results_by_parent_collection
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_populates_precomputed_maps_for_skipped

tests.unit.analytics.test_result_analysis ‑ test_objective_target_eval_hash_for_legacy_direct_target
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_is_enabled_and_passed_to_scenario
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_restores_same_branch_state_including_failed_runs
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_database_is_published_even_after_failure
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_defaults_to_quick_profile_with_full_profile_available
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_passes_tap_tuning_through_the_generic_technique_args_flag
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Collect benchmark diagnostics]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Run benchmark and capture result snapshot]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_run_step_reads_every_environment_variable_it_declares
tests.unit.datasets.test_adversarial_benchmark_v1_dataset ‑ test_adversarial_benchmark_v1_resolves_by_name_async
…

♻️ This comment has been updated with latest results.

github-actions · 2026-09-28T16:16:02Z

Pytest (3.11, dev_all)

1 files ± 0 1 suites ±0 6m 42s ⏱️ -27s
21 204 tests +91 21 198 ✅ +91 6 💤 ±0 0 ❌ ±0
21 271 runs +91 21 265 ✅ +91 6 💤 ±0 0 ❌ ±0

Results for commit cccc43e. ± Comparison against base commit 82143b7.

This pull request removes 9 and adds 100 tests. Note that renamed tests count towards both.

tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_can_be_set_true
tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_defaults_to_false
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_analytics_lookup_exception_is_swallowed_per_hash
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_identifier_construction_failure_falls_back_to_empty
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_display_group_map_updated_for_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_no_injection_when_no_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_precomputed_results_injected_into_attack_results
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_filters_results_by_parent_collection
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_populates_precomputed_maps_for_skipped

tests.unit.analytics.test_result_analysis ‑ test_objective_target_eval_hash_for_legacy_direct_target
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_is_enabled_and_passed_to_scenario
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_restores_same_branch_state_including_failed_runs
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_database_is_published_even_after_failure
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_defaults_to_quick_profile_with_full_profile_available
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_passes_tap_tuning_through_the_generic_technique_args_flag
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Collect benchmark diagnostics]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Run benchmark and capture result snapshot]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_run_step_reads_every_environment_variable_it_declares
tests.unit.datasets.test_adversarial_benchmark_v1_dataset ‑ test_adversarial_benchmark_v1_resolves_by_name_async
…

♻️ This comment has been updated with latest results.

github-actions · 2026-09-28T16:16:05Z

Pytest (3.13, dev_all)

1 files ± 0 1 suites ±0 7m 36s ⏱️ +27s
21 204 tests +91 21 198 ✅ +91 6 💤 ±0 0 ❌ ±0
21 271 runs +91 21 265 ✅ +91 6 💤 ±0 0 ❌ ±0

Results for commit cccc43e. ± Comparison against base commit 82143b7.

This pull request removes 9 and adds 100 tests. Note that renamed tests count towards both.

tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_can_be_set_true
tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_defaults_to_false
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_analytics_lookup_exception_is_swallowed_per_hash
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_identifier_construction_failure_falls_back_to_empty
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_display_group_map_updated_for_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_no_injection_when_no_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_precomputed_results_injected_into_attack_results
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_filters_results_by_parent_collection
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_populates_precomputed_maps_for_skipped

tests.unit.analytics.test_result_analysis ‑ test_objective_target_eval_hash_for_legacy_direct_target
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_is_enabled_and_passed_to_scenario
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_restores_same_branch_state_including_failed_runs
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_database_is_published_even_after_failure
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_defaults_to_quick_profile_with_full_profile_available
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_passes_tap_tuning_through_the_generic_technique_args_flag
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Collect benchmark diagnostics]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Run benchmark and capture result snapshot]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_run_step_reads_every_environment_variable_it_declares
tests.unit.datasets.test_adversarial_benchmark_v1_dataset ‑ test_adversarial_benchmark_v1_resolves_by_name_async
…

♻️ This comment has been updated with latest results.

github-actions · 2026-09-28T16:16:30Z

Pytest (3.12, dev_all)

1 files ± 0 1 suites ±0 8m 24s ⏱️ +31s
21 204 tests +91 21 198 ✅ +91 6 💤 ±0 0 ❌ ±0
21 271 runs +91 21 265 ✅ +91 6 💤 ±0 0 ❌ ±0

Results for commit cccc43e. ± Comparison against base commit 82143b7.

This pull request removes 9 and adds 100 tests. Note that renamed tests count towards both.

tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_can_be_set_true
tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_defaults_to_false
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_analytics_lookup_exception_is_swallowed_per_hash
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_identifier_construction_failure_falls_back_to_empty
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_display_group_map_updated_for_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_no_injection_when_no_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_precomputed_results_injected_into_attack_results
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_filters_results_by_parent_collection
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_populates_precomputed_maps_for_skipped

tests.unit.analytics.test_result_analysis ‑ test_objective_target_eval_hash_for_legacy_direct_target
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_is_enabled_and_passed_to_scenario
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_restores_same_branch_state_including_failed_runs
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_database_is_published_even_after_failure
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_defaults_to_quick_profile_with_full_profile_available
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_passes_tap_tuning_through_the_generic_technique_args_flag
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Collect benchmark diagnostics]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Run benchmark and capture result snapshot]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_run_step_reads_every_environment_variable_it_declares
tests.unit.datasets.test_adversarial_benchmark_v1_dataset ‑ test_adversarial_benchmark_v1_resolves_by_name_async
…

♻️ This comment has been updated with latest results.

github-actions · 2026-09-28T16:16:43Z

Pytest (3.14, dev_all)

1 files ± 0 1 suites ±0 5m 49s ⏱️ - 1m 30s
21 204 tests +91 21 198 ✅ +91 6 💤 ±0 0 ❌ ±0
21 271 runs +91 21 265 ✅ +91 6 💤 ±0 0 ❌ ±0

Results for commit cccc43e. ± Comparison against base commit 82143b7.

This pull request removes 9 and adds 100 tests. Note that renamed tests count towards both.

tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_can_be_set_true
tests.unit.scenario.benchmark.test_adversarial.TestAdversarialBenchmarkInit ‑ test_skip_cached_defaults_to_false
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_analytics_lookup_exception_is_swallowed_per_hash
tests.unit.scenario.benchmark.test_adversarial.TestCollectCachedCompletionPairs ‑ test_identifier_construction_failure_falls_back_to_empty
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_display_group_map_updated_for_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_no_injection_when_no_cached_attacks
tests.unit.scenario.benchmark.test_adversarial.TestRunAsyncCacheInjection ‑ test_precomputed_results_injected_into_attack_results
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_filters_results_by_parent_collection
tests.unit.scenario.benchmark.test_adversarial.TestSkipCachedFilter ‑ test_use_cached_true_populates_precomputed_maps_for_skipped

tests.unit.analytics.test_result_analysis ‑ test_objective_target_eval_hash_for_legacy_direct_target
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_is_enabled_and_passed_to_scenario
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_cache_restores_same_branch_state_including_failed_runs
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_database_is_published_even_after_failure
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_defaults_to_quick_profile_with_full_profile_available
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_benchmark_passes_tap_tuning_through_the_generic_technique_args_flag
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Collect benchmark diagnostics]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_every_consumed_benchmark_variable_is_emitted_by_profile_resolution[Run benchmark and capture result snapshot]
tests.unit.build_scripts.test_adversarial_benchmark_pipeline ‑ test_run_step_reads_every_environment_variable_it_declares
tests.unit.datasets.test_adversarial_benchmark_v1_dataset ‑ test_adversarial_benchmark_v1_resolves_by_name_async
…

♻️ This comment has been updated with latest results.

hannahwestra25 and others added 12 commits August 27, 2026 15:48

Add adversarial benchmark objectives dataset

d5731e3

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Align benchmark pipeline technique defaults

b4556d9

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Rename adversarial benchmark dataset to v1

0b27591

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Refactor adversarial benchmark output

cd131ef

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Revert "Refactor adversarial benchmark output"

7d27e54

This reverts commit cd131ef.

Use existing output printers for benchmark artifacts

80d9511

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Expand adversarial benchmark dataset

e7590be

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Fix adversarial benchmark scoring

5124bb3

Use task-achievement evaluation so benchmark objectives outside Azure Content Safety categories are scored correctly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

FEAT: persist adversarial benchmark cache

68c5369

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Merge origin/main into copilot/adversarial_dataset

9a435de

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Merge remote-tracking branch 'origin/main' into copilot/adversarial_d…

fc84590

…ataset

Merge remote-tracking branch 'origin/main' into copilot/adversarial_d…

517e11d

…ataset

hannahwestra25 mentioned this pull request Sep 15, 2026

Add metrics dashboard: scorer quality and benchmark leaderboard #2667

Closed

hannahwestra25 added 6 commits September 15, 2026 17:11

PERF: add quick adversarial benchmark profile

218bc92

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

Merge remote-tracking branch 'origin/main' into copilot/adversarial_d…

d35779e

…ataset

Merge remote-tracking branch 'origin/main' into copilot/adversarial_d…

43ed84d

…ataset

Merge remote-tracking branch 'origin/main' into copilot/adversarial_d…

889fcef

…ataset

hannahwestra25 added 4 commits September 16, 2026 15:01

CI update adversarial benchmark metrics store

666a699

Run the benchmark exporter with committed-store updates enabled and cover the pipeline command contract. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b6deb5-a62a-403d-9ed9-2eb369b5174d

hannahwestra25 mentioned this pull request Sep 16, 2026

Hackathon demo: adversarial benchmark pipeline + metrics dashboard #2684

Merged

hannahwestra25 and others added 3 commits September 16, 2026 16:00

Hackathon demo: adversarial benchmark pipeline + metrics dashboard

e8f64ff

Combines PR #2551 benchmark-store/pipeline automation with PR #2667 metrics dashboard for an end-to-end hackathon demo run.

varunj-msft mentioned this pull request Sep 18, 2026

[BREAKING] FEAT: Normalize simulated conversation prompts to SeedPrompt #2698

Merged

MAINT: Update adversarial benchmark store (build 16273) (#2690)

285c7e5

Co-authored-by: PyRIT Benchmark Bot <pyrit-benchmark-bot@users.noreply.github.com>

Copilot AI and others added 2 commits September 23, 2026 10:53

Merge branch 'main' into copilot/adversarial_dataset

2c3f625

varunj-msft reviewed Sep 23, 2026

View reviewed changes

Comment thread pyrit/executor/attack/component/adversarial_conversation_manager.py

Comment thread pyrit/scenario/core/scenario.py Outdated

Comment thread pyrit/scenario/core/scenario.py Outdated

Comment thread pyrit/executor/attack/multi_turn/tree_of_attacks.py

hannahwestra25 and others added 7 commits September 23, 2026 16:23

Merge branch 'main' into copilot/adversarial_dataset

21c74da

Merge branch 'main' into copilot/adversarial_dataset

41fabb5

Merge branch 'main' into copilot/adversarial_dataset

0e7079d

Adrian Gavrila (adrian-gavrila) reviewed Sep 25, 2026

View reviewed changes

Comment thread pyrit/executor/attack/multi_turn/red_teaming.py

Comment thread pyrit/scenario/scenarios/benchmark/adversarial.py

Comment thread pyrit/scenario/scenarios/benchmark/adversarial.py Outdated

Comment thread pyrit/scenario/scenarios/benchmark/adversarial.py Outdated

Utkarsh Bahuguna (u7k4rs6) mentioned this pull request Sep 28, 2026

FIX Share scenario success statistics across SDK, backend and reports #2820

Merged

Copilot AI added 2 commits September 28, 2026 11:33

Merge branch 'main' into copilot/adversarial_dataset

7df44f9

Adrian Gavrila (adrian-gavrila) approved these changes Sep 28, 2026

View reviewed changes

Merge branch 'main' into copilot/adversarial_dataset

cccc43e

hannahwestra25 enabled auto-merge September 28, 2026 21:14

hannahwestra25 added this pull request to the merge queue Sep 28, 2026

Merged via the queue into main with commit 9f4b0c2 Sep 28, 2026
59 checks passed

Roman Lutz (romanlutz) mentioned this pull request Oct 8, 2026

FEAT Add role-aware Scenario target-attempt accounting #3043

Open

Navigation Menu

Repository navigation

FEAT: Improve adversarial benchmark dataset and scoring - #2551

FEAT: Improve adversarial benchmark dataset and scoring#2551
hannahwestra25 merged 46 commits into
mainfrom
copilot/adversarial_dataset

hannahwestra25 commented Sep 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

hannahwestra25 commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Provider blocks vs. adversarial-model refusals

Notable behavior change

Cache reuse break for TAP consumers

Other changes

Tests and Documentation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pytest (3.12, dev)

Uh oh!

github-actions Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pytest (3.11, dev)

Uh oh!

github-actions Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pytest (3.13, dev)

Uh oh!

github-actions Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pytest (3.14, dev)

Uh oh!

github-actions Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pytest (3.11, dev_all)

Uh oh!

github-actions Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pytest (3.13, dev_all)

Uh oh!

github-actions Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pytest (3.12, dev_all)

Uh oh!

github-actions Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pytest (3.14, dev_all)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hannahwestra25 commented Sep 3, 2026 •

edited

Loading

github-actions Bot commented Sep 28, 2026 •

edited

Loading

github-actions Bot commented Sep 28, 2026 •

edited

Loading

github-actions Bot commented Sep 28, 2026 •

edited

Loading

github-actions Bot commented Sep 28, 2026 •

edited

Loading

github-actions Bot commented Sep 28, 2026 •

edited

Loading

github-actions Bot commented Sep 28, 2026 •

edited

Loading

github-actions Bot commented Sep 28, 2026 •

edited

Loading

github-actions Bot commented Sep 28, 2026 •

edited

Loading