Is your feature request related to a problem? Please describe.
PyRIT has no loader for CyberSecEval, although #511 lists it as a tool to reach parity with. Its MITRE benchmark is a compact way to test how readily a target helps with cyberattacks: 1,000 prompts, 100 for each of ten MITRE ATT&CK categories, each labelled with an ATT&CK technique.
Describe the solution you'd like
A _CyberSecEvalMitreDataset remote loader that fetches mitre_benchmark_100_per_category_with_augmentation.json from a pinned commit of meta-llama/PurpleLlama (MIT licensed) and emits one SeedPrompt per row, with a CyberSecEvalMitreCategory enum filter. The ATT&CK category, technique ID and technique name go in metadata.
I have a working implementation with unit tests and can open a PR.
Describe alternatives you've considered, if relevant
Each row has a short base_prompt and the longer mutated_prompt that CyberSecEval actually sends to the model. I used mutated_prompt as a SeedPrompt value so the seeds reproduce the benchmark, and kept base_prompt in metadata. If you would rather have base_prompt as a SeedObjective, I can switch.
Additional context
Scope is the English MITRE prompts only. The judge and expansion steps, the false refusal rate set and the machine-translated multilingual set are left out.
Is your feature request related to a problem? Please describe.
PyRIT has no loader for CyberSecEval, although #511 lists it as a tool to reach parity with. Its MITRE benchmark is a compact way to test how readily a target helps with cyberattacks: 1,000 prompts, 100 for each of ten MITRE ATT&CK categories, each labelled with an ATT&CK technique.
Describe the solution you'd like
A
_CyberSecEvalMitreDatasetremote loader that fetchesmitre_benchmark_100_per_category_with_augmentation.jsonfrom a pinned commit of meta-llama/PurpleLlama (MIT licensed) and emits oneSeedPromptper row, with aCyberSecEvalMitreCategoryenum filter. The ATT&CK category, technique ID and technique name go inmetadata.I have a working implementation with unit tests and can open a PR.
Describe alternatives you've considered, if relevant
Each row has a short
base_promptand the longermutated_promptthat CyberSecEval actually sends to the model. I usedmutated_promptas aSeedPromptvalue so the seeds reproduce the benchmark, and keptbase_promptin metadata. If you would rather havebase_promptas aSeedObjective, I can switch.Additional context
Scope is the English MITRE prompts only. The judge and expansion steps, the false refusal rate set and the machine-translated multilingual set are left out.