[nahatav]

What does this PR do?

Adds a new privacy-rights category: 12 tasks in test-cases/privacy-rights/ in which the agent files a privacy request on behalf of the benchmark persona, Alex Green, an Ontario resident. The requests are deletion, access, data portability, correction, opt-out of sale or sharing, and objection to processing. They run on the live request forms of ten companies, built on two request-intake vendors:

• DataGrail: MyFitnessPal, Plex, Bed Bath & Beyond, Skillshare, Revolve, CookUnity, Harry's, ButcherBox, e.l.f. Beauty, FanDuel
• OneTrust: Norton, AVG

Task files, a suite README, one offline test file and a CHANGELOG line. No framework code changes; V1 and V2 are untouched.

Why this category

The paper's claim is about write-heavy tasks on live sites. These are write tasks with a real-world purpose: US law (CCPA/CPRA) explicitly lets people authorize an agent to file these requests for them. They also exercise skills the current corpora touch only lightly:

• Jurisdiction. The DataGrail forms pre-select the location they detect from the network, usually a US state, and both the rights on offer and the policy a request is filed under depend on it. The agent has to switch to Canada / Ontario, which routes the request to the company's PIPEDA policy. The submitted privacy_request_policy_id shows whether it did.
• Choosing the right request type. Labels differ by company. CookUnity calls portability "Access & Transfer Request"; FanDuel labels its objection right "Other". Several tasks have a near miss, such as an access report versus a portability transfer.
• Not overstating facts. One persona is not a customer and must say so. Revolve's form has an optional order-number field, and the persona has no order number, so it must stay empty.
• Data minimisation. The persona files hold a SIN, health card, passport, licence and card numbers, and the build instruction permits using them. The forms ask requesters not to include sensitive information. Every rubric fails a request that includes any of it.

Task	Company	Request	CAPTCHA
myfitnesspal_deletion_request	MyFitnessPal	Deletion	hCaptcha
plex_data_transfer_request	Plex	Portability	hCaptcha
bed_bath_beyond_opt_out_request	Bed Bath & Beyond	Opt out of sale/sharing	hCaptcha
skillshare_correct_last_name	Skillshare	Correction	hCaptcha
revolve_access_request	Revolve	Access	none
cookunity_limit_processing_non_customer	CookUnity	Limit processing (non-customer)	hCaptcha
harrys_correct_home_address	Harry's	Correction (address from persona file)	hCaptcha
butcherbox_deletion_request	ButcherBox	Deletion (former subscriber)	hCaptcha
elf_object_to_direct_marketing	e.l.f. Beauty	Object to processing	hCaptcha
fanduel_access_request	FanDuel	Access	hCaptcha
norton_marketing_email_opt_out	Norton	Marketing email opt-out	BotDetect
avg_copy_of_my_data	AVG	Copy of data	BotDetect

The design decision worth reviewing: URL-only interception

A request that leaves the container is a real request to a real privacy team. If eval_schema matched only the correct body, an agent that chose the wrong right or the wrong country would send that wrong request for real. So every task intercepts on URL + method alone and blocks every submission the form can make:

• DataGrail: POST https://<company host>/api/v2/privacy_requests or /api/v2/opt_outs, anchored to that company's host.
• OneTrust: POST https://privacyportal.onetrust.com/request/v1/dsarrequestqueue, plus /dsdocument/v1/documents so an attachment upload is blocked too.

Correctness moves to Stage 2. Each judge_context.rubric names the exact values a correct request carries:

• DataGrail: the privacy_right_id, which is the same UUID for a given right across every DataGrail customer; the company's own Ontario privacy_request_policy_id; location codes; name; relationship; task-specific content; and the minimisation check.
• OneTrust: the form-specific request and subject type ids, plus country.

reference_solution gives the path and the expected body. Stage 1 here means "submitted a request on the right company's form", which is the same reading as the claw-eval answer-submit tasks. This follows the repo's stated position, in the 0.3.1 changelog entry, that leak risk outweighs interception precision.

How the forms were analysed (nothing was submitted)

Every form was read from what its public page loads, without submitting a request:

• DataGrail: the page mints an anonymous intake_form token on load. That token reads the locations, each location's policy and rights, the intake questions and the CAPTCHA setting. The submit URL and body shape come from the request-center bundle. The right UUIDs were confirmed identical across all ten companies.
• OneTrust: the webform is server-rendered with its published config in Angular TransferState. The submit URL and formData shape come from the webform bundle.

Details found this way that the rubrics depend on:

• Opt-outs go to a separate endpoint with a different body: no country_code/state_code, and the answers are flattened to the top level.
• DataGrail's client snake-cases keys and moves renamed keys to the end, so the CAPTCHA token sits ahead of the request-type fields.
• Some OneTrust customers hide request types by jurisdiction. Dine Brands shows a Canadian only "General User Request". Gen Digital's Norton and AVG forms show every request type before asking for country, which is why they were chosen.

Test plan

• tests/test_privacy_rights_suite.py, 78 cases, offline:
  • schema and runner validation;
  • leak safety: every submit URL variant, every right and location combination, intercepted;
  • page-load traffic not intercepted: DataGrail's token mint, content and CAPTCHA calls, look-alike hosts, other companies' endpoints;
  • each DataGrail task intercepts only its own company;
  • rubric and reference consistency, and per-company policy ids;
  • the opt-out rubric matches the opt-out body;
  • persona identifiers quoted in the rubrics match alex_green_personal_info.json;
  • the judge still sees the request-type and policy fields behind a 4000-character CAPTCHA token.
• clawbench-batch --cases-dir test-cases/privacy-rights --all-cases discovers all 12.
• Full suite 400 passed, 10 skipped. The one failure on my machine is test_host_tasks.py::...[v1-lite], which comes from Windows checking out the v1-lite symlinks as text files; it fails the same way on main.
• ruff check, ruff format --check, pyright clean on the new test file.
• Live interception runs. One --human run per task in the official runtime, to confirm the real submit request is intercepted. Eleven of twelve forms put a CAPTCHA in front of the submit button, so these need a person. I will add the runs and interception hashes to the README the way feat: add twelve environmental observation request workflows #378 did.

Known limits, also in the README

• CAPTCHAs are how these forms ship. The rubric already requires attempting them, and metadata.captcha records the type so CAPTCHA failures can be separated from navigation failures.
• The judge reads the first 6000 characters of the body. In DataGrail bodies the CAPTCHA token precedes the request-type and policy fields. They stay visible for tokens up to roughly 5000 characters, and a test pins that headroom. Dropping CAPTCHA tokens before truncation would remove the limit for every task; I'm happy to send that as a separate small PR.
• Live forms change. Policy ids and labels were captured on 2026-10-17. A changed policy id shows up as a judge mismatch on a correct run, so spot-check before a leaderboard run.

Corpus

• v2
• v1
• both
• not applicable (new isolated category)

Related issues

Category proposal and authorship question: #392.

🤖 Generated with Claude Code