feat: add optional also_consider input to adversarial review task

[alexeyv]

Summary

• Adds an optional also_consider parameter to review-adversarial-general.xml that allows callers to pass domain-specific areas to keep in mind during review
• This gently nudges the reviewer toward specific concerns without overriding normal adversarial analysis
• Backward compatible — existing callers passing only content continue to work unchanged

Motivation

While testing the adversarial review task in a documentation workflow, comparing outputs from the generic task vs a custom prompt with explicit focus categories showed that guided categories steered toward more actionable, user-relevant findings.

The generic task's "find at least ten issues" directive sometimes pushed toward technical nitpicks to hit the quota, while domain-specific hints steered toward issues that matter for that particular review context.

Implementation

Simple input parameter with descriptive text — no flow changes needed:

<input name="also_consider" required="false"
  desc="Optional areas to keep in mind during review alongside normal adversarial analysis" />

The LLM sees the input description and naturally incorporates it into its analysis.

Testing

Ran 10 test cases comparing baseline (no also_consider) against various configurations:

Test Case	also_consider	Result
TC01	(none)	Baseline: 12 findings, technical/systematic spread
TC02	Reader confusion, unanswered questions, ambiguities, unexplained jargon	Findings framed as questions a confused reader would ask
TC03	Missing examples, unclear usage, edge cases	Edge cases & examples highlighted
TC04	Auth vulnerabilities, token handling, input validation, info disclosure	16 findings — deep security dive, most findings of any test
TC05	REST conventions, response formats, pagination, versioning	REST conventions and versioning prominent
TC06	Error handling completeness (single item)	Light emphasis without domination
TC07	Quality issues, things that seem off (vague)	Minimal steering — similar to baseline
TC08	Is JWT expiration documented? Refresh tokens? Concurrent sessions? (specific questions)	First 3 findings directly answer the questions
TC09	Security, reader confusion, API design, performance (mixed)	Balanced spread across all domains
TC10	Things too detailed, things not detailed enough (contradictory)	Handled gracefully, similar to baseline

Key Findings

1. Specific items steer strongly — questions get directly answered first
2. Domain-focused items shift the lens — security focus produced 16 findings vs baseline's 12
3. Vague items have minimal effect — reviewer falls back to normal analysis
4. Single items nudge without dominating — light influence, balanced output
5. Contradictory items handled gracefully — no breakdown, sensible output

Test Artifacts

Included in test/adversarial-review-tests/:

• sample-content.md — Deliberately imperfect API docs used as review target
• test-cases.yaml — All 10 test configurations with expected behaviors
• README.md — Test instructions and evaluation criteria

Test plan

• Ran 10 test cases comparing baseline vs various also_consider configurations
• Verified backward compatibility (TC01 baseline works without parameter)
• All existing tests pass (npm run test)

🤖 Generated with Claude Code

[coderabbitai]

📝 Walkthrough

Walkthrough

This change adds an optional also_consider input field to the Adversarial Review (General) task, enabling reviewers to specify additional focus areas during analysis. Supporting test infrastructure includes test cases, sample content, and documentation describing how the input influences review outputs.

Changes

Cohort / File(s)	Summary
Core Task Modification src/core/tasks/review-adversarial-general.xml	Added optional input also_consider (non-required) with description "Optional areas to keep in mind during review alongside normal adversarial analysis"
Test Infrastructure test/adversarial-review-tests/README.md	New documentation file describing test suite purpose, structure, test content, execution method, evaluation criteria, and expected outcomes for TC01–TC10
Test Content test/adversarial-review-tests/sample-content.md	New documentation sample covering a User Authentication API with endpoints (login, logout, profile retrieval), request/response examples, error handling, and rate limiting
Test Cases test/adversarial-review-tests/test-cases.yaml	New YAML file defining 10 test cases (TC01–TC10) covering baseline, documentation-focused, security-focused, API design, and various also_consider configurations with expected behavior descriptions

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• feat: add editorial review tasks for structure and prose #1307: Modifies the same task file src/core/tasks/review-adversarial-general.xml (injects governance text into the <llm> block).

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change—adding an optional input parameter to the adversarial review task. It is concise, directly related to the primary changeset, and accurately summarizes the feature addition.
Description check	✅ Passed	The description is comprehensive and directly related to the changeset. It explains the motivation, implementation, testing approach, and key findings. All described changes align with the actual modifications shown in the raw summary.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@test/adversarial-review-tests/sample-content.md`:
- Around line 21-28: Replace the JWT-like string under the JSON "token" key with
an explicit non-secret placeholder to avoid secret-scanner hits: locate the
block containing "token": "eyJhbGciOiJIUzI1NiIs..." in the sample response and
change the value to a clearly redacted pattern such as "REDACTED_TOKEN" or
"<REDACTED_JWT>" so the file contains no real-looking secrets.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

Replace the JWT-like token to avoid secret-scanner hits.
Gitleaks flagged the token string as a generic API key. Even as sample content, this can fail CI or encourage unsafe copy‑paste. Replace it with an obvious placeholder or redacted pattern.

🧰 Tools

🪛 Gitleaks (8.30.0)

[high] 24-24: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.

(generic-api-key)

🤖 Prompt for AI Agents

In `@test/adversarial-review-tests/sample-content.md` around lines 21 - 28,
Replace the JWT-like string under the JSON "token" key with an explicit
non-secret placeholder to avoid secret-scanner hits: locate the block containing
"token": "eyJhbGciOiJIUzI1NiIs..." in the sample response and change the value
to a clearly redacted pattern such as "REDACTED_TOKEN" or "<REDACTED_JWT>" so
the file contains no real-looking secrets.