feat(agent-wiki): add compare-outcomes pass for contrastive guidelines#274
Open
vinodmut wants to merge 4 commits into
Open
feat(agent-wiki): add compare-outcomes pass for contrastive guidelines#274vinodmut wants to merge 4 commits into
vinodmut wants to merge 4 commits into
Conversation
Adds a new pipeline skill, agent-wiki-compare-outcomes, that derives *contrastive* guidelines by comparing successful vs failed trajectories for the same/similar task — rather than mining rules from a single trajectory. It LLM-judges success/failure from the normalized transcript (no dependency on benchmark-specific outcome labels) and grounds each rule in evidence (task wording, observed tool/API calls, transcript/doc snippets). The bundled compare_outcomes.py is self-contained (stdlib only). Wires it into the ingest orchestrator as a conditional Step 4.5 (after synthesize, before consolidate): the description, subagent list, pipeline diagram, and a new step section that spawns one agent-wiki-compare-outcomes subagent over the corpus when there's a success/failure contrast, renders any strong contrastive guidelines, and skips cleanly when there's no contrast. Documents the new pass in the overview docs so it's discoverable: the README skills tree, and design.md's pipeline diagram, stage table, ingest narrative, and a short "learning from contrast" rationale. Ported from the appworld-agent-wiki-experiment branch; scoped to just the new skill + its ingest wiring (the branch's separate consolidate "mine step" and synthesize changes are intentionally not included). Builder/CI conventions followed: file-local `# mypy: ignore-errors` header matching sibling scripts.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds an ChangesCompare-outcomes skill and pipeline docs
Compare-outcomes script
Estimated code review effort: 4 (Complex) | ~60 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@explorations/agent-wiki/skills/agent-wiki-compare-outcomes/scripts/compare_outcomes.py`:
- Around line 261-276: The client.chat.completions.create() call lacks an
explicit timeout configuration, which could cause the script to block
indefinitely if the API becomes unresponsive during batch processing. Add a
timeout to prevent excessive blocking: either add a timeout parameter when
instantiating the OpenAI client (e.g., timeout=60.0), or use the with_options()
method on the client immediately before calling chat.completions.create() to
apply the timeout at the request level. Choose whichever approach fits your
codebase structure best.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: d9b9045a-28f8-42e2-8491-cb9edc3918d0
📒 Files selected for processing (5)
explorations/agent-wiki/README.mdexplorations/agent-wiki/docs/design.mdexplorations/agent-wiki/skills/agent-wiki-compare-outcomes/SKILL.mdexplorations/agent-wiki/skills/agent-wiki-compare-outcomes/scripts/compare_outcomes.pyexplorations/agent-wiki/skills/agent-wiki-ingest/SKILL.md
Addresses CodeRabbit review finding: Add a timeout to the LLM API call A 60s client-level timeout prevents the batch judge loop from blocking indefinitely if the API becomes unresponsive.
…ki-compare-outcomes
…ki-compare-outcomes
Contributor
Author
|
@visahak @illeatmyhat @jayaramkr this is green and ready for review 🟢 All checks pass (GitHub Actions, CodeRabbit, and Tekton), and the branch is up to date with |
Collaborator
|
@vinodmut This is part of the explorations, so it does not conflict with the full and lite usage paths right? |
illeatmyhat
approved these changes
Jul 9, 2026
illeatmyhat
left a comment
illeatmyhat
left a comment
Collaborator
There was a problem hiding this comment.
explorations don't hurt anything
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A new pass in the
agent-wikiingest pipeline —agent-wiki-compare-outcomes— that derives contrastive guidelines by comparing successful vs failed trajectories for the same (or similar) task, rather than mining rules from a single trajectory.Every other pass in the pipeline (summarize / extract / synthesize) learns from one trajectory at a time. This pass learns from the contrast: a rule is only promoted when it's backed by a failed path, a successful path, and concrete trajectory evidence (task wording, observed tool/API calls, transcript/doc snippets). It can LLM-judge success/failure straight from the normalized transcript, so it does not depend on benchmark-specific outcome labels.
Extends the agent-wiki exploration merged in #268; related to the offline extraction/consolidation idea in #256.
Changes
SKILL.md): a 3-step workflow — build an evidence pack over normalized trajectories (grouped bytask_id, success/failure judged or stored), inspect candidate rules, and promote only strong ones (one failed + one successful run in the same group, a task-action tool/API or workflow difference, source IDs for both sides). Weak candidates stay hypotheses, not rules.compare_outcomes.py): groups traces, contrasts success/failed runs, extracts tool/API calls + transcript evidence, optionally LLM-judges outcomes (--judge-outcomes never|missing|always), and emits an analysis JSON + Markdown (and optional render-ready guideline entities). Stdlib-only, no repo-internal deps.Scope
This ports only the compare-outcomes capability + its ingest wiring from the
appworld-agent-wiki-experimentbranch. That branch also bundled unrelated changes (a consolidate "mine step" rewrite, a synthesize faithfulness rule, arun_agent_wiki_skill_pass.pyhelper) — those are intentionally not included here, to keep this PR focused.Verification
ruff check+ruff format --check: clean.mypy .: clean (the script carries the# mypy: ignore-errorsheader used by every sibling exploration/reference script).detect-secrets: passes.task_id, contrasts the two runs, and emits the analysis without error.No changes outside
explorations/agent-wiki/.Summary by CodeRabbit
New Features
Documentation