feat(agent): inference-time plan selection — rank candidate plans with TaskOutcomeJudge (best-of-N) #10583

Inference-time plan selection — use TaskOutcomeJudge to rank candidate plans before executing (best-of-N)

Part of umbrella #10542 · Theme A (collaboration core / eval).

Research origin: AgentRM (arxiv 2605.02801) — a reward model guides agents to choose among candidate actions/trajectories at inference, not just to train weights. AutoBot already has the reward model (TaskOutcomeJudge) but only fires it after the fact.

The gap (verified)

• judges/task_outcome_judge.py scores task outcomes on accuracy/completeness/efficiency and persists them.
• Its only callers are post-hoc: phase_progression_manager.py:735-737 (idle self-improvement) and api/agents_self_improvement.py (API read). It never influences a live decision.
• The planner generates a single plan — grep shows best-of-N / rerank machinery exists only for RAG retrieval (advanced_rag_optimizer.py), not for plans. judges/multi_agent_arbitrator.py arbitrates agent responses, but nothing arbitrates plans.

Net: AutoBot has a working judge and never uses it to pick a better plan before committing compute to execution.

What to build

• Add an opt-in best-of-N path in the planner: sample N candidate plans (vary strategy/temperature), score each with TaskOutcomeJudge (or a lighter plan-scoring judge) on predicted quality, execute the top-ranked.
• Gate behind a config flag + N cap so single-plan remains the default (cost control — this multiplies plan-time LLM calls).
• Reuse the existing judge dimensions; do not add a parallel scorer.
• Record the rejected candidates + scores in the trajectory for explainability and as future TaskPatternLearner signal.

Acceptance criteria

• With the flag on and N>1, the executed plan is the judge's top-ranked candidate (test with a seeded judge returning fixed scores).
• With the flag off (default), behavior and cost are unchanged (exactly one plan, no extra judge calls).
• Candidate scores are surfaced in the trajectory.

Entry points

autobot-backend/orchestration/workflow_planner.py (candidate generation + selection) · autobot-backend/judges/task_outcome_judge.py (reuse) · autobot-backend/autobot_shared/ssot_config.py (flag + N cap)

Relationship

Complements #10580 / #10581 (which feed the winning plan/prompt back into learning). Together with those, this closes the Ornith-style loop in-context: propose candidate scaffolds → reward-rank → execute → feed reward back.