Parent: #16108
Gap
ResponseQualityEvaluator returns one scalar plus prose. There are no itemized checks, and a
repair pass is never re-audited against the items the previous pass failed.
autobot-backend/rlm/evaluator.py:24-52 — _EVAL_PROMPT asks for SCORE / CRITIQUE / HINT
:137-153 — the three-line parse
autobot-backend/chat_workflow/graph.py:775 — reflect_on_response re-runs generation with the
hint, then re-scores the same scalar way
Proposal
An itemized check list where each item carries an explicit pass/fail, and the audit restarts
from item 1 after any repair rather than passing once — so a fix that breaks an earlier item is
caught. Keep the fixed-format parse that small models can satisfy (:137-153); this adds items,
not free-form reasoning.
Conditions
- The existing
max_reflections ceiling is the hard stop — no new unbounded loop
- Per-item outcomes recorded in
reflection_history so oscillation is diagnosable
Acceptance criteria
Parent: #16108
Gap
ResponseQualityEvaluatorreturns one scalar plus prose. There are no itemized checks, and arepair pass is never re-audited against the items the previous pass failed.
autobot-backend/rlm/evaluator.py:24-52—_EVAL_PROMPTasks forSCORE/CRITIQUE/HINT:137-153— the three-line parseautobot-backend/chat_workflow/graph.py:775—reflect_on_responsere-runs generation with thehint, then re-scores the same scalar way
Proposal
An itemized check list where each item carries an explicit pass/fail, and the audit restarts
from item 1 after any repair rather than passing once — so a fix that breaks an earlier item is
caught. Keep the fixed-format parse that small models can satisfy (
:137-153); this adds items,not free-form reasoning.
Conditions
max_reflectionsceiling is the hard stop — no new unbounded loopreflection_historyso oscillation is diagnosableAcceptance criteria
ReflectionResultcarries per-item outcomesmax_reflections; a test asserts termination underconflicting items