You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
tech-debt(quality): LLM-judged quality is implemented three times — ReflectionResult, JudgmentResult and ThinkCategory.SELF_REFLECTION #16133
One concept, three vocabularies, no shared abstraction
"Have an LLM score an output 0.0–1.0, return a critique, compare against a threshold, decide
whether to accept or redo" is implemented at least three separate ways with no common type.
The judges plane is six modules — agent_response_judge, workflow_step_judge, multi_agent_arbitrator, task_outcome_judge, security_risk_judge — and grep -rn "from rlm\|ResponseQualityEvaluator" autobot-backend/judges returns nothing: not one
of them reuses the evaluator that does the same job.
The clearest instance
agent_loop/types.py:101 reads:
SELF_REFLECTION=auto() # RLM: evaluating own response quality (#1373)
and think_tool.py:116-126 then re-implements the RLM scoring prompt inline — "Provide your quality
score (0.0-1.0) and specific critique" — rather than calling ResponseQualityEvaluator, which was
built under that same issue #1373. A comment naming the canonical implementation, directly above a
fork of it. It also has no non-test caller.
Why this matters now rather than later
#16112 and #16114 both change ReflectionResult — adding per-item outcomes and provenance. Landing
those improves one of three planes and widens the gap to the other two. Consolidating first means
those two children improve every quality-judging path instead of one.
This is the repo's own standard: one canonical name, consolidate rather than fork.
Acceptance criteria
One result type for LLM-judged quality (score, critique, verdict, threshold outcome), with ReflectionResult and JudgmentResult collapsed onto it or one expressed in terms of the other
ThinkCategory.SELF_REFLECTION either calls ResponseQualityEvaluator or is removed as an
unwired fork — decided, not left as both
The judges import the shared evaluator rather than each parsing their own score format
ReflectionVerdict and JudgmentConfidence/JudgmentDimension are reconciled — one enum per
concept, no parallel vocabulary
No behaviour change to existing callers; a test pins each plane's current verdicts across the move
Selector
grep -rnE "^class .*(Enum)|^class Judgment|^class Reflection" over judges/, rlm/, agent_loop/types.py
plus grep -rn "from rlm\|ResponseQualityEvaluator" autobot-backend/judges (empty — the absence is
the finding, and the control is that the same selector returns hits in chat_workflow/graph.py, knowledge/pipeline/cognifiers/recursive_summarizer.py and eval/runner.py, which do import it).
Activity