Skip to content

tech-debt(quality): LLM-judged quality is implemented three times — ReflectionResult, JudgmentResult and ThinkCategory.SELF_REFLECTION #16133

Open

Description

@mrveiss

Parent: #16108

One concept, three vocabularies, no shared abstraction

"Have an LLM score an output 0.0–1.0, return a critique, compare against a threshold, decide
whether to accept or redo" is implemented at least three separate ways with no common type.

Plane Types file:line
RLM ReflectionVerdict, ReflectionResult (quality_score, critique, refinement_hint, iteration) autobot-backend/rlm/types.py:24, :37
Judges JudgmentConfidence, JudgmentDimension, JudgmentResult autobot-backend/judges/__init__.py:31, :41, :68
Think tool ThinkCategory.SELF_REFLECTION + its own scoring prompt autobot-backend/agent_loop/types.py:101, agent_loop/think_tool.py:116-126

The judges plane is six modules — agent_response_judge, workflow_step_judge,
multi_agent_arbitrator, task_outcome_judge, security_risk_judge — and
grep -rn "from rlm\|ResponseQualityEvaluator" autobot-backend/judges returns nothing: not one
of them reuses the evaluator that does the same job.

The clearest instance

agent_loop/types.py:101 reads:

SELF_REFLECTION = auto()  # RLM: evaluating own response quality (#1373)

and think_tool.py:116-126 then re-implements the RLM scoring prompt inline — "Provide your quality
score (0.0-1.0) and specific critique" — rather than calling ResponseQualityEvaluator, which was
built under that same issue #1373. A comment naming the canonical implementation, directly above a
fork of it. It also has no non-test caller.

Why this matters now rather than later

#16112 and #16114 both change ReflectionResult — adding per-item outcomes and provenance. Landing
those improves one of three planes and widens the gap to the other two. Consolidating first means
those two children improve every quality-judging path instead of one.

This is the repo's own standard: one canonical name, consolidate rather than fork.

Acceptance criteria

  • One result type for LLM-judged quality (score, critique, verdict, threshold outcome), with
    ReflectionResult and JudgmentResult collapsed onto it or one expressed in terms of the other
  • ThinkCategory.SELF_REFLECTION either calls ResponseQualityEvaluator or is removed as an
    unwired fork — decided, not left as both
  • The judges import the shared evaluator rather than each parsing their own score format
  • ReflectionVerdict and JudgmentConfidence/JudgmentDimension are reconciled — one enum per
    concept, no parallel vocabulary
  • No behaviour change to existing callers; a test pins each plane's current verdicts across the move

Selector

grep -rnE "^class .*(Enum)|^class Judgment|^class Reflection" over judges/, rlm/, agent_loop/types.py
plus grep -rn "from rlm\|ResponseQualityEvaluator" autobot-backend/judges (empty — the absence is
the finding, and the control is that the same selector returns hits in chat_workflow/graph.py,
knowledge/pipeline/cognifiers/recursive_summarizer.py and eval/runner.py, which do import it).

Relates to #1373, #16112, #16114.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions