Skip to content

feat(rlm): reflection is a single scalar — no itemized checks and no re-audit after repair #16112

Description

@mrveiss

Parent: #16108

Gap

ResponseQualityEvaluator returns one scalar plus prose. There are no itemized checks, and a
repair pass is never re-audited against the items the previous pass failed.

  • autobot-backend/rlm/evaluator.py:24-52 — _EVAL_PROMPT asks for SCORE / CRITIQUE / HINT
  • :137-153 — the three-line parse
  • autobot-backend/chat_workflow/graph.py:775 — reflect_on_response re-runs generation with the
    hint, then re-scores the same scalar way

Proposal

An itemized check list where each item carries an explicit pass/fail, and the audit restarts
from item 1 after any repair
rather than passing once — so a fix that breaks an earlier item is
caught. Keep the fixed-format parse that small models can satisfy (:137-153); this adds items,
not free-form reasoning.

Conditions

  • The existing max_reflections ceiling is the hard stop — no new unbounded loop
  • Per-item outcomes recorded in reflection_history so oscillation is diagnosable

Acceptance criteria

  • ReflectionResult carries per-item outcomes
  • A repair that regresses an earlier item is caught by a test
  • Iteration remains bounded by max_reflections; a test asserts termination under
    conflicting items

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions