PR-review sidecar eval: next injection-format experiments

[legreffier]

Context

Branch experiment/pr-review-sidecar-eval contains the full A/B/C experiment (5 PRs × 4 arms: baseline, sidecar, filler, intent-claims). See RESULTS.md in .exploration/pr-review-eval/.

Key finding: all anchored formats (sidecar, intent-claims) constrain the reviewer's attention to what the author discussed. Filler control disproved token-dilution — mechanism is targeted anchoring on author-framed areas. No single format is a strict win over baseline.

Candidate next experiments

Ordered by cheapness / informativeness:

• Dual-output — require ## Defects (independent of claims) + ## Intent-vs-diff gaps in one response. Forces both review modes.
• Sidecar-after-diff — positional test. Reviewer reads diff cold, then reconciles with sidecar.
• Tool-use / on-demand — reviewer starts with no sidecar, can call entries_search / diary_tags to pull context when needed. No upfront anchor.
• Per-file filtering by refs — smaller anchoring surface, only inject entries relevant to the file under review.
• Adversarial framing variant — "find where diff contradicts claims" + "find defects unrelated to any claim" as two explicit passes.
• Structured table instead of prose — compact entry format to reduce attention competition with diff tokens.

Related

• Diary entries: e:24c01c13 (semantic decision), e:301a169f (reflection), e:76306cbb (CLI limit bug)
• Issues: entries list: expose ids filter (route + SDK + CLI + MCP) #819 (entries list ids filter), CLI: surface REST error bodies + validate --limit against REST cap #827 (CLI silent 400 on --limit > 100)
• Scripts: scripts/pr-review-eval/ on the experiment branch

[getlarge]

New idea around the sidecar, we could use the injected context (entries from the branch) to evaluate the complexity of the PR and enable auto merge for simple PRs.
Criteria could be:

• number of lines
• new DB migrations?
• touching the authentication or authorization parts?
• new API or API surface changes?
• removing tests?
• changing infrastructure?
• what else?

I could well imagine this being run via a DSPy CoT - using the rendered pack judge or eval runner judge as an example.
If we want to make it reusable, that would mean:

• custom criteria
• repo URL (inferred from the diary id? access should be granted via the Github app)
• PR number (allow local reviews?)

[getlarge]

part of assess_brief task type