Skip to content

refactor(work-driver): resolve project policy by comprehension + verified citation, not by regexing prose #407

Description

@randomm

The problem

Merge authority is decided by running five English regexes over AGENTS.md (work-driver-merge-authority.ts:66-76). This fails in both directions, silently:

  • randomm/nessie AGENTS.md:263"Agents may squash-merge a PR to main once CI is green…" — a deliberate, unambiguous grant. All three GRANT_PATTERNS miss it: the regex needs PRs directly after merge, and the indefinite article breaks it. The gate reported authoritySource: "none", indistinguishable from no grant at all, and the operator's opt-in was silently ignored.
  • This repo's own AGENTS.md:506"LLMs may squash-merge when gates pass" — fails the same way. It only works because :344 happens to be phrased the way pattern 1 wants. Two sentences saying the same thing; one matches.
  • :342"does NOT merge to main without explicit human approval" — escapes DENY_PATTERNS only because the regex wants do not, not does not. A one-word copy-edit silently flips the repo to denied.

And every pattern assumes English. A Finnish or Japanese AGENTS.md is unconditionally "no grant".

What the field does

Six harnesses surveyed (Claude Code, Cursor, Aider, Cline, Roo, Copilot coding agent). None of them regexes a doctrine file to extract a boolean. Doctrine is always consumed by a model; where a boolean must be reliable, it goes to a separate structured layer.

But the consensus is not "prose for everything" either:

Prose expresses preference, convention and judgment. Structured config expresses capability boundaries.

Anthropic states it twice — "Settings rules are enforced by the client regardless of what Claude decides to do. CLAUDE.md instructions shape Claude's behavior but are not a hard enforcement layer."

The tier that reconciles this appeared in 2026: prose-valued config judged by a second model. Claude Code's autoMode.{soft_deny,hard_deny} takes English sentences and "the classifier reads the same CLAUDE.md content Claude itself loads." Cursor's autoRun.block_instructions is the same shape. Both vendors fence it identically — the classifier is a second gate, never the durable one.

What to build

Replace the regexes with a general policy seam. The contract to copy is not ## Spec markdown — it is lens-review's report_finding tool call (lens-review-format.ts:40-94), whose schema Pi validates inside the child process, so "malformed calls never reach this code". That removes the text-parsing failure class rather than hardening it.

askPolicy(question, doctrineFiles) → tool call →
  { verdict: "permitted" | "forbidden" | "unstated",
    quote: "<verbatim sentence>",
    sourceFile: "AGENTS.md" }

The driver then decides deterministically, with no regex and no language assumption:

  1. permitted and quote appears verbatim in sourceFile → permit
  2. quote not found in the file → deny, and emit an event saying the citation failed — a hallucinated grant is a signal worth surfacing, not a silent fallback
  3. anything else, including an unparseable or absent answer → deny

MergeAuthority.quote and workflow-state-schema.ts:187 already exist for this and are documented as "The AGENTS.md sentence that granted or forbade it, verbatim". Only the producer changes.

The judge must not see the agent's own justification. Per Anthropic's auto-mode write-up: "The agent could generate persuasive rationalizations… If the classifier reads those, it can be talked into the wrong decision." It gets the doctrine text and the proposed action. Nothing else.

The durable tier stays in code and is not repo-controlled: default deny, the --merge operator grant, and the env kill-switch. Prose grants the exception; it cannot grant the rule.

Acceptance criteria

  • CANARY nessie's actual sentence — "Agents may squash-merge a PR to main once CI is green…" — resolves to permitted. It is the sentence that broke the regex, so this fails against main today.
  • A grant written in a non-English language resolves.
  • A fabricated quote not present in the file denies, and emits a citation-failure event.
  • A doctrine file that says nothing about merging denies.
  • An explicit prohibition denies even when a grant appears elsewhere in the file.
  • No regex anywhere in the resolution path; grep for GRANT_PATTERNS returns nothing.
  • Quality gate per AGENTS.md §1; docs updated.

Known limit, deliberately accepted for now

Citation existence is a hallucination guard, not a relevance guard — Deterministic Quoting says so of its own technique: "the AI can choose an irrelevant (but still verbatim) quote." LACE adds an NLI entailment check as the complement. Not building that until a verbatim-but-irrelevant citation is actually observed passing.

Depends on

#406 must land first. Making the grant easier to express in natural language, while a cycle can still write that natural language into the file the gate reads, widens the hole rather than closing it.

This work must ship as its own separate PR, independent of any other open issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions