fix(routing): fail-close verify mode and stop substring auto-bills by cursor[bot] · Pull Request #612 · ContextualWisdomLab/contextual-orchestrator

cursor · 2026-08-16T15:52:15Z

Summary

Successor to #149. Keep the request-level reasoning_effort thread and mode=verify, but make the advertised check real.

A buyer who asked for a checked judgment on #149 got a rubber stamp: route_and_verify passed the user prompt as thinker_output, so _judge_verifier_output fallback-accepted any worker answer. A rejected verify still returned the worker text as a normal chat.completion. Default auto substring-matched check/review/confirm and 2×-billed everyday English (preview, checkbox, confirmation). Docs claimed per-role Fugu/Conductor/TRINITY allocation that this stack does not implement.

This head:

Fail-closes verify when the verifier report has no explicit accept/reject terms.
Does not serve a rejected worker answer as the public completion (worker text stays on the trace).
Drops ambiguous auto-verify tokens and matches ASCII hints on word boundaries.
Echoes routing_decision and applied-or-dropped reasoning_effort on the chat surface; redacts orchestration.verification.
Reports reasoning_effort.status=dropped on batch HTTP 202.
Omits unset reasoning_effort on stream_chat so older doubles keep working.
Corrects docs/architecture.md to request-level only and leaves issue [Product Gap] Provider-neutral role reasoning-effort profiles with equal-budget ablation #568 open. APA 7th references added for Fugu, Trinity, and Conductor.

Do not merge #149 at e3f7588. Prefer this successor. Independent non-author approval is still required. Per-role reasoning_effort_profile and equal-budget ablation stay #568.

Test plan

python3 tests/test_verify_mode_honesty.py — neutral verify is not fallback-accepted; rejected verify does not serve worker text; preview/checkbox/confirmation/Check the logs. stay route; Verify this answer. stays verify; architecture note is request-level; chat response echoes routing + redacts verification; batch 202 reports dropped effort; unset stream effort does not TypeError.
python3 tests/test_paper_contracts.py
python3 tests/test_adaptive_default_routing.py
python3 tests/test_reasoning_effort_and_verify_mode.py
python3 tests/test_cost_router.py
python3 tests/test_security_hardening.py
python3 tests/test_model_judge.py
python3 tests/test_self_check.py test_conventions.py test_api_contract.py test_admin_contract.py test_product_planning_contract.py

Closes the #149 honesty gap. Does not close #568.

Closes the test-time-compute-allocation gap between what Fugu, Conductor, and TRINITY (docs/architecture.md, arXiv:2512.04695, arXiv:2512.04388) call for and what main actually does: reasoning_effort was accepted at the HTTP edge but silently dropped on the orchestrated route/conduct path, and there was no way to get a single checked judgment without paying for the full four-step conduct() workflow. - ModelClient.chat/stream_chat forward reasoning_effort (OpenAI-compatible minimal/low/medium/high) to the provider payload when set, omitted otherwise -- unaffected for providers/callers that never opt in. - reasoning_effort threads through the whole call chain (server body -> CostRoutingCoordinator.complete -> TaskOrchestrator.run/complete/_dispatch -> route_once/conduct/route_and_verify -> _invoke -> ModelClient.chat), including the plan-generation and model-judge calls, and is folded into the response cache key so a cached low-effort answer can't be served for a high-effort request. Batch-channel requests intentionally drop it today (BatchRequest has no such field) -- documented, not silently wrong. - New mode="verify" (TaskOrchestrator.route_and_verify): one worker call plus one checked verifier judgment, for adjudication-shaped requests ("does B follow from A?") that need a verified verdict without the thinker/worker/verifier/synthesizer workflow's cost. - _client_chat() call-site helper keeps every existing ModelClient-shaped test double/subclass in this repo working unchanged when reasoning_effort is unset (the default) -- no test double needed touching. Tests: tests/test_paper_contracts.py (reasoning_effort reaches every provider call in a conduct() run; omitted by default; verify mode's trace shape and reasoning_effort propagation) and a new tests/test_reasoning_effort_and_verify_mode.py (HTTP-level: verify mode, invalid/valid reasoning_effort validation). Full suite: 307 passed. Does not touch any of the ~20 other open PRs' surface (OpenAI-compat headers, security/session hardening, pricing/routing) -- verified no existing open PR claims reasoning_effort or a partial-conduct mode before starting this.

…s diff This PR's Semgrep check failed on 5 findings, none introduced by this change (line numbers only shifted because earlier edits in this branch added lines above them): - cost_ledger.py:586,605,625 (sqlalchemy-execute-raw-query): already bandit-suppressed (# nosec B608) with the same rationale -- the interpolated pieces are a DB-API placeholder character and fixed internal column-name constants, never request data; actual values always go through the parameterized second argument. Semgrep doesn't read bandit's nosec syntax, so it re-flags what bandit already accepted. Added the matching # nosemgrep suppression alongside the existing nosec comment -- no SQL construction logic changed. - orchestrator.py (unverified-ssl-context, dynamic-urllib-use-detected): same pattern -- both already carry a bandit nosec with an accepted rationale (verify_tls=False is an explicit opt-in dev-only argument, not a default; the request URL is validated by _provider_url()/ _validate_provider() -- https-only, path-injection-safe, private/ loopback/link-local/reserved-IP-rejecting -- before urlopen is ever reached). Added the matching # nosemgrep suppression with the same rationale spelled out for the urllib case. Verified locally: 'semgrep --config auto --severity WARNING --severity ERROR --error' now reports 0 findings on both files (was 5). Full test suite still 307 passed (comment-only change, no behavior touched).

The later quality-cost staging scripts were collected by pytest because stage_quality_cost_policy_test.py matches *_test.py. Importing that module wrote tests/test_quality_cost_adaptive_default.py during collection, which then failed the Full unit suite. The apply workflow also used contents:write (Scorecard Token-Permissions) and regex-patched orchestrator.py into a U+0001 SyntaxError on red-green-verify. Keep the already-landed reasoning_effort + verify mode and adaptive route/verify/conduct dispatch. Ignore scripts/ during collection so helper modules cannot inject tests again. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>

A scripts/*_test.py helper was collected as a test and wrote a failing file into tests/ during import. Keep collect_ignore covering scripts/ and fuzz/, and document why so the Full unit suite cannot pick up staging helpers again. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>

mode=verify no longer fallback-accepts a neutral verifier report or returns a rejected worker answer as a normal completion. Auto verify hints drop ambiguous check/review/confirm tokens and match ASCII terms on word boundaries. The chat surface echoes routing_decision and applied-or-dropped reasoning_effort; batch 202 reports the drop. Architecture notes now say request-level only and leave issue #568 open. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>

The ledger counted only the public completion text, so a two-call verify invoice looked like a single route. Sum worker and verifier outputs (and any multi-step trace) before recording usage. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>

cursor · 2026-08-16T15:59:37Z

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

cursor

Stale comment

Verdict: do not merge #612 at 95393a0. Prefer #618.

#612 closed the #149 fallback-accept and preview/checkbox traps, but a buyer can still get a rubber-stamped worker answer, a silent reasoning_effort, an unredacted stream verification blob, and a surprise verify bill.

Remaining holes on this head

Substring verdicts still accept. pass ⊂ password, good ⊂ looks good, and accepted ⊂ I have not accepted this. Those paths serve the worker text as a normal chat.completion.

run() drops reasoning_effort. /v1/chat/completions therefore never echoes the paid knob. The old honesty test only built a synthetic dict.

SSE leaks raw verification. Sync redacts a Bearer token; the final stream chunk ships it.

Everyday auto-bills remain. Please validate the form., Don't judge me., 확인해주세요, and 평가 부탁 still select verify.

Ledger under-invoices when the verifier output is empty and the worker is long (falls back to the short public rejection).

Conduct still serves the worker on reject.

#618 (aa188cb) requires a first-line or whole-token ACCEPT/REJECT, copies effort through run()/HTTP/SSE, redacts stream verification, invoices every trace step, and keeps rejected worker text off the public answer on verify and conduct.

Independent non-author approval is still required. Do not self-approve. Per-role reasoning_effort_profile stays issue #568 — do not start it on this stack.

Next action: review and land #618, then close #149 and #612 without merging them.

View PR

_{Sent by Cursor Automation: Fix Issues}

cursor

Do not merge this PR. The fail-close slice is real, but #612 at 95393a0 still rubber-stamps and mis-bills a paying caller.

Reproduced on this head:

_judge_verifier_output substring-matches pass / good / accepted. password, looks good, and I have not accepted this all accept and serve the worker text.
TaskOrchestrator.run() drops reasoning_effort, so the HTTP/coordinator path cannot echo applied effort. The honesty test used a synthetic dict and missed this.
chat_completion_chunks does not redact_value verification. A Bearer token on the SSE path leaks; sync chat_completion_response already redacts.
Auto-verify leftovers: validate, judge, 확인, 평가, 검토 still select verify. English confirm was correctly dropped.
Ledger len(outputs) >= 2 skips empty step outputs and ignores step["usage"].
conduct() still serves worker text on reject. Model judge ACCEPT in I DO NOT ACCEPT is a latent accept.

Successor landing vehicle: the follow-up PR from cursor/bc-c9b5dd9c-907a-43f0-a66b-dcbe95e7def0-49b7 (1ec6a76). Review and merge that one. Close this PR after it lands.

Next product gap after that land: issue #568 (per-role reasoning-effort profiles and equal-budget ablation). Do not start #568 on this branch.

_{Sent by Cursor Automation: Fix Issues}

cursor · 2026-08-16T16:06:36Z

    )
+    VERIFICATION_HINTS = (
+        "verify",
+        "validate",


validate, judge, 확인, 평가, and 검토 still auto-select verify. Reproduced: Please validate the form fields., Do not judge the draft yet., 일정 확인해주세요., and 성과 평가 요약만 적어 주세요. all take the two-call path. Keep only explicit adjudication tokens (verify, adjudicate, 검증, 심사).

cursor · 2026-08-16T16:06:37Z

            if budget["exceeded"]:
                raise BudgetExceededError("spend budget exceeded", detail=budget)
-        result = self.complete(messages, mode=mode)
+        result = self.complete(messages, mode=mode, reasoning_effort=reasoning_effort)


run() forwards reasoning_effort into complete() but the persisted HTTP record omits it. CostRoutingCoordinator.complete() therefore cannot echo applied-or-dropped effort on the sync path. Copy result["reasoning_effort"] onto the record.

cursor · 2026-08-16T16:06:37Z

+                )
            answer = outputs[steps[-1].id]
            if not verification["accepted"] and self.policy.verifier_required and last_output("worker"):
                answer = last_output("worker")


Rejected conduct still serves the worker text as the public completion (answer = last_output("worker") / outputs[steps[1].id]). That is the same honesty bug verify mode just closed. Return _rejected_verify_answer(verification) and keep worker text on the trace.

cursor · 2026-08-16T16:06:37Z

+        thinker_output: str,
+        worker_output: str,
+        *,
+        require_explicit_verdict: bool = False,


term in lowered still rubber-stamps password (pass), looks good (good), and I have not accepted this (accepted). Use word boundaries and treat a negated accept token as reject.

cursor · 2026-08-16T16:06:37Z

+            for step in result.get("trace") or []
+            if isinstance(step, dict) and step.get("output")
+        ]
+        if len(outputs) >= 2:


len(outputs) >= 2 skips empty step text and falls back to the public envelope, and it ignores step["usage"].completion_tokens. A two-step verify with an empty verifier report under-invoices; provider reasoning tokens never appear. Sum every step and prefer reported usage.

seonghobae · 2026-08-17T13:43:26Z

Superseded by #622. The current successor explicitly retains this PR’s verify/reasoning-effort honesty work and closes additional buyer-facing defects: word-bounded verdicts, narrower auto-verify routing, SSE redaction/parity, usage-ledger fidelity, conduct rejection parity, and leading ACCEPT/REJECT judge semantics. #622 instructs reviewers to use it instead of #612. No checks, reviews, or approvals transfer.

seonghobae and others added 11 commits August 13, 2026 10:08

chore: stage adaptive routing migration

ec32b3a

ci: apply and verify adaptive routing default

7fcb51c

feat(routing): default to adaptive performance-cost orchestration

da092fc

test: stage adaptive quality-cost policy regressions

b89c26c

feat: stage adaptive quality-cost policy implementation

188693b

ci: apply and verify quality-cost adaptive policy

76af82a

cursor Bot requested a review from seonghobae August 16, 2026 15:52

seonghobae marked this pull request as ready for review August 16, 2026 15:59

seonghobae enabled auto-merge (squash) August 16, 2026 15:59

cursor Bot mentioned this pull request Aug 16, 2026

fix(routing): explicit ACCEPT/REJECT verdicts and honest effort/SSE #618

Closed

4 tasks

cursor Bot commented Aug 16, 2026

View reviewed changes

This was referenced Aug 16, 2026

fix(routing): word-bound verify verdicts and stream redaction #622

Open

fix(routing): withhold rejected conduct answers and incidental accepts #634

Open

seonghobae mentioned this pull request Aug 16, 2026

fix(routing): label omitted-verifier conduct as unchecked #664

Open

3 tasks

cursor Bot mentioned this pull request Aug 16, 2026

test(routing): lock HTTP/SSE echo of unchecked conduct #683

Closed

3 tasks

opencode-agent Bot disabled auto-merge August 16, 2026 18:04

seonghobae enabled auto-merge (squash) August 16, 2026 20:00

opencode-agent Bot disabled auto-merge August 17, 2026 02:23

seonghobae enabled auto-merge (squash) August 17, 2026 07:04

opencode-agent Bot disabled auto-merge August 17, 2026 07:50

seonghobae closed this Aug 17, 2026

This was referenced Aug 17, 2026

feat(routing): request-level reasoning_effort + verify mode #149

Closed

test(routing): lock HTTP/SSE echo of unchecked conduct #743

Open

Navigation Menu

Uh oh!

fix(routing): fail-close verify mode and stop substring auto-bills - #612

fix(routing): fail-close verify mode and stop substring auto-bills#612
cursor[bot] wants to merge 12 commits into
mainfrom
cursor/bc-37be1587-d428-4011-91c9-47f42962e8e7-5667

cursor Bot commented Aug 16, 2026

Uh oh!

cursor Bot commented Aug 16, 2026

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Aug 16, 2026

Uh oh!

cursor Bot Aug 16, 2026

Uh oh!

cursor Bot Aug 16, 2026

Uh oh!

cursor Bot Aug 16, 2026

Uh oh!

cursor Bot Aug 16, 2026

Uh oh!

seonghobae commented Aug 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

cursor Bot commented Aug 16, 2026

Summary

Test plan

Uh oh!

cursor Bot commented Aug 16, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Verdict: do not merge #612 at 95393a0. Prefer #618.

Remaining holes on this head

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

seonghobae commented Aug 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cursor Bot left a comment •

edited

Loading

Verdict: do not merge #612 at `95393a0`. Prefer #618.