fix(routing): explicit ACCEPT/REJECT verdicts and honest effort/SSE by cursor[bot] · Pull Request #618 · ContextualWisdomLab/contextual-orchestrator

cursor · 2026-08-16T16:02:06Z

Summary

Successor to #612. Keep the request-level reasoning_effort thread and mode=verify, but close the remaining rubber-stamp and billing holes that #612 left green.

A buyer who asked for a checked judgment on #612 at 95393a0 could still get the worker answer: password matched pass, looks good matched good, and I have not accepted this matched accepted. TaskOrchestrator.run() dropped reasoning_effort, so /v1/chat/completions never echoed the paid knob. Streaming verify shipped the raw verification blob (Bearer tokens included). Everyday validate / judge / 확인해주세요 / 평가 부탁 still auto-billed verify. An empty verifier plus a long worker invoiced only the short rejection envelope. Conduct still served the worker text on reject.

This head:

Accepts verify/conduct only on an explicit first-line or whole-token ACCEPT/REJECT (negated accepts fail closed).
Does not serve a rejected worker answer as the public completion on verify or conduct.
Drops everyday auto-verify tokens (validate, judge, 확인, 평가) and keeps verify / adjudicate / 검증 / 심사.
Copies reasoning_effort through run() so HTTP and SSE echo applied-or-dropped effort.
Redacts orchestration.verification on the final SSE chunk and echoes routing + effort there.
Invoices every trace step, including an empty verifier plus a long worker.
Leaves issue [Product Gap] Provider-neutral role reasoning-effort profiles with equal-budget ablation #568 open for per-role reasoning_effort_profile and equal-budget ablation.

Do not merge #149 at e3f7588 or #612 at 95393a0. Prefer this successor. Independent non-author approval is still required.

Test plan

python3 tests/test_verify_mode_honesty.py — password / looks good / not accepted fail closed; validate/judge/확인해주세요/평가 부탁 stay route; persisted run() echoes applied effort; SSE redacts Bearer; empty-verifier ledger counts the worker; conduct reject does not serve worker text.
python3 tests/test_reasoning_effort_and_verify_mode.py — live /v1/chat/completions echoes orchestration.reasoning_effort.status=applied.
python3 tests/test_model_judge.py — I DO NOT ACCEPT rejects; rejected conduct does not serve worker text.
python3 tests/test_paper_contracts.py test_adaptive_default_routing.py test_generated_workflow.py test_streaming.py test_cost_router.py test_security_hardening.py test_self_check.py test_conventions.py test_api_contract.py

Closes the leftover #612 honesty gap. Does not close #568.

Closes the test-time-compute-allocation gap between what Fugu, Conductor, and TRINITY (docs/architecture.md, arXiv:2512.04695, arXiv:2512.04388) call for and what main actually does: reasoning_effort was accepted at the HTTP edge but silently dropped on the orchestrated route/conduct path, and there was no way to get a single checked judgment without paying for the full four-step conduct() workflow. - ModelClient.chat/stream_chat forward reasoning_effort (OpenAI-compatible minimal/low/medium/high) to the provider payload when set, omitted otherwise -- unaffected for providers/callers that never opt in. - reasoning_effort threads through the whole call chain (server body -> CostRoutingCoordinator.complete -> TaskOrchestrator.run/complete/_dispatch -> route_once/conduct/route_and_verify -> _invoke -> ModelClient.chat), including the plan-generation and model-judge calls, and is folded into the response cache key so a cached low-effort answer can't be served for a high-effort request. Batch-channel requests intentionally drop it today (BatchRequest has no such field) -- documented, not silently wrong. - New mode="verify" (TaskOrchestrator.route_and_verify): one worker call plus one checked verifier judgment, for adjudication-shaped requests ("does B follow from A?") that need a verified verdict without the thinker/worker/verifier/synthesizer workflow's cost. - _client_chat() call-site helper keeps every existing ModelClient-shaped test double/subclass in this repo working unchanged when reasoning_effort is unset (the default) -- no test double needed touching. Tests: tests/test_paper_contracts.py (reasoning_effort reaches every provider call in a conduct() run; omitted by default; verify mode's trace shape and reasoning_effort propagation) and a new tests/test_reasoning_effort_and_verify_mode.py (HTTP-level: verify mode, invalid/valid reasoning_effort validation). Full suite: 307 passed. Does not touch any of the ~20 other open PRs' surface (OpenAI-compat headers, security/session hardening, pricing/routing) -- verified no existing open PR claims reasoning_effort or a partial-conduct mode before starting this.

…s diff This PR's Semgrep check failed on 5 findings, none introduced by this change (line numbers only shifted because earlier edits in this branch added lines above them): - cost_ledger.py:586,605,625 (sqlalchemy-execute-raw-query): already bandit-suppressed (# nosec B608) with the same rationale -- the interpolated pieces are a DB-API placeholder character and fixed internal column-name constants, never request data; actual values always go through the parameterized second argument. Semgrep doesn't read bandit's nosec syntax, so it re-flags what bandit already accepted. Added the matching # nosemgrep suppression alongside the existing nosec comment -- no SQL construction logic changed. - orchestrator.py (unverified-ssl-context, dynamic-urllib-use-detected): same pattern -- both already carry a bandit nosec with an accepted rationale (verify_tls=False is an explicit opt-in dev-only argument, not a default; the request URL is validated by _provider_url()/ _validate_provider() -- https-only, path-injection-safe, private/ loopback/link-local/reserved-IP-rejecting -- before urlopen is ever reached). Added the matching # nosemgrep suppression with the same rationale spelled out for the urllib case. Verified locally: 'semgrep --config auto --severity WARNING --severity ERROR --error' now reports 0 findings on both files (was 5). Full test suite still 307 passed (comment-only change, no behavior touched).

The later quality-cost staging scripts were collected by pytest because stage_quality_cost_policy_test.py matches *_test.py. Importing that module wrote tests/test_quality_cost_adaptive_default.py during collection, which then failed the Full unit suite. The apply workflow also used contents:write (Scorecard Token-Permissions) and regex-patched orchestrator.py into a U+0001 SyntaxError on red-green-verify. Keep the already-landed reasoning_effort + verify mode and adaptive route/verify/conduct dispatch. Ignore scripts/ during collection so helper modules cannot inject tests again. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>

A scripts/*_test.py helper was collected as a test and wrote a failing file into tests/ during import. Keep collect_ignore covering scripts/ and fuzz/, and document why so the Full unit suite cannot pick up staging helpers again. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>

mode=verify no longer fallback-accepts a neutral verifier report or returns a rejected worker answer as a normal completion. Auto verify hints drop ambiguous check/review/confirm tokens and match ASCII terms on word boundaries. The chat surface echoes routing_decision and applied-or-dropped reasoning_effort; batch 202 reports the drop. Architecture notes now say request-level only and leave issue #568 open. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>

The ledger counted only the public completion text, so a two-call verify invoice looked like a single route. Sum worker and verifier outputs (and any multi-step trace) before recording usage. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>

mode=verify still accepted password/looks good/not accepted via substring terms, run() dropped reasoning_effort so HTTP never echoed it, and SSE shipped raw verification. Invoice every trace step, including an empty verifier plus a long worker. Conduct no longer serves the worker answer on reject. Everyday validate/judge/확인/평가 stay on the single-worker route. Per-role profiles remain issue #568. Co-authored-by: Seongho Bae <seonghobae@users.noreply.github.com>

cursor

Stale comment

This is the verify/conduct honesty landing vehicle. Keep it.

#612 at 95393a0 still rubber-stamps substring accept terms, drops reasoning_effort on run(), leaks SSE verification, auto-bills everyday validate/judge/확인/평가, under-invoices an empty verifier step, and serves worker text on rejected conduct. Do not merge #612.

#622 1ec6a76 is a parallel residual slice from a later #612 ready_for_review run. Prefer this PR. Close #612 and #622 after this lands.

Do not start issue #568 here. Independent reviewer approval is still required. Do not self-approve.

View PR

_{Sent by Cursor Automation: Fix Issues}

cursor · 2026-08-16T16:11:06Z

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

cursor

This slice is no longer the honesty landing vehicle.

#618 at aa188cb still rubber-stamps incidental body accepted (the password was accepted) and the model-judge equivalent (THE PASSWORD WAS ACCEPTED BY THE IDP). Rejected conduct still serves the last-step synthesizer, so a synthesizer that repeats the worker secret becomes the public completion. HTTP/SSE never echo answer_status.

Prefer successor #634 62cb671. Close this PR after that lands. Do not merge #612 or #622. Do not start issue #568 here. Independent non-author approval is still required. Do not self-approve.

_{Sent by Cursor Automation: Fix Issues}

cursor · 2026-08-16T16:17:07Z

+                verification = self._model_judge_verification(
+                    task, verification, reasoning_effort=reasoning_effort
+                )
            answer = outputs[steps[-1].id]


Rejected conduct still assigns the last-step synthesizer here. A synthesizer that repeats the worker secret becomes the public completion (finish_reason=stop). Successor #634 serves the same rejection envelope used by verify and keeps worker/synthesizer text on the trace.

cursor · 2026-08-16T16:17:07Z

+        if re.search(r"(?<![\w])REJECT(?:ED)?(?![\w])", upper) and not re.search(r"(?<![\w])ACCEPT", upper):
+            return {"accepted": False, "reason": "model judge rejected the verifier report",
+                    "verifier_output": verifier_output, "judge": "model"}
+        if re.search(r"(?<![\w])ACCEPT(?:ED)?(?![\w])", upper) and not re.search(r"(?<![\w])REJECT", upper):


A first-token miss still accepts any whole-token ACCEPT/ACCEPTED in the judge reply. THE PASSWORD WAS ACCEPTED BY THE IDP therefore overrides a fail-closed term verdict. Successor #634 keeps first-token ACCEPT only and leaves incidental body text on the term fallback.

cursor · 2026-08-16T16:17:07Z

+                "reason": "verifier negated an accept verdict",
+                "verifier_output": verifier_output,
+            }
+        if any(


Body-level whole-token accepted still accepts before the explicit-verdict fail-closed path. The worker correctly described how the password was accepted by the IdP is therefore an accept. Successor #634 requires a first-line or whole-report ACCEPT when require_explicit_verdict=True.

seonghobae · 2026-08-16T18:35:22Z

Deferring to tip #691 for product-gate focus (Full unit + Semgrep). Reopen if this delta is not on tip substrate.

seonghobae and others added 13 commits August 13, 2026 10:08

chore: stage adaptive routing migration

ec32b3a

ci: apply and verify adaptive routing default

7fcb51c

feat(routing): default to adaptive performance-cost orchestration

da092fc

test: stage adaptive quality-cost policy regressions

b89c26c

feat: stage adaptive quality-cost policy implementation

188693b

ci: apply and verify quality-cost adaptive policy

76af82a

cursor Bot requested a review from seonghobae August 16, 2026 16:02

This was referenced Aug 16, 2026

fix(security): resolve gateway Bearer tokens from the credential KV #621

Closed

fix(routing): fail-close verify mode and stop substring auto-bills #612

Closed

fix(routing): word-bound verify verdicts and stream redaction #622

Open

cursor Bot commented Aug 16, 2026

View reviewed changes

seonghobae marked this pull request as ready for review August 16, 2026 16:11

seonghobae enabled auto-merge (squash) August 16, 2026 16:11

cursor Bot mentioned this pull request Aug 16, 2026

fix(routing): withhold rejected conduct answers and incidental accepts #634

Open

4 tasks

cursor Bot commented Aug 16, 2026

View reviewed changes

This was referenced Aug 16, 2026

fix(api): mock SSE tool_calls and streamed route sampling #620

Closed

fix(api): treat padded tool_choice none as omit on mock tools #639

Open

seonghobae mentioned this pull request Aug 16, 2026

fix(routing): label omitted-verifier conduct as unchecked #664

Open

3 tasks

cursor Bot mentioned this pull request Aug 16, 2026

test(routing): lock HTTP/SSE echo of unchecked conduct #683

Closed

3 tasks

opencode-agent Bot disabled auto-merge August 16, 2026 18:28

seonghobae closed this Aug 16, 2026

Navigation Menu

Uh oh!

fix(routing): explicit ACCEPT/REJECT verdicts and honest effort/SSE - #618

fix(routing): explicit ACCEPT/REJECT verdicts and honest effort/SSE#618
cursor[bot] wants to merge 13 commits into
mainfrom
cursor/bc-952b6144-5f0f-4b68-bf64-99742bfd4d5b-f000

cursor Bot commented Aug 16, 2026

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

cursor Bot commented Aug 16, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Aug 16, 2026

Uh oh!

cursor Bot Aug 16, 2026

Uh oh!

cursor Bot Aug 16, 2026

Uh oh!

seonghobae commented Aug 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

cursor Bot commented Aug 16, 2026

Summary

Test plan

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot commented Aug 16, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 16, 2026

Choose a reason for hiding this comment

Uh oh!

seonghobae commented Aug 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cursor Bot left a comment •

edited

Loading