Skip to content

test: add opt-in AgentRuntime tool-stream boundary evaluation - #1298

Draft
canblmz1 wants to merge 1 commit into
codexu:devfrom
canblmz1:test/runtime-response-boundary-20261019
Draft

canblmz1 wants to merge 1 commit into
codexu:devfrom
canblmz1:test/runtime-response-boundary-20261019

Conversation

@canblmz1

Copy link
Copy Markdown

This draft adds an opt-in, offline differential evaluation of the checkout's actual AgentRuntime using native chunks and an optional PSJ preflight. It is a concrete development-dependency trial: I maintain prefix-safe-json and would like to learn whether retaining this comparison helps NoteGen's normal provider/runtime upgrade checks.

Run with installed development dependencies on Node 22+:

pnpm test:agent-boundary

The proposal adds one exact development dependency (prefix-safe-json@0.6.1), a command, a test and scope documentation. The pnpm lock change is 11 added lines; its existing Ajv 8.20.0 dependency is reused. Application source, production dependencies and CI workflows are unchanged.

The 24 cases use the unchanged runtime, context manager, permission engine, retry manager, queue and recorder. Healthy one/two-call controls verify exact inputs, original result IDs in the next request and the consumed final answer. Existing malformed-JSON, read-error and user-stop guards require zero callbacks. Other native cases emit diagnostic observations; a native refusal is accepted instead of requiring today's permissive behavior to persist.

The optional PSJ policy refuses closed JSON with length/content_filter, missing finish evidence, late provider-error chunks, duplicate original IDs, schema-invalid input and a malformed sibling before any callback. Whole-batch/schema refusal is explicit. Authority is consumed once, matching original ID, exact name and exact input in the verified sequential fixture.

Provider/application utilities, prompt, hooks, RAG, token count and store/service access are explicit stubs. Unreviewed module imports fail before evaluation; unexpected fetch/MCP/store/skill/image access throws. The only registered read tool appends input to memory. No Rust SSE decoder, Tauri host, raw HTTP bytes, real MCP/model/tool, note mutation, approval workflow, argument repair or desktop/mobile UI is tested. The finite preflight is an evaluation helper, not a production transport adapter or general concurrency/approval binding.

Validation on Windows / Node 22.23.2, using pnpm 9.15.9:

  • Frozen installation with lifecycle scripts disabled.
  • All 24 new cases and all 5 existing native helper cases pass.
  • Full tsc --noEmit --incremental false, test-file ESLint, plugin protocol check and diff checks pass.
  • A temporary native terminal/error refusal also passes all 24 cases; the runtime was restored to its exact original bytes afterward. No native patch is proposed here.

The Linux / Node 24 fork verification also passes frozen installation, all 24 new cases, all 5 existing helper cases and the plugin protocol check. It asserts the exact reviewed source SHA before running: 7ff5fcd4f0eab698199e0194da8f5edd61a93037. Its manual workflow lives only in my fork and is not part of this PR.

The earlier pinned-source standalone recipe records the current baseline separately. A native terminal/error check remains a valid fix; PSJ is optional. The useful feedback is whether this test dependency belongs in your development workflow or creates more maintenance than it saves. This is a source-level test proposal, not a claimed run of the latest desktop binary.

Signed-off-by: Can <116688414+canblmz1@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant