Skip to content

test(e2e): cover authorized approval responses - #1669

Closed
benpankow wants to merge 0 commit into
ben/hitl-v2-docs-contractsfrom
ben/hitl-v2-authorized-response-e2e
Closed

test(e2e): cover authorized approval responses#1669
benpankow wants to merge 0 commit into
ben/hitl-v2-docs-contractsfrom
ben/hitl-v2-authorized-response-e2e

Conversation

@benpankow

Copy link
Copy Markdown
Collaborator

Summary

Adds fixture-owned e2e coverage for the responder-authorized HITL lifecycle introduced by this stack.

The deterministic fixture authenticates the eval driver as a fixed user and defines a tool with explicit request and response policies:

approval: {
  request: always(),
  response: ({ response, responder }) =>
    response.decision === "approve" && responder.principalId === "e2e-approval-responder"
      ? { status: "allowed" }
      : { status: "rejected", safeReason: "Unexpected eval responder." },
}

The eval asserts:

input.requested
→ approval.candidate pending
→ approval.settled approved
→ action.result completed

It uses no external OAuth provider or injected credential, so the result is deterministic and self-contained.

This also adds compiler/runtime normalization coverage for the explicit { request, response } approval object; previously only the function shorthand was accepted by runtime normalization even though the public type allowed the object form.

Depends on #1374.

Validation

  • pnpm --filter agent-tools-hitl typecheck
  • pnpm --filter eve typecheck
  • pnpm guard:invariants
  • focused authored-definition unit tests

@benpankow
benpankow force-pushed the ben/hitl-v2-docs-contracts branch from 25a9558 to 3d51048 Compare August 5, 2026 15:27
@vercel

vercel Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
eve-docs Ready Ready Preview Aug 7, 2026 10:47pm
eve-docs-4759 Ready Ready Preview, v0 Aug 7, 2026 10:47pm

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Bundle + Package Summary: apps/fixtures/weather-agent

Key takeaways

  • No notable deltas vs ben/hitl-v2-docs-contracts (25a9558).

Delta vs ben/hitl-v2-docs-contracts (25a9558)

Area Metric Baseline Current Delta
Package Packed tarball 7.64 MB 7.64 MB +363 B ⚠️
Package Unpacked publish size 28.65 MB 28.65 MB +597 B ⚠️
Package Installed footprint 71.98 MB 71.98 MB +597 B ⚠️
Package Published files 3012 3014 +2
Package Installed files 6711 6713 +2
Runtime Unique function payloads 2 2 0
Runtime Total function bytes 16.91 MB 16.91 MB -40 B ✅
Runtime Public routes 13 13 0
Changed function payloads vs ben/hitl-v2-docs-contracts (25a9558) (2)
Function Status Baseline Current Delta Route changes
functions/__server.func changed 8.46 MB 8.46 MB -20 B ✅ none
functions/.well-known/workflow/v1/flow.func changed 8.46 MB 8.46 MB -20 B ✅ none

eve init install

Metric Baseline Current Delta
Installed footprint 110.39 MB 110.39 MB +597 B ⚠️
Installed packages 121 121 0
dependencies 4 4 0
devDependencies 2 2 0
Dependency package bytes 43.06 MB 43.06 MB +597 B ⚠️
devDependency package bytes 5.04 MB 5.04 MB 0 B ➖
Build Metadata
  • Preset: vercel
  • Nitro: nitro@3.0.260610-beta
  • Output directory: apps/fixtures/weather-agent/.vercel/output
  • Build metadata timestamp: 2026-08-05T15:29:50.920Z
  • Route aliases: 13 public, 1 internal (14 total aliases)
  • Vercel routes in config: 16
  • Severity legend: 🔴 dominant/large, 🟠 notable, 🟡 watch, ⚪ small
Package Drill-Down

Package Details

  • Package: eve@0.30.2
  • Package directory: packages/eve
  • Tarball: 7.64 MB (eve-0.30.2.tgz)
  • Unpacked payload: 28.65 MB across 3014 published files
  • Installed footprint: 71.98 MB across 6713 installed files
  • Installed root package: 27.29 MB
  • Installed dependencies: 44.68 MB
  • Runtime dependencies: 2
  • Peer dependencies: 5 (4 optional)

Installed footprint is measured from an isolated temporary npm install of the packed tarball.

Heavy installed dependencies

  • eve: 27.29 MB (37.9%)
  • @rolldown/binding-linux-x64-gnu: 19.35 MB (26.9%)
  • ai: 6.60 MB (9.2%)
  • zod: 5.07 MB (7.0%)
  • undici: 3.50 MB (4.9%)
Publish payload breakdown
Published file size
🔴 dist/src/compiled/shadcn-registry/index.js       [#################.......] 9.68 MB 33.8%
🟠 dist/src/compiled/@photon-ai/chat-adapter-ime... [####....................] 2.42 MB 8.5%
🟠 dist/src/compiled/experimental-ai-sdk-code-mo... [###.....................] 1.51 MB 5.3%
🟡 dist/src/compiled/_chunks/node/undici-DWL_MYm... [#.......................] 502.4 kB 1.8%
🟡 dist/src/compiled/_chunks/workflow/undici-DWL... [#.......................] 502.4 kB 1.8%
🔴 Other published files                            [########################] 14.03 MB 49.0%
Installed footprint breakdown
Installed package size
🔴 eve                             [########################] 27.29 MB 37.9%
🔴 @rolldown/binding-linux-x64-gnu [#################.......] 19.35 MB 26.9%
🔴 ai                              [######..................] 6.60 MB 9.2%
🔴 zod                             [####....................] 5.07 MB 7.0%
🟠 undici                          [###.....................] 3.50 MB 4.9%
🟠 nitro                           [##......................] 2.41 MB 3.3%
🔴 Other installed packages        [#######.................] 7.76 MB 10.8%
Runtime dependencies (2)
Package Range Notes
nitro 3.0.260610-beta
undici 8.9.0
Peer dependencies (5)
Package Range Notes
@opentelemetry/api ^1.0.0 optional peer
ai catalog:
braintrust ^3.0.0 optional peer
just-bash ^3.0.0 optional peer
microsandbox ^0.5.0 optional peer
eve init install drill-down

eve init install details

  • Command: eve init my-agent
  • Package manager: npm
  • Installed footprint: 110.39 MB across 8589 installed files
  • Installed packages: 121 total (115 transitive-only)
  • dependencies: 4 direct packages totaling 43.06 MB
  • devDependencies: 2 direct packages totaling 5.04 MB
  • Other transitive package files: 62.29 MB

Installed footprint is measured from an isolated temporary eve init my-agent using the current packed eve tarball.

Heavy installed dependencies

  • @typescript/typescript-linux-x64: 27.95 MB (25.3%)
  • eve: 27.29 MB (24.7%)
  • @rolldown/binding-linux-x64-gnu: 19.35 MB (17.5%)
  • zod: 9.02 MB (8.2%)
  • ai: 6.60 MB (6.0%)
Installed footprint breakdown
Installed package size
🔴 @typescript/typescript-linux-x64 [########################] 27.95 MB 25.3%
🔴 eve                              [#######################.] 27.29 MB 24.7%
🔴 @rolldown/binding-linux-x64-gnu  [#################.......] 19.35 MB 17.5%
🔴 zod                              [########................] 9.02 MB 8.2%
🔴 ai                               [######..................] 6.60 MB 6.0%
🟠 undici                           [###.....................] 3.50 MB 3.2%
🔴 Other installed packages         [##############..........] 16.67 MB 15.1%
dependencies (4)
Package Range Installed size Share
@vercel/connect 0.4.3 141.0 kB 0.1%
ai ^7.0.38 6.60 MB 6.0%
eve file:eve-0.30.2.tgz 27.29 MB 24.7%
zod 4.4.3 9.02 MB 8.2%
devDependencies (2)
Package Range Installed size Share
@types/node 24.x 2.54 MB 2.3%
typescript 7.0.2 2.50 MB 2.3%
Function Drill-Down

Payload Size Graph

Unique function payload size and share of total
🔴 functions/.well-known/workflow/v1/flow.func     [########################] 8.46 MB 50.0%
🔴 functions/__server.func                         [########################] 8.46 MB 50.0%

Top Function Payloads

🟠 functions/.well-known/workflow/v1/flow.func • 1 public route • 8.46 MB
Metric Value
Public routes /.well-known/workflow/v1/flow
Runtime nodejs24.x
Handler index.mjs
Payload 8.46 MB
Function files 8.46 MB across 43 files
Traced dependencies 0 B
Signal 🟠 Bundled file index.mjs is 2.28 MB (26.9%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                       [#######################.] 2.28 MB 26.9%
🟠 _chunks/runtime-artifacts.mjs   [################........] 1.61 MB 19.1%
🟡 _libs/undici.mjs                [##########..............] 980.5 kB 11.6%
🟡 _chunks/sandbox.mjs             [########................] 769.0 kB 9.1%
🟡 _libs/@ai-sdk/gateway+[...].mjs [####....................] 432.8 kB 5.1%
🟠 Other bundled files             [########################] 2.38 MB 28.2%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x",
  "maxDuration": "max",
  "experimentalTriggers": [
    {
      "type": "queue/v2beta",
      "topic": "__eve776561746865722d6167656e74_wkf_workflow_*",
      "consumer": "default",
      "retryAfterSeconds": 5,
      "initialDelaySeconds": 0
    }
  ],
  "environment": {
    "WORKFLOW_PRECONDITION_GUARD": "1"
  }
}

🟠 functions/__server.func • 12 public routes, 1 internal alias • 8.46 MB
Metric Value
Public routes /
/eve/v1/callback/[token]
/eve/v1/connections/[name]/callback/[token]
/eve/v1/health
/eve/v1/info
/eve/v1/session
/eve/v1/session/[sessionId]
/eve/v1/session/[sessionId]/cancel
/eve/v1/session/[sessionId]/stream
/eve/v1/session/clear
/eve/v1/session/compact
/eve/v1/session/reset
Internal aliases /__server
Runtime nodejs24.x
Handler index.mjs
Payload 8.46 MB
Function files 8.46 MB across 43 files
Traced dependencies 0 B
Signal 🟠 Bundled file index.mjs is 2.28 MB (26.9%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                       [#######################.] 2.28 MB 26.9%
🟠 _chunks/runtime-artifacts.mjs   [################........] 1.61 MB 19.1%
🟡 _libs/undici.mjs                [##########..............] 980.5 kB 11.6%
🟡 _chunks/sandbox.mjs             [########................] 769.0 kB 9.1%
🟡 _libs/@ai-sdk/gateway+[...].mjs [####....................] 432.8 kB 5.1%
🟠 Other bundled files             [########################] 2.38 MB 28.2%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x"
}

Build Timing: e2e/fixtures/agent-tools-sandbox

This is an informational timing measurement inside eve build, from preflight through publication. Output-size measurement and profile writing are excluded.

Build mode: deployable Vercel build with sandbox template prewarm included.

  • Build pipeline: 1.80 s -> 1.71 s (-93.3 ms) vs ben/hitl-v2-docs-contracts (25a9558).
  • Timing is informational: shared GitHub runners are too variable for a hard timing budget.
Detailed phase timings vs `ben/hitl-v2-docs-contracts (25a9558)`
Phase Baseline Current Delta
extension.check 7.2 ms 9.1 ms +1.9 ms
project.resolve 1.4 ms 3.5 ms +2.1 ms
workspace.create 14.2 ms 0.9 ms -13.3 ms
host.prepare 143.2 ms 146.2 ms +3.0 ms
vercel.service-prefix.resolve 1.5 ms 1.7 ms +0.2 ms
nitro.create 179.5 ms 146.9 ms -32.6 ms
sandbox.prewarm 267.5 ms 245.4 ms -22.1 ms
nitro.cache.prepare 0.2 ms 0.2 ms 0.0 ms
nitro.prepare 0.6 ms 0.6 ms 0.0 ms
nitro.public-assets 0.6 ms 0.6 ms 0.0 ms
nitro.prerender 1.2 ms 0.4 ms -0.8 ms
nitro.bundle 1.13 s 1.10 s -33.8 ms
nitro.cache.write 0.3 ms 0.3 ms 0.0 ms
vercel.workflow-function.materialize 46.6 ms 37.4 ms -9.2 ms
agent-summary.emit 0.5 ms 0.5 ms 0.0 ms
nitro.close 0.1 ms 0.1 ms 0.0 ms
output.publish 2.7 ms 14.0 ms +11.3 ms
workspace.remove 1.4 ms 1.5 ms +0.1 ms

@vercel vercel Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Suggestions:

  1. Object-form connection approval ({ request, response? }) is silently dropped at runtime, removing the human-in-the-loop approval gate.
  1. Runtime tool resolver rejects object-form { request, response } approval, throwing on any tool that declares approval as an object instead of a function.

Fix on Vercel

@benpankow
benpankow force-pushed the ben/hitl-v2-authorized-response-e2e branch from 74ba16b to 1704ded Compare August 5, 2026 16:21
@vercel
vercel Bot temporarily deployed to Preview – eve-docs-4759 August 5, 2026 16:21 Inactive
@benpankow
benpankow force-pushed the ben/hitl-v2-docs-contracts branch from 3d51048 to 46f6df1 Compare August 5, 2026 19:19
@benpankow
benpankow force-pushed the ben/hitl-v2-authorized-response-e2e branch from 1704ded to 03adc4c Compare August 5, 2026 19:19
@benpankow
benpankow force-pushed the ben/hitl-v2-docs-contracts branch from 46f6df1 to ec4bf43 Compare August 5, 2026 21:08
@benpankow
benpankow force-pushed the ben/hitl-v2-authorized-response-e2e branch from 03adc4c to ea0dec5 Compare August 5, 2026 21:08
@benpankow
benpankow force-pushed the ben/hitl-v2-docs-contracts branch from ec4bf43 to e6e113b Compare August 7, 2026 20:18
@benpankow
benpankow force-pushed the ben/hitl-v2-authorized-response-e2e branch from ea0dec5 to 7b47bb3 Compare August 7, 2026 20:19
@benpankow
benpankow force-pushed the ben/hitl-v2-docs-contracts branch from 2cd1ee9 to 56906e4 Compare August 7, 2026 20:31
@benpankow
benpankow force-pushed the ben/hitl-v2-authorized-response-e2e branch from 7b47bb3 to 6fc23a9 Compare August 7, 2026 20:31
@benpankow
benpankow force-pushed the ben/hitl-v2-docs-contracts branch from 56906e4 to d8c5cb2 Compare August 7, 2026 20:41
@benpankow
benpankow force-pushed the ben/hitl-v2-authorized-response-e2e branch from 6fc23a9 to a2443e6 Compare August 7, 2026 20:42
@benpankow
benpankow force-pushed the ben/hitl-v2-docs-contracts branch from d8c5cb2 to 13031e7 Compare August 7, 2026 20:46
@benpankow
benpankow force-pushed the ben/hitl-v2-authorized-response-e2e branch from a2443e6 to 2e8096d Compare August 7, 2026 20:46
@benpankow
benpankow force-pushed the ben/hitl-v2-docs-contracts branch from 13031e7 to 89d4cc0 Compare August 7, 2026 20:56
@benpankow
benpankow force-pushed the ben/hitl-v2-authorized-response-e2e branch from 2e8096d to 1896d8a Compare August 7, 2026 20:56
@benpankow
benpankow force-pushed the ben/hitl-v2-docs-contracts branch from 89d4cc0 to 76e9e61 Compare August 7, 2026 21:05
@benpankow
benpankow force-pushed the ben/hitl-v2-authorized-response-e2e branch from 1896d8a to 937dbb7 Compare August 7, 2026 21:06
@benpankow
benpankow force-pushed the ben/hitl-v2-docs-contracts branch from 76e9e61 to bc592c7 Compare August 7, 2026 22:37
@benpankow
benpankow force-pushed the ben/hitl-v2-authorized-response-e2e branch from 937dbb7 to 235be6e Compare August 7, 2026 22:37
@benpankow
benpankow force-pushed the ben/hitl-v2-docs-contracts branch from bc592c7 to d079c62 Compare August 7, 2026 22:46
@benpankow
benpankow force-pushed the ben/hitl-v2-authorized-response-e2e branch from 235be6e to 1c355b8 Compare August 7, 2026 22:46
@benpankow
benpankow force-pushed the ben/hitl-v2-docs-contracts branch from d079c62 to cf74bb9 Compare August 7, 2026 22:57
@benpankow benpankow closed this Aug 7, 2026
@benpankow
benpankow force-pushed the ben/hitl-v2-authorized-response-e2e branch from 1c355b8 to cf74bb9 Compare August 7, 2026 22:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant