Allow `eve eval --url` to authenticate to a protected deployment from ambient environment variables

[rrcobb]

What problem are you trying to solve?

eve eval --url <deployment> can only authenticate to a Vercel deployment with Deployment Protection enabled if the machine has an interactively logged-in Vercel CLI. Supplying VERCEL_OIDC_TOKEN and VERCEL_AUTOMATION_BYPASS_SECRET through the environment does not work. The credential gate authorizes only after a successful vercel api subprocess call. Without one, it silently sends no credentials at all.

Tracing a remote target: createEvalClient({kind: "remote", url}, {workspaceRoot}) in dist/src/evals/cli/eval-client.js calls resolveVerifiedRemoteDevelopmentClient in dist/src/setup/verified-remote-client.js, which calls resolveVercelDeployment in dist/src/setup/vercel-deployment.js. That last function spawns the Vercel CLI (vercel api /v13/deployments/<host> --raw). Only if it returns kind: "resolved" is gate.authorize(...) called.

createDevelopmentCredentialGate in dist/src/services/dev-client/credential-gate.js starts in {kind: "anonymous"}, and both resolvers short-circuit on that state:

resolveBypassHeaders: async () => {
  if (r.kind === `anonymous`) return {};
  const t = process.env.VERCEL_AUTOMATION_BYPASS_SECRET?.trim();
  return t ? { [VERCEL_PROTECTION_BYPASS_HEADER]: t } : {};
},
resolveToken: async () => {
  const e = r;
  if (e.kind === `anonymous`) return ``;
  ...
}

So when the vercel api call fails, every request goes out unauthenticated regardless of what is in the environment, and a protected deployment rejects it.

Running evals against a deployed agent seems like the intent of the --url flag. The natural places to do that are ones without an interactive CLI session: CI jobs, containers, and ephemeral runners.

The available workaround is to supply a VERCEL_TOKEN, which does work. However, it trades a narrow credential for a broad one: a VERCEL_TOKEN is team-scoped and can read and mutate resources far beyond the deployment under test. For a runner executing eval files from an untrusted or semi-trusted source, that is a worse posture than the scoped credentials.

The header plumbing is already built. Once the gate is authorized, resolveRemoteDevelopmentClientOptions produces a per-request OIDC resolver plus the bypass header — everything needed to reach a protected deployment:

keys:      [ "auth", "headers", "host", "redirect" ]
redirect:  manual
auth.vercelOidc.token() => <the OIDC token>
headers()                => {"x-vercel-protection-bypass":"<the bypass secret>"}

Similarly the async getVercelOidcToken already prefers an ambient token, reading process.env.VERCEL_OIDC_TOKEN and refreshing through the CLI only when it is missing or expired. So the OIDC half already honors the environment; it is the gate authorization in front of it that does not.

The one documented escape hatch, EVE_EVAL_AUTH_TOKEN, sets auth: {bearer} and returns early in resolveEvalClientOptions, so it never produces an x-vercel-protection-bypass header. It solves a different problem — a static bearer the agent itself checks — not Deployment Protection.

Reproduction

Any Vercel-deployed eve agent with Deployment Protection enabled, on a machine with the Vercel CLI installed but not logged in (vercel whoami fails):

export VERCEL_OIDC_TOKEN="<a valid, unexpired OIDC token>"
export VERCEL_AUTOMATION_BYPASS_SECRET="<the project's bypass secret>"
eve eval --url https://<protected-deployment> --json

Expected: the evals run against the deployment. Actual: requests are unauthenticated and the deployment rejects them.

The gate step can also be isolated without a deployment by calling resolveVercelDeployment({workspaceRoot, host}) directly: with no logged-in CLI it returns { kind: "forbidden" }, and any non-resolved kind means gate.authorize is never called, so both credential resolvers return empty.

Proposed solution

Let the credential gate be authorized directly from the environment, skipping the vercel api probe, when the caller opts in and credentials are present. In resolveVerifiedRemoteDevelopmentClient: if an opt-in signal is set (maybe an environment variable such as EVE_EVAL_TRUST_AMBIENT_VERCEL_CREDS=1, or an explicit option on createEvalClient) and VERCEL_OIDC_TOKEN and VERCEL_AUTOMATION_BYPASS_SECRET are both present, authorize the gate from those values rather than from resolveVercelDeployment.

The gate's origin check needs ownerId and projectId, which is what the probe fetches — but the OIDC token payload already carries owner_id and project_id claims, and validateDevelopmentOidcToken already decodes and validates exactly those. So the check can be satisfied from the token itself without the network call.

Keeping it opt-in preserves today's default behavior of verifying against the Vercel API first, and confines the change to verified-remote-client.js.

Alternatives considered

Supplying a VERCEL_TOKEN so the spawned vercel api call succeeds. This works — runVercel's buildSpawnEnv passes process.env through to the spawned CLI, which consumes the variable as --token. It can be confirmed with a deliberately malformed value:

VERCEL_TOKEN=invalid-token-probe node repro.mjs
→ kind: "failed", stderr: 'Error: You defined "--token", but its contents are invalid. Must not contain: "-"'

As noted, this puts a team-scoped API credential in a process that executes eval files, which is a materially broader grant than the run requires.

Using EVE_EVAL_AUTH_TOKEN. It does not emit an x-vercel-protection-bypass header, so it does not address Deployment Protection.

Not using the CLI for this at all, and constructing a client directly against eve/client. This is what we do today, and it works, but it means reimplementing what the CLI already does correctly.

[e0-gh-vercel-connect]

classified #1707 as a p2 evals enhancement. it stays open and is labeled p2, evals, and enhancement; no similar existing issue was found.