Authoring agent/instrumentation.ts removes the agent OTel instrumentation, so exported traces contain no agent.* spans
Summary
The eve.agent span family — agent.session, agent.turn, agent.step, agent.action,
agent.turn.terminal, plus the ai.streamText.doStream / ai.toolCall spans that carry the
prompt and tool bodies — is created by createAgentOtelInstrumentation(). That function has
exactly one caller, installLocalInstrumentationRuntime(), and that runtime is installed only
when the app has no agent/instrumentation.ts.
So the two paths are mutually exclusive. Author instrumentation.ts to export OTLP to your own
backend — the documented way to get traces off the box — and you silently opt out of the entire
agent.* vocabulary. What your collector receives instead is Workflow plumbing, one ai.eve.turn
span per turn, and the AI SDK's gen_ai.* spans.
The AI SDK spans do carry prompts and tool payloads, so this is not "telemetry is empty". It is
narrower and easier to miss: the span vocabulary you see in eve traces is not the vocabulary
your backend receives, and one identifier — the root session id — has no equivalent on the
exported side at all, so subagent → root-session lineage cannot be reconstructed downstream.
docs/guides/instrumentation.md does say that authoring instrumentation.ts replaces local
recording, so an empty eve traces is expected. It does not say the agent instrumentation goes
with it — its surfaces table describes the OpenTelemetry export row as "Where agent and AI spans
are exported" (guides/instrumentation.md), which is what we built against.
This is not a 0.30.x regression. We first assumed it was, and it is not: the gate below is
byte-identical in every release we unpacked, 0.29.5 through 0.30.6.
The two files
dist/src/internal/nitro/host/create-application-nitro.js, inside
createDevelopmentApplicationNitro (minified; the gate is the whole story):
e.compiledArtifacts.instrumentationPluginPath === void 0 &&
a.unshift(resolvePackageSourceFilePath(`src/internal/nitro/host/local-tracing-runtime-plugin.ts`))
compiledArtifacts.instrumentationPluginPath is set whenever resolveInstrumentationModule()
finds agent/instrumentation.{ts,mts,js,mjs} — file existence alone, before any default export is
inspected.
dist/src/tracing/local-instrumentation-runtime.js is the only place the agent instrumentation is
constructed, and it does so in the same function that installs the local spool processor:
registerOTel({ /* … */ spanProcessors: [i] }); // i wraps LocalTraceSpanProcessor
// …
let c = createAgentOtelInstrumentation({
captureContent: process.env.EVE_TRACES_CONTENT !== `off`,
frameworkVersion: t.frameworkVersion,
stateStore: new ContextAgentTraceStateStore,
tracer: trace.getTracer(`eve.agent`, t.frameworkVersion),
});
return registerInstrumentationRuntime({ /* … */ hooks: createInstrumentationHooks([c.hook, l]), … });Those are one unit today: a span processor that writes .eve/traces/v1 and an instrumentation
that produces the agent.* spans. Only the first is dev-local by nature. The second is the part
an external backend wants.
Three static facts follow, all checkable in a published tarball:
createAgentOtelInstrumentation appears in exactly two non-vendored modules: its own and
tracing/local-instrumentation-runtime.js.registerInstrumentationRuntime( — the only way a runtime becomes visible to the harness — is
called from exactly one module, the same one.- No specifier in eve's
exports map resolves into dist/src/tracing/, and eve/instrumentation
exports only defineInstrumentation and isChannel, so an app cannot install the agent
instrumentation itself.
From an agent directory with eve 0.30.6 installed:
$ grep -o 'instrumentationPluginPath===void 0&&[^;]\{0,90\}' \
node_modules/eve/dist/src/internal/nitro/host/create-application-nitro.js
instrumentationPluginPath===void 0&&a.unshift(resolvePackageSourceFilePath(`src/internal/nitro/host/local-tracing-runtime-plug
$ grep -rl createAgentOtelInstrumentation node_modules/eve/dist/src | grep -v /compiled/
node_modules/eve/dist/src/tracing/local-instrumentation-runtime.js
node_modules/eve/dist/src/tracing/agent-otel-provider.js
node_modules/eve/dist/src/tracing/agent-otel-provider.d.ts
$ grep -rl 'getTracer(`eve.agent`' node_modules/eve/dist/src | grep -v /compiled/
node_modules/eve/dist/src/tracing/local-instrumentation-runtime.jscreateProductionApplicationNitro in the same file never references the local tracing plugin at
all, so on our reading a built agent has no agent.* spans regardless of instrumentation.ts. We
have not run a production build to confirm that.
Reproduction
eve 0.30.6, @vercel/otel@2.1.3, ai@7.0.51, @ai-sdk/openai@2.x, model gpt-5-mini, Node
25.2.1, macOS 26.6, Postgres workflow world.
-
Scaffold an agent, eve dev, send one message that calls a tool, then eve traces. Spans:
agent.session → agent.turn → agent.step → ai.streamText.doStream, and
agent.action → ai.toolCall.
-
Add agent/instrumentation.ts exporting to any OTLP endpoint:
import { OTLPHttpJsonTraceExporter, registerOTel } from "@vercel/otel";
import { defineInstrumentation } from "eve/instrumentation";
export default defineInstrumentation({
setup: ({ agentName }) =>
registerOTel({
serviceName: agentName,
traceExporter: new OTLPHttpJsonTraceExporter({ url: process.env.OTLP_URL! }),
}),
recordInputs: true,
recordOutputs: true,
});
- Restart
eve dev, send the same message. Group the received spans by instrumentation scope.
There is no eve.agent scope. eve traces is also empty, which is documented.
Per the gate, step 2 does not need an exporter at all — an agent/instrumentation.ts containing
only export default defineInstrumentation({}) should be enough, since the plugin path is keyed on
the file existing. We read that from resolveInstrumentationModule; we did not run that variant.
What we measured
Our collector is Postgres, one row per span, so these are counts of real rows.
Before — the last trace eve wrote to .eve/traces/v1 before agent/instrumentation.ts existed
(eve 0.29.5; the spans are stamped scope_version = 0.29.5), imported into the same table by hand:
scope_name | scope_version | name | count
eve.agent | 0.29.5 | agent.session | 1
eve.agent | 0.29.5 | agent.turn | 2
eve.agent | 0.29.5 | agent.step | 3
eve.agent | 0.29.5 | agent.turn.terminal | 2
eve.agent | 0.29.5 | agent.action | 1
eve.agent | 0.29.5 | ai.streamText | 3
eve.agent | 0.29.5 | ai.streamText.doStream | 3
eve.agent | 0.29.5 | ai.toolCall | 1
After — every span our authored exporter has delivered since, across 37 turns (snapshot; the
agent is still running). The last three columns are attributes ->> 'agent.session.id',
'agent.root.session.id' and 'agent.turn.id':
scope_name | spans | agent.session.id | agent.root.session.id | agent.turn.id
workflow | 4750 | 0 | 0 | 0
@vercel/otel/fetch | 322 | 0 | 0 | 0
gen_ai | 138 | 0 | 0 | 0
eve | 37 | 0 | 0 | 0
No eve.agent scope, and 5,247 spans with none of the three ids. .eve/traces/v1 has likewise not
been written since the moment agent/instrumentation.ts appeared: it still holds exactly one trace
directory, timestamped 19 minutes before that file was created, across a day of eve dev sessions.
The prompts are not lost — they arrive under the GenAI conventions instead:
gen_ai / invoke_agent gpt-5-mini
gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions,
gen_ai.usage.*, gen_ai.response.finish_reasons,
ai.settings.context.eve.session.id, ai.settings.context.eve.turn.id,
ai.settings.context.eve.turn.sequence, ai.settings.context.eve.step.index
gen_ai / execute_tool recall
gen_ai.tool.name, gen_ai.tool.call.id, gen_ai.tool.call.arguments, gen_ai.tool.call.result
eve / ai.eve.turn
eve.session.id, eve.turn.id, eve.version, eve.environment, ai.telemetry.functionId
That list is also where the concrete gap shows. agent.session.id has a counterpart; the root
session id does not. The framework half of the runtime context documented in
guides/instrumentation.md is eve.version, eve.session.id, eve.environment, eve.turn.id,
eve.turn.sequence, eve.step.index, eve.channel.kind — no root. So a subagent's spans cannot be
attributed to the session that spawned it from exported telemetry alone, whereas agent.session
and agent.turn carry agent.root.session.id directly.
Why this matters beyond us
guides/instrumentation.md points users at Braintrust, PostHog, Raindrop, Arize, Honeycomb,
Datadog and Jaeger, and the /traces TUI shows them the agent.* tree while they build. Anyone
who models their backend on what they saw locally — dashboards, span-name filters, id joins —
gets a working pipeline that quietly receives a different shape the moment they wire up the
exporter. Nothing errors; the panels are just empty. Ours were: three generated Postgres columns
keyed on agent.session.id / agent.root.session.id / agent.turn.id, NULL on every exported
row.
It also means the richest thing eve knows about its own execution — the agent-level span tree,
which is eve's own contribution over the AI SDK's — is reachable only from a dev-only spool with
bounded retention, and never from a deployed agent.
What we would like
In rough order of preference:
-
Install the agent instrumentation independently of the local trace spool. They are already
two objects in installLocalInstrumentationRuntime: keep LocalTraceSpanProcessor gated on
"no authored instrumentation", and install createAgentOtelInstrumentation unconditionally —
in production builds too. Authored instrumentation then gets agent.* spans through whatever
provider it registered, and eve traces keeps its current behaviour.
-
Or make it opt-in from authored instrumentation, e.g. a field on defineInstrumentation
(agentSpans: true) or an exported installer from eve/instrumentation. Less good — it is a
flag users must learn exists — but it removes the "cannot be done at all" part.
-
At minimum, close the id gap and say this in the docs. Add a root-session key to the
framework runtime context so exported spans can be stitched to a root session, and state
plainly in guides/instrumentation.md that the local agent.* vocabulary is not what an
authored exporter receives. Today's surfaces table reads as though it is.
Happy to open a PR for (1) if that direction is welcome.
Verified vs inferred
Verified here, against eve 0.30.6 as published on npm:
- the gate, the single call site, the single
registerInstrumentationRuntime caller, the eve.agent
tracer name, and the absence of any public specifier resolving into dist/src/tracing;
- the same gate, unchanged, in the 0.29.5, 0.30.0, 0.30.1, 0.30.2, 0.30.3, 0.30.4 and 0.30.5 tarballs;
- the span counts above, from a live agent against Postgres, and the untouched
.eve/traces/v1.
Inferred from source and not executed:
- that an
agent/instrumentation.ts with no exporter (or no default export) is enough to trigger it;
- that production builds never install the agent instrumentation, since
createProductionApplicationNitro does not reference the local tracing plugin;
- the intent behind the coupling. It reads like the instrumentation and the spool writer were built
as one unit for the dev TUI, and the export path inherited the split — but that is a guess, and if
it is deliberate we would like to know what the intended path to agent.* spans in a backend is.
Authoring
agent/instrumentation.tsremoves the agent OTel instrumentation, so exported traces contain noagent.*spansSummary
The
eve.agentspan family —agent.session,agent.turn,agent.step,agent.action,agent.turn.terminal, plus theai.streamText.doStream/ai.toolCallspans that carry theprompt and tool bodies — is created by
createAgentOtelInstrumentation(). That function hasexactly one caller,
installLocalInstrumentationRuntime(), and that runtime is installed onlywhen the app has no
agent/instrumentation.ts.So the two paths are mutually exclusive. Author
instrumentation.tsto export OTLP to your ownbackend — the documented way to get traces off the box — and you silently opt out of the entire
agent.*vocabulary. What your collector receives instead is Workflow plumbing, oneai.eve.turnspan per turn, and the AI SDK's
gen_ai.*spans.The AI SDK spans do carry prompts and tool payloads, so this is not "telemetry is empty". It is
narrower and easier to miss: the span vocabulary you see in
eve tracesis not the vocabularyyour backend receives, and one identifier — the root session id — has no equivalent on the
exported side at all, so subagent → root-session lineage cannot be reconstructed downstream.
docs/guides/instrumentation.mddoes say that authoringinstrumentation.tsreplaces localrecording, so an empty
eve tracesis expected. It does not say the agent instrumentation goeswith it — its surfaces table describes the OpenTelemetry export row as "Where agent and AI spans
are exported" (
guides/instrumentation.md), which is what we built against.This is not a 0.30.x regression. We first assumed it was, and it is not: the gate below is
byte-identical in every release we unpacked, 0.29.5 through 0.30.6.
The two files
dist/src/internal/nitro/host/create-application-nitro.js, insidecreateDevelopmentApplicationNitro(minified; the gate is the whole story):compiledArtifacts.instrumentationPluginPathis set wheneverresolveInstrumentationModule()finds
agent/instrumentation.{ts,mts,js,mjs}— file existence alone, before any default export isinspected.
dist/src/tracing/local-instrumentation-runtime.jsis the only place the agent instrumentation isconstructed, and it does so in the same function that installs the local spool processor:
Those are one unit today: a span processor that writes
.eve/traces/v1and an instrumentationthat produces the
agent.*spans. Only the first is dev-local by nature. The second is the partan external backend wants.
Three static facts follow, all checkable in a published tarball:
createAgentOtelInstrumentationappears in exactly two non-vendored modules: its own andtracing/local-instrumentation-runtime.js.registerInstrumentationRuntime(— the only way a runtime becomes visible to the harness — iscalled from exactly one module, the same one.
exportsmap resolves intodist/src/tracing/, andeve/instrumentationexports only
defineInstrumentationandisChannel, so an app cannot install the agentinstrumentation itself.
From an agent directory with eve 0.30.6 installed:
createProductionApplicationNitroin the same file never references the local tracing plugin atall, so on our reading a built agent has no
agent.*spans regardless ofinstrumentation.ts. Wehave not run a production build to confirm that.
Reproduction
eve 0.30.6,
@vercel/otel@2.1.3,ai@7.0.51,@ai-sdk/openai@2.x, modelgpt-5-mini, Node25.2.1, macOS 26.6, Postgres workflow world.
Scaffold an agent,
eve dev, send one message that calls a tool, theneve traces. Spans:agent.session→agent.turn→agent.step→ai.streamText.doStream, andagent.action→ai.toolCall.Add
agent/instrumentation.tsexporting to any OTLP endpoint:eve dev, send the same message. Group the received spans by instrumentation scope.There is no
eve.agentscope.eve tracesis also empty, which is documented.Per the gate, step 2 does not need an exporter at all — an
agent/instrumentation.tscontainingonly
export default defineInstrumentation({})should be enough, since the plugin path is keyed onthe file existing. We read that from
resolveInstrumentationModule; we did not run that variant.What we measured
Our collector is Postgres, one row per span, so these are counts of real rows.
Before — the last trace eve wrote to
.eve/traces/v1beforeagent/instrumentation.tsexisted(eve 0.29.5; the spans are stamped
scope_version = 0.29.5), imported into the same table by hand:After — every span our authored exporter has delivered since, across 37 turns (snapshot; the
agent is still running). The last three columns are
attributes ->> 'agent.session.id','agent.root.session.id'and'agent.turn.id':No
eve.agentscope, and 5,247 spans with none of the three ids..eve/traces/v1has likewise notbeen written since the moment
agent/instrumentation.tsappeared: it still holds exactly one tracedirectory, timestamped 19 minutes before that file was created, across a day of
eve devsessions.The prompts are not lost — they arrive under the GenAI conventions instead:
That list is also where the concrete gap shows.
agent.session.idhas a counterpart; the rootsession id does not. The framework half of the runtime context documented in
guides/instrumentation.mdiseve.version,eve.session.id,eve.environment,eve.turn.id,eve.turn.sequence,eve.step.index,eve.channel.kind— no root. So a subagent's spans cannot beattributed to the session that spawned it from exported telemetry alone, whereas
agent.sessionand
agent.turncarryagent.root.session.iddirectly.Why this matters beyond us
guides/instrumentation.mdpoints users at Braintrust, PostHog, Raindrop, Arize, Honeycomb,Datadog and Jaeger, and the
/tracesTUI shows them theagent.*tree while they build. Anyonewho models their backend on what they saw locally — dashboards, span-name filters, id joins —
gets a working pipeline that quietly receives a different shape the moment they wire up the
exporter. Nothing errors; the panels are just empty. Ours were: three generated Postgres columns
keyed on
agent.session.id/agent.root.session.id/agent.turn.id, NULL on every exportedrow.
It also means the richest thing eve knows about its own execution — the agent-level span tree,
which is eve's own contribution over the AI SDK's — is reachable only from a dev-only spool with
bounded retention, and never from a deployed agent.
What we would like
In rough order of preference:
Install the agent instrumentation independently of the local trace spool. They are already
two objects in
installLocalInstrumentationRuntime: keepLocalTraceSpanProcessorgated on"no authored instrumentation", and install
createAgentOtelInstrumentationunconditionally —in production builds too. Authored instrumentation then gets
agent.*spans through whateverprovider it registered, and
eve traceskeeps its current behaviour.Or make it opt-in from authored instrumentation, e.g. a field on
defineInstrumentation(
agentSpans: true) or an exported installer fromeve/instrumentation. Less good — it is aflag users must learn exists — but it removes the "cannot be done at all" part.
At minimum, close the id gap and say this in the docs. Add a root-session key to the
framework runtime context so exported spans can be stitched to a root session, and state
plainly in
guides/instrumentation.mdthat the localagent.*vocabulary is not what anauthored exporter receives. Today's surfaces table reads as though it is.
Happy to open a PR for (1) if that direction is welcome.
Verified vs inferred
Verified here, against eve 0.30.6 as published on npm:
registerInstrumentationRuntimecaller, theeve.agenttracer name, and the absence of any public specifier resolving into
dist/src/tracing;.eve/traces/v1.Inferred from source and not executed:
agent/instrumentation.tswith no exporter (or no default export) is enough to trigger it;createProductionApplicationNitrodoes not reference the local tracing plugin;as one unit for the dev TUI, and the export path inherited the split — but that is a guess, and if
it is deliberate we would like to know what the intended path to
agent.*spans in a backend is.