Skip to content

feat(eve): let a provider say how much of an event it needs #1823

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
chadhietala wants to merge 1 commit into from
Draft

feat(eve): let a provider say how much of an event it needs #1823

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .changeset/capture-below-the-otel-layer.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
"eve": patch
---

Let an instrumentation provider declare how much of each event it wants. Under
the experimental provider layout, `capture: "content"` opts a provider into the
prompt, the response, and tool payloads; the default `"metadata"` leaves it
structure, usage, and timing. Content is now built only when something asked for
it, so an agent whose providers and destinations all decline never serializes a
prompt at all.
47 changes: 45 additions & 2 deletions docs/guides/instrumentation-providers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,46 @@ Pass `spanProcessors` instead when the destination needs its own batching, sampl

`otel()` and `otelIntegration()` come from `eve/instrumentation/otel`, a separate entrypoint from `eve/instrumentation`.

## Content is per provider

A provider declares how much of each event it wants. The default is
`"metadata"`: structure, identity, usage, and timing, but not what the
conversation said. `"content"` adds the prompt, the response, tool arguments,
and tool results.

```ts title="agent/instrumentation/audit.ts"
export default defineInstrumentation({
capture: "content",
events: {
"model.call.completed": (event) => {
console.log(event.content);
},
},
});
```

Asking is what makes eve build the projection at all. A directory in which no
provider asked — and whose destinations all declined — never serializes a
prompt, so declining is cheaper than filtering as well as safer. Providers that
did not ask receive the same events with the content fields absent; a
`capture: "content"` provider beside them changes nothing about what they see.

Structure survives declining. `action.completed` still says whether the tool
returned or threw, and `model.call.completed` still carries its finish reason
and token usage — a provider counting failures or cost never has to ask for
content to get them.

Failure details and opaque provider metadata can contain prompts, tool output,
search queries, or retrieved text, so metadata providers do not receive them.
`step.attempt.metadata` keeps only the gateway cost and generation ID by
default; `capture: "content"` exposes the complete provider payload and failure
objects.

Content fields are therefore optional on the event types that carry them:
`input` on `model.call.started`, `action.started`, and `tool.call.started`;
`content` on `model.call.completed`; and the payloads inside action and tool-call
outputs.

## Content is per destination

`recordInputs` and `recordOutputs` belong to a destination, not to the process. Content is written onto a span if any destination wants it, and each destination that declined never exports it. A local spool and a hosted backend no longer have to agree:
Expand All @@ -94,7 +134,7 @@ export default otelIntegration({
});
```

An agent whose every destination declines still never materializes a prompt — the union of nothing is nothing. Declining wraps every processor in that file, an author's included: they are this destination, and the point of declining is that nothing under it sees what was said. The wrapper copies the span rather than editing it, because the span it is handed is shared with every other destination in the pipeline.
An agent whose every destination declines still never materializes a prompt — the OpenTelemetry pipeline is itself one provider, and a pipeline whose destinations all declined asks for `"metadata"` like any other. Declining wraps every processor in that file, an author's included: they are this destination, and the point of declining is that nothing under it sees what was said. The wrapper copies the span rather than editing it, because the span it is handed is shared with every other destination in the pipeline.

For sensitive, regulated, or production data, decline content on any destination whose retention path you have not reviewed. You are responsible for ensuring an observability or eval provider is approved for what is exported to it.

Expand Down Expand Up @@ -147,7 +187,10 @@ A provider's `events` map takes one handler per event type, each called with `(e
| `action.started`, `action.completed`, `action.failed` | Every eve dispatch: tool call, skill load, subagent, or remote agent |
| `tool.call.started`, `tool.call.completed`, `tool.call.failed` | The AI SDK execution boundary for an ordinary tool call |

An ordinary tool emits both families. `action.*` is eve's durable dispatch boundary and covers work that can settle in another worker; `tool.call.*` is the model SDK's in-process execution boundary. Handle one unless you intentionally want both views.
An ordinary tool emits both families. `action.*` is eve's durable dispatch
boundary and covers work that can settle in another worker; `tool.call.*` is the
model SDK's in-process execution boundary. Handle one unless you intentionally
want both views.

Every event carries an `idempotencyKey` naming the operation it is about. A start and its terminal share a key when the terminal arrives. An incomplete model stream can close without one, so live resources need step-attempt cleanup or their own expiry.

Expand Down
155 changes: 138 additions & 17 deletions packages/eve/src/harness/ai-sdk-hook-bridge.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@ import { describe, expect, it, vi } from "vitest";

import { createAiSdkHookBridge } from "#harness/ai-sdk-hook-bridge.js";
import {
attemptIdempotencyKey,
createInstrumentationHooks,
modelCallIdempotencyKey,
type InstrumentationAttemptScope,
type InstrumentationModelCallStartedEvent,
type InstrumentationModelCallTerminalEvent,
Expand Down Expand Up @@ -58,7 +60,7 @@ describe("createAiSdkHookBridge", () => {
},
]);

const id = `model:${scope.attemptId}:0`;
const id = modelCallIdempotencyKey(scope, 0);
expect(calls).toEqual([
`a:started:${id}`,
`b:started:${id}`,
Expand Down Expand Up @@ -94,7 +96,10 @@ describe("createAiSdkHookBridge", () => {
it("passes the identity captured at model-call start to the context runner", async () => {
const ids: string[] = [];
const hooks = createInstrumentationHooks([
{ events: { "model.call.started": (event) => void ids.push(event.idempotencyKey) } },
{
events: { "model.call.started": (event) => void ids.push(event.idempotencyKey) },
name: "recorder",
},
]);
const bridge = createAiSdkHookBridge(scope, hooks, (operation, execute) => {
ids.push(operation.idempotencyKey);
Expand All @@ -111,7 +116,7 @@ describe("createAiSdkHookBridge", () => {

await bridge.executeLanguageModelCall!({ callId: "call-1", execute: async () => "result" });

const expected = `model:${scope.attemptId}:0`;
const expected = modelCallIdempotencyKey(scope, 0);
expect(ids).toEqual([expected, expected]);
});

Expand All @@ -132,7 +137,10 @@ describe("createAiSdkHookBridge", () => {
it("derives replay-stable model identity without the AI SDK call ID", async () => {
const keys: string[] = [];
const hooks = createInstrumentationHooks([
{ events: { "model.call.started": (event) => void keys.push(event.idempotencyKey) } },
{
events: { "model.call.started": (event) => void keys.push(event.idempotencyKey) },
name: "keys",
},
]);

for (const callId of ["sdk-random-1", "sdk-random-2"]) {
Expand All @@ -143,7 +151,7 @@ describe("createAiSdkHookBridge", () => {
]);
}

expect(keys).toEqual([`model:${scope.attemptId}:2`, `model:${scope.attemptId}:2`]);
expect(keys).toEqual([modelCallIdempotencyKey(scope, 2), modelCallIdempotencyKey(scope, 2)]);
});

it("publishes step provider metadata as step.metadata, skipping steps without any", async () => {
Expand All @@ -155,19 +163,29 @@ describe("createAiSdkHookBridge", () => {
events.push(event);
},
},
name: "metadata",
},
]);
const bridge = createAiSdkHookBridge(scope, hooks);

await Reflect.apply(bridge.onStepEnd!, bridge, [
{ providerMetadata: { gateway: { cost: "0.000082" } } },
{
providerMetadata: {
gateway: {
cost: "0.000082",
generationId: "generation-1",
groundingSegments: ["private result"],
},
google: { searchQueries: ["private query"] },
},
},
]);
await Reflect.apply(bridge.onStepEnd!, bridge, [{ providerMetadata: undefined }]);

expect(events).toEqual([
{
idempotencyKey: `step:${scope.attemptId}`,
providerMetadata: { gateway: { cost: "0.000082" } },
idempotencyKey: attemptIdempotencyKey(scope),
providerMetadata: { gateway: { cost: "0.000082", generationId: "generation-1" } },
scope,
type: "step.attempt.metadata",
},
Expand All @@ -183,11 +201,13 @@ describe("createAiSdkHookBridge", () => {
throw new Error("provider failed");
},
},
name: "thrower",
},
{
events: {
"model.call.completed": after,
},
name: "after",
},
]);
const bridge = createAiSdkHookBridge(scope, hooks);
Expand All @@ -211,7 +231,9 @@ describe("createAiSdkHookBridge", () => {

it("terminalizes started operations when the attempt errors", async () => {
const after = vi.fn();
const hooks = createInstrumentationHooks([{ events: { "model.call.failed": after } }]);
const hooks = createInstrumentationHooks([
{ capture: "content", events: { "model.call.failed": after }, name: "after" },
]);
const bridge = createAiSdkHookBridge(scope, hooks);

await Reflect.apply(bridge.onLanguageModelCallStart!, bridge, [
Expand All @@ -236,8 +258,8 @@ describe("createAiSdkHookBridge", () => {
});
const started = vi.fn();
const hooks = createInstrumentationHooks([
{ events: { "step.attempt.started": mutator } },
{ events: { "step.attempt.started": started } },
{ events: { "step.attempt.started": mutator }, name: "mutator" },
{ events: { "step.attempt.started": started }, name: "started" },
]);
const bridge = createAiSdkHookBridge(scope, hooks);

Expand All @@ -247,7 +269,7 @@ describe("createAiSdkHookBridge", () => {
await Reflect.apply(bridge.onStepStart!, bridge, [{ callId: "call-1", stepNumber: 0 }]);

const expected = {
idempotencyKey: `step:${scope.attemptId}`,
idempotencyKey: attemptIdempotencyKey(scope),
operation: { modelId: "model", operationId: "ai.streamText", provider: "test" },
scope,
type: "step.attempt.started",
Expand All @@ -258,21 +280,27 @@ describe("createAiSdkHookBridge", () => {

it("projects the model call callbacks onto eve fields only", async () => {
const before = vi.fn((event: InstrumentationModelCallStartedEvent) => {
if (event.input === undefined) throw new Error("expected model input");
expect(Object.isFrozen(event)).toBe(true);
expect(Object.isFrozen(event.input)).toBe(true);
expect(Object.isFrozen(event.input.messages)).toBe(true);
expect(Object.isFrozen(event.model)).toBe(true);
});
const after = vi.fn((event: InstrumentationModelCallTerminalEvent) => {
if (event.type !== "model.call.completed") throw new Error("expected completed model call");
if (event.content === undefined) throw new Error("expected model content");
expect(Object.isFrozen(event)).toBe(true);
expect(Object.isFrozen(event.content)).toBe(true);
expect(event.content.every((part) => Object.isFrozen(part))).toBe(true);
expect(Object.isFrozen(event.usage)).toBe(true);
expect(Object.isFrozen(event.usage.inputTokenDetails)).toBe(true);
});
const hooks = createInstrumentationHooks([
{ events: { "model.call.completed": after, "model.call.started": before } },
{
capture: "content",
events: { "model.call.completed": after, "model.call.started": before },
name: "spy",
},
]);
const bridge = createAiSdkHookBridge(scope, hooks);

Expand Down Expand Up @@ -310,7 +338,7 @@ describe("createAiSdkHookBridge", () => {

expect(before).toHaveBeenCalledExactlyOnceWith(
{
idempotencyKey: `model:${scope.attemptId}:0`,
idempotencyKey: modelCallIdempotencyKey(scope, 0),
input: { instructions: "be brief", messages: [{ content: "hi", role: "user" }] },
model: { modelId: "model", provider: "test" },
scope,
Expand All @@ -330,7 +358,7 @@ describe("createAiSdkHookBridge", () => {
{ error: "boom", input: { a: 2 }, toolName: "search", type: "tool-error" },
],
finishReason: "tool-calls",
idempotencyKey: `model:${scope.attemptId}:0`,
idempotencyKey: modelCallIdempotencyKey(scope, 0),
scope,
type: "model.call.completed",
usage: {
Expand Down Expand Up @@ -363,8 +391,17 @@ describe("createAiSdkHookBridge", () => {
expect(Object.isFrozen(event)).toBe(true);
expect(Object.isFrozen(event.output)).toBe(true);
});
const actionStarted = vi.fn();
const hooks = createInstrumentationHooks([
{ events: { "tool.call.completed": after, "tool.call.started": before } },
{
capture: "content",
events: {
"action.started": actionStarted,
"tool.call.completed": after,
"tool.call.started": before,
},
name: "spy",
},
]);
const bridge = createAiSdkHookBridge(scope, hooks);
const toolCall = { input: { q: "eve" }, toolCallId: "tool-1", toolName: "search" };
Expand Down Expand Up @@ -394,9 +431,90 @@ describe("createAiSdkHookBridge", () => {
},
expect.anything(),
);
expect(actionStarted).not.toHaveBeenCalled();
},
);

it("omits content from the projection when no provider asked for it", async () => {
const modelStarted = vi.fn();
const modelCompleted = vi.fn();
const toolStarted = vi.fn();
const toolCompleted = vi.fn();
const hooks = createInstrumentationHooks([
{
events: {
"model.call.completed": modelCompleted,
"model.call.started": modelStarted,
"tool.call.completed": toolCompleted,
"tool.call.started": toolStarted,
},
name: "metadata-only",
},
]);
const bridge = createAiSdkHookBridge(scope, hooks);
const toolCall = { input: { q: "eve" }, toolCallId: "tool-1", toolName: "search" };

await Reflect.apply(bridge.onLanguageModelCallStart!, bridge, [
{
callId: "call-1",
instructions: "be brief",
messages: [{ content: "hi", role: "user" }],
modelId: "model",
provider: "test",
tools: undefined,
},
]);
await Reflect.apply(bridge.onLanguageModelCallEnd!, bridge, [
{
callId: "call-1",
content: [{ text: "hello", type: "text" }],
finishReason: "stop",
performance: { responseTimeMs: 1 },
responseId: "response-1",
usage: { inputTokens: 1, outputTokens: 2 },
},
]);
await Reflect.apply(bridge.onToolExecutionStart!, bridge, [{ callId: "call-1", toolCall }]);
await Reflect.apply(bridge.onToolExecutionEnd!, bridge, [
{
callId: "call-1",
toolCall,
toolExecutionMs: 1,
toolOutput: { output: "ok", type: "tool-result" },
},
]);

expect(modelStarted.mock.calls[0]?.[0].input).toBeUndefined();
expect(modelCompleted.mock.calls[0]?.[0].content).toBeUndefined();
// Structure survives: usage, the finish reason, and the tool's identity are
// not what was said.
expect(modelCompleted.mock.calls[0]?.[0].finishReason).toBe("stop");
expect(toolStarted.mock.calls[0]?.[0].input).toBeUndefined();
expect(toolStarted.mock.calls[0]?.[0].toolName).toBe("search");
expect(toolCompleted.mock.calls[0]?.[0].output).toEqual({ type: "result" });
});

it("withholds content from a metadata provider sharing a bus with a content one", async () => {
const metadataOnly = vi.fn();
const wantsContent = vi.fn();
const hooks = createInstrumentationHooks([
{ events: { "tool.call.started": metadataOnly }, name: "metadata-only" },
{
capture: "content",
events: { "tool.call.started": wantsContent },
name: "wants-content",
},
]);
const bridge = createAiSdkHookBridge(scope, hooks);

await Reflect.apply(bridge.onToolExecutionStart!, bridge, [
{ callId: "call-1", toolCall: { input: { q: "eve" }, toolCallId: "t", toolName: "search" } },
]);

expect(wantsContent.mock.calls[0]?.[0].input).toEqual({ q: "eve" });
expect(metadataOnly.mock.calls[0]?.[0].input).toBeUndefined();
expect(metadataOnly.mock.calls[0]?.[0].toolName).toBe("search");
});
it("keeps each provider's state to itself", async () => {
const observed = new Map<string, unknown>();
const provider = (name: string): InstrumentationProviderDefinition => {
Expand Down Expand Up @@ -438,7 +556,9 @@ describe("createAiSdkHookBridge", () => {

it("skips a terminal handler when the operation never started", async () => {
const completed = vi.fn();
const hooks = createInstrumentationHooks([{ events: { "model.call.completed": completed } }]);
const hooks = createInstrumentationHooks([
{ events: { "model.call.completed": completed }, name: "completed" },
]);
const bridge = createAiSdkHookBridge(scope, hooks);

// No onLanguageModelCallStart, so the bridge holds no id and publishes
Expand Down Expand Up @@ -476,6 +596,7 @@ describe("createAiSdkHookBridge", () => {
terminalStates.set(event.idempotencyKey, started.get(event.idempotencyKey));
},
},
name: "parallel",
},
]);
const bridge = createAiSdkHookBridge(scope, hooks);
Expand Down
Loading