[RFC] Session Inspector: a per-session runtime event trace, replay, and cost-attribution surface #1625

Activity

1. ARE404 commented on Jul 29, 2026

   

   ARE404

   on Jul 29, 2026

   ContributorAuthor

   More actions

   I'd like to take this and own the implementation.

   I'll start with PR 1 — the pure RuntimeEventTrace read model + node tests — since it's behavior-preserving and independently mergeable, so it won't block anything else. Happy to settle the four open questions before I open a branch; my current leanings are noted in the RFC. I'll wait for your confirmation / assignment before starting the code.
2. Astro-Han commented on Jul 29, 2026

   

   Astro-Han

   on Jul 29, 2026

   Contributor

   More actions

   Thanks for the writeup — the framing looks right to us: the ledger already carries the causal detail, and the gap is a read model plus a surface. Here's our take on the four questions, plus a few things we ran into while checking the current code. Treat all of it as input rather than a ruling — you're closer to the design than we are.

   On the four questions

   Q1 — Panel home. Worth noting the two options are the same surface: SessionWorkbar is the right rail (apps/desktop/src/renderer/session-workbar.tsx:55), and the Task ledger is its tasks tab. So the lightest path is probably a fifth tab — extending SessionWorkbarTab (session-workbar-layout.ts:8) with 'inspector'. One thing to avoid: attaching it to AgentGraphPanel, which is mounted in the chat column and gated on orchestrationMode === 'graph' (app-shell.tsx:2169), so ordinary sessions wouldn't see the Inspector at all.

   Q2 — Where the trace read model lives. We'd suggest following the #1549 split: contract in packages/core, builder in packages/runtime. Note the reason given for leaning runtime ("it depends on the pricing authority") may not survive — see the pricing note below.

   Q3 — Cost attribution. Your proposal reads right to us: cost on the enclosing LLM invocation, tool steps at $0. RuntimeEventTokenUsage rides the LLM event rather than the tool dispatch, so a per-tool cost would mean a second attribution rule. The part we'd look at more closely isn't the $0 but totals — below.

   Q4 — Reconstruction entry point. It's already exposed. packages/runtime/src/agent-graph-timeline.ts exports buildAgentGraphTimeline, buildAgentGraphTimelineCurrentState, paginateAgentGraphTimeline, cursor encode/decode, and AGENT_GRAPH_TIMELINE_SCHEMA_VERSION — so PR 1 shouldn't need to define its own read boundary.

   Things we ran into while checking

   1. The pricing parameter. The sketch signature likely can't be implemented as written. PricingAuthorityReader does exist (packages/storage/src/usage-stores.ts:68), but it's async (snapshot(): Promise<PricingSnapshot>), lives in packages/storage, and is reached through packages/runtime-host/src/server/usage-pricing-coordinator.ts under a branded reader/writer single-writer discipline. A synchronous pure function in packages/runtime can't hold it, and pulling storage into runtime would likely trip the import-graph check in packages/runtime-host/src/__tests__/dependency-boundary.test.ts. Taking a resolved PricingSnapshot (packages/storage/src/pricing-store.ts:23) and letting the caller await snapshot() at the edge would keep PR 1 synchronous and pure — and it happens to settle Q2 too.

   2. costUsd in SessionTrace.totals. This is the spot we'd think hardest about, because it's where the Inspector could end up as a second cost authority without anyone intending it. Settings → Usage aggregates the telemetry index (TelemetryIndexReader); the Inspector projects the RuntimeEvent ledger — two record paths over one pricing table, so the same session could show two numbers. The concrete case: the #1549 ledger is replayable, and a replayed LLM call appears twice in events but once in telemetry, so a self-summed total would double-count. One option is to keep in totals only what the ledger is authoritative for (duration, tokens, compaction count, retry count) and, if per-session cost should appear, read the existing telemetry aggregate rather than summing steps. If you see a way to reconcile the two paths instead, that'd be worth hearing.

   3. The searchTrace API. The PR 3 search bullet looks like two different things to us. Structured predicates ("jump to compaction", "steps over $X", "failed tools", by tool name) are filters over TraceStep[] — possibly just a caller-side Array.filter rather than new API surface. For free-text matching, runtimeEventSearchText (packages/runtime/src/runtime-event-history-search.ts:126) already extracts searchable text from an event and could be reused. Worth knowing that searchRuntimeEventHistory itself is a context-budget retrieval path (token ceilings, policy.enabled gating), so it's probably not the right thing behind a UI search box.

   One thing worth weighing

   packages/runtime already carries run-trace.ts, stream-graph-trace.ts, agent-graph-timeline.ts, and runtime-event-history-search.ts. Before adding a fifth, it might be worth comparing against extending agent-graph-timeline.ts, which already does turn-scoped reconstruction with pagination — the latency / cost / failure-attribution projection could plausibly live there. If they can't reasonably share a builder, a line in the PR description explaining why would help reviewers.

   None of this blocks the RFC. The non-goals are drawn well, and PR 1 being pure and independently mergeable is the right shape to start from.

   中文对照

   感谢这份 RFC，在我们看来方向是对的：账本里已经有因果细节，缺的是一个读模型和一个展示面。下面是我们对四个问题的看法，以及在核对现有代码时遇到的几点。这些都作为参考意见，而不是结论——你比我们更贴近这个设计。

   关于四个问题

   Q1 — 面板归属。 值得一提的是，这两个选项其实是同一个地方：SessionWorkbar 本身就是右栏（apps/desktop/src/renderer/session-workbar.tsx:55），Task ledger 就是它的 tasks tab。所以最轻的路径可能是加第五个 tab——在 SessionWorkbarTab（session-workbar-layout.ts:8）里扩展一个 'inspector'。有一点建议避开：挂到 AgentGraphPanel 上，它挂在 chat 主列且受 orchestrationMode === 'graph' 门禁（app-shell.tsx:2169），普通会话会完全看不到 Inspector。

   Q2 — trace 读模型放哪层。 我们建议沿用 #1549 的分法：契约放 packages/core，构建放 packages/runtime。另外，原文倾向 runtime 的理由（“它依赖 pricing authority”）可能站不住——见下面关于定价的那条。

   Q3 — 成本归因粒度。 你的提案在我们看来是对的：成本挂在外层 LLM invocation 上，tool step 显示 $0。RuntimeEventTokenUsage 是随 LLM 事件走而非随 tool dispatch 走的，派生 per-tool 成本意味着新增一条归因规则。我们更想细看的不是这个 $0，而是 totals——见下。

   Q4 — 重建入口。 已经暴露了。packages/runtime/src/agent-graph-timeline.ts 导出了 buildAgentGraphTimeline、buildAgentGraphTimelineCurrentState、paginateAgentGraphTimeline、游标编解码，以及 AGENT_GRAPH_TIMELINE_SCHEMA_VERSION，所以 PR 1 应该不需要另立读边界。

   核对时遇到的几点

   1. 定价参数。 草图里的签名照写可能实现不出来。PricingAuthorityReader 确实存在（packages/storage/src/usage-stores.ts:68），但它是异步的（snapshot(): Promise<PricingSnapshot>）、住在 packages/storage，且需经 packages/runtime-host/src/server/usage-pricing-coordinator.ts 在带 brand 的 reader/writer 单写者纪律下访问。packages/runtime 里的同步纯函数拿不住它，而把 storage 拖进 runtime 很可能会撞上 packages/runtime-host/src/__tests__/dependency-boundary.test.ts 的 import 图检查。改成接收已解析的 PricingSnapshot（packages/storage/src/pricing-store.ts:23）、由调用方在边缘 await snapshot()，可以让 PR 1 保持同步且纯——顺带也把 Q2 解决了。

   2. SessionTrace.totals 里的 costUsd。 这是我们最想斟酌的一处，因为 Inspector 可能在无人有意为之的情况下从这里变成第二个成本权威。Settings → Usage 聚合的是 telemetry 索引（TelemetryIndexReader），Inspector 投影的是 RuntimeEvent 账本——两条记录路径、一张定价表，同一个 session 可能显示出两个数字。具体的例子：#1549 的账本可重放，一次被重放的 LLM 调用在事件里出现两次、在 telemetry 里只有一次，自行求和就会双算。一种做法是 totals 里只保留账本本身有权威的量（时长、token、compaction 次数、retry 次数），如果确实要显示这个会话的花费，就读 telemetry 已有的聚合而不是把各 step 加起来。如果你有办法把两条路径对齐，我们也很想听。

   3. searchTrace 这个 API。 PR 3 的 search 那一条在我们看来是两件事。结构化谓词（“跳到 compaction”、“超过 $X 的 step”、“失败的工具”、按工具名）是对 TraceStep[] 的过滤，也许调用方一个 Array.filter 就够，不必新增 API 面。全文匹配方面，runtimeEventSearchText（packages/runtime/src/runtime-event-history-search.ts:126）已经能从事件里抽出可搜文本，可以复用。另外值得知道的是，searchRuntimeEventHistory 本身是上下文预算的检索路径（有 token 上限、policy.enabled 门禁），大概不适合直接放到 UI 搜索框后面。

   一处值得权衡的地方

   packages/runtime 里已经有 run-trace.ts、stream-graph-trace.ts、agent-graph-timeline.ts 和 runtime-event-history-search.ts。在加第五个之前，也许值得和“扩展 agent-graph-timeline.ts”做个对比——后者已经做了按 turn 的重建和分页，延迟 / 成本 / 失败归因这层投影未必不能放在那里。如果两者确实不好共用一个 builder，在 PR 描述里写一句原因会对 reviewer 有帮助。

   以上都不构成对这份 RFC 的阻塞。非目标划得很好，PR 1 保持纯粹且可独立合并也是个好的起点。
3. Astro-Han

   assigned

   ARE404

   on Jul 29, 2026
4. ARE404 commented on Jul 30, 2026

   

   ARE404

   on Jul 30, 2026

   ContributorAuthor

   More actions

   Thanks — this is exactly the kind of grounding I was hoping for, and it tightens PR 1 nicely. Taking your points in order.

   Q1 — Inspector as a fifth SessionWorkbar tab. Agreed, and thanks for the AgentGraphPanel warning — a graph-gated surface would defeat the purpose. I'll extend SessionWorkbarTab (session-workbar-layout.ts:8) with 'inspector' and leave readSessionWorkbarTab's persistence set as-is.

   Q2 + the pricing parameter. Both accepted, and they collapse into one decision: contract in packages/core, builder in packages/runtime, and the builder takes a resolved PricingSnapshot (pricing-store.ts:23) as a plain value rather than a reader. The caller awaits PricingAuthorityReader.snapshot() at the edge. That keeps PR 1 synchronous and pure and avoids pulling @maka/storage into @maka/runtime, so it stays clear of the dependency-boundary check. My "leaning runtime because it depends on pricing" reasoning is retracted — the snapshot-as-value shape removes the dependency entirely.

   Q4 + "extend agent-graph-timeline.ts?" I read through it before answering, and I think the honest answer is compose, not merge — for a reason worth stating explicitly in the PR description:

   • agent-graph-timeline.ts projects orchestration facts at run/operator altitude (supervisor_turn_*, schedule_committed, operator_provisioned, activation_*, supervisor_wake_*) from graph metadata, and is async/paginated.
   • The Inspector needs the intra-turn causal step trace (LLM → tool dispatch → result → compaction → retry → permission) projected from the RuntimeEvent content ledger — the granularity that lives inside a single activation / supervisor_turn node.

   So I don't think they can share one builder without conflating two altitudes. What I'd like to do instead is have the step projection consume the timeline as its outer scaffold: use buildAgentGraphTimeline / paginateAgentGraphTimeline for run/turn grouping, pagination, cursor, and AGENT_GRAPH_TIMELINE_SCHEMA_VERSION discipline (this is the reconstruction entry point from Q4), and hang TraceStep[] off each node rather than re-deriving run structure. Net: a new runtime-event-trace.ts builder, but it plugs into the existing timeline instead of duplicating it, and I'll say exactly that in the PR description. If you'd rather I prototype the step projection as an option on the timeline page first to prove they don't fit, I'm happy to.

   Cost in totals — agreed, the Inspector must not become a second cost authority. So totals carries only what the ledger is authoritative for: durationMs, tokens, compactions, retries. Per-session cost, if it appears at all, reads the existing telemetry aggregate (TelemetryIndexReader) for the session rather than self-summing steps.

   On reconciling the two paths, since you asked: the double-count you described comes specifically from replayed events, and the ledger already marks those — recoveryMode (replay_safe / idempotent / reconcile / reattach) plus the terminal-fact classification the read model uses. A per-step cost that sums only fresh billable invocations (i.e. excludes events carrying a recovery/replay marker) should agree with telemetry's single count rather than exceed it. I'd treat that as the mechanism, but keep per-step costUsd as an explicitly non-authoritative attribution hint derived from the passed-in snapshot — never the source for a session total. Two questions fall out of that:

   1. Would you rather I gate per-step costUsd out of PR 1 / PR 2 entirely and reintroduce it only once the fresh-invocation filter is proven to match telemetry on a replayed session? I'm fine either way; it keeps the first cut unambiguously non-authoritative.
   2. Is there a per-session entry point on the telemetry aggregate already, or should PR 2 add that read?

   Search — agreed it's two things, and both shrink. Structured predicates (jump to compaction, steps over $X, failed tools, by tool name) become caller-side Array.filter over TraceStep[]; no searchTrace API in the contract. Free-text reuses runtimeEventSearchText (runtime-event-history-search.ts:126). I'll stay off searchRuntimeEventHistory since it's a budgeted, policy.enabled-gated retrieval path, not a UI filter.

   Revised PR 1 scope, to confirm before I branch: a pure step-projection — RuntimeEventTrace contract in packages/core, builder in packages/runtime that consumes the agent-graph-timeline reconstruction plus a resolved PricingSnapshot, producing TraceStep[] with duration/token/status/recovery/failure-attribution and a ledger-authoritative totals (no cost). Node tests for step grouping, latency/token deltas, compaction-boundary detection, retry/recovery classification, failure attribution across a failed-tool → overflow → compact chain, and replayed-ledger dedup. No UI, no searchTrace, no cost authority.

   If that shape looks right, I'll open the PR 1 branch off current main.
5. Astro-Han commented on Jul 30, 2026

   

   Astro-Han

   on Jul 30, 2026

   Contributor

   More actions

   Two corrections first. Both are mine, and one of them currently points you at work that shouldn't be done. Then a larger finding that changes PR 1's scope.

   Corrections

   The replay double-count doesn't exist. Please drop the fresh-invocation filter.

   I wrote that a replayed LLM call appears twice in events but once in telemetry. That was inferred from naming, never verified, and it's wrong. runtime_events is keyed event_id PRIMARY KEY with UNIQUE (invocation_id, event_seq). "Replay" in this codebase means feeding existing events back into provider context, not re-appending them. Continuation asserts fresh invocation/run/turn identities and issues a genuinely new billable call, which both ledgers record once each. Compaction checkpoints synthesize an event carrying no actions at all. Ledger repair dedups on a content key that includes tokenUsage.

   There is nothing to filter, so your question 1 is moot in the direction it was asked. Sorry for the detour.

   Second: "consume buildAgentGraphTimeline" was bad advice, because it carries the same gate I warned you about. readAgentGraphTimelinePage requires a graphId and hard-asserts it (assertIdentity(input.graphId, 'graph id')), plus an AgentGraphTimelineMetadataStore. Plain chat sessions have no graph, so scaffolding on the timeline makes the Inspector graph-only. That is the AgentGraphPanel failure mode, one layer down. The per-session read boundary you want is RuntimeEventStore.readSessionRuntimeEvents(sessionId) (packages/core/src/runtime-event-store.ts:42), which is what the timeline itself consumes.

   Your "compose, not merge" altitude argument still holds. The timeline carries no intra-turn step kinds. It just isn't the scaffold.

   Per-step attribution isn't in the ledger

   This is the part that affects scope. From packages/runtime/src/ai-sdk-backend.ts:1591-1598:

   The send-level owner is completedStepUsage, which spans every Runtime loop step and retry.

   And RuntimeEventTokenUsage.runtimeSteps is documented as "Number of provider runtime/tool-loop steps represented by this usage."

   One aggregated usage event per send, carrying a field that states how many steps it stands for. Per-step latency and cost attribution, the headline of the RFC, is not derivable from the RuntimeEvent ledger. The granularity was never recorded there.

   The workaround doesn't hold either

   Per-attempt data does exist. recordProviderRequestAttempt (packages/runtime/src/agent-run.ts:292) writes provider_request_attempt_recorded into the AgentRun ledger with step, attempt, usage, latencyMs, timeToFirstTokenMs. It is readable today via listSessionRuns + readEvents, which context-diagnostics.ts already does. I designed an overlay around it and then abandoned it. The reasons are worth recording so nobody re-proposes it.

   The write path is built to lose data silently. In runtime-kernel.ts:2462-2477, two lines apart, recordProviderRequestCapture does if (!run) return Promise.reject(...) while recordProviderRequestAttempt does run?.recordProviderRequestAttempt(attempt). A late finalize after turn teardown vanishes without a trace, and the best-effort failure record can itself fail to write.

   The gaps then concentrate on the turns most worth inspecting. From model-adapter.ts:105: "Main-agent provider-call tracker. Auxiliary model calls intentionally omit it." Compaction's model calls produce no attempt records by design, so a long, compacted, expensive turn renders per-step detail that looks complete while silently omitting compaction cost.

   Absence is detectable; incompleteness is not. Nothing records an expected attempt count, and captures are deduped per (step, requestHash) across retries, so they can't serve as a lower bound either. A coverage flag that only fires when detail is entirely missing misses the dangerous case.

   AgentGraphTimelineCoverageLimitation is not the precedent I suggested. It is a static, always-listed set of by-design structural gaps, and it sits next to runtimeRecords: 'complete'. The attempt ledger can never honestly assert that counterpart.

   Per-step cost is unavailable regardless: ProviderRequestUsage carries token fields only, no costUsd (provider-request-telemetry.ts:9-19).

   A per-step view that under-reports on the most expensive turns, with no way to prove it is doing so, is worse than shipping nothing.

   Suggested revised scope for PR 1: per-turn

   Skeleton from readSessionRuntimeEvents, grouped per turn. Two things it has to handle regardless of any overlay: a turn can carry multiple aggregate usage events, and kernel-level compaction writes a phantom turn with input: 0, output: 0 (runtime-kernel.ts:816-822).

   Cost: surface the costUsd already stored on the usage event, and drop the pricing parameter entirely. You don't need it, PricingSnapshot lives in @maka/storage which is only a devDependency of @maka/runtime, and recomputing from a current snapshot drifts from the value stored at call time. Note that costUsd is optional. computeTokenUsageCostUsd returns undefined when the pricing lookup misses (ai-sdk-backend.ts:1979-1997), and pi and fake backends never produce it. Render that as "cost unavailable", never $0.

   On totals, I was wrong in the other direction too. The ledger doesn't over-count, it under-counts. provider_retry is barred from the ledger (agent-run.ts:482), and semantic compaction writes 0/0 there while recording real usage to telemetry (runtime-kernel.ts:816-822 against ai-sdk-compaction.ts:1150). So retries isn't derivable from a pure RuntimeEvent projection, and tokens should be labeled a ledger view with known gaps rather than ledger-authoritative.

   If you want attempts in PR 1 at all, make them a separate list labeled "observed provider attempts (coverage unverified)", never summed into turn cost or latency. Join on refs.providerRequestTraceId (runtime-event.ts:322), which exists precisely to link aggregate usage to physical attempts. Not turnId, which would attribute main-request attempts to compaction calls.

   Your question 2, the per-session telemetry entry point

   There isn't one. UsageQuery (packages/core/src/usage-stats/types.ts:5-12) has no sessionId, and the predicate in telemetry-repo.ts filters only time, slug, provider, model, and status. Rows do carry sessionId, and storage is file-based with in-memory filtering, so adding it needs no migration. But it changes the Usage authority's query contract and belongs to that surface rather than the Inspector PR.

   The bigger problem with reading per-session cost from telemetry: the pi backend records none at all. recordLlmCall has zero references in pi-agent-backend.ts. Those sessions show tokens in the ledger and $0 in telemetry. Worth settling before either surface promises a per-session cost number.

   None of this stops you from branching. Pure projection, contract in packages/core, builder in packages/runtime is still the right shape. The granularity promise is what needs to come down.

   中文对照

   先更正两处。都是我的问题，其中一处正把你引向不该做的工作。然后是一个我认为会改变 PR 1 范围的发现。

   更正

   重放双算不存在，请去掉那个 fresh-invocation filter。

   我写过「一次被重放的 LLM 调用在事件里出现两次、在 telemetry 里只有一次」。那是从命名推断的，没有验证过，是错的。runtime_events 以 event_id PRIMARY KEY 加 UNIQUE (invocation_id, event_seq) 为键。本仓库里的 "replay" 指把既有事件重新喂进 provider 上下文，不是重新 append。Continuation 会断言全新的 invocation/run/turn 身份并发起一次真实计费的新调用，两边账本各记一次。压缩检查点合成的事件根本不带 actions。账本修复按含 tokenUsage 的内容键去重。

   没有东西需要过滤，所以你的问题 1 在它被提出的那个方向上不成立。抱歉绕了这一圈。

   第二处：「消费 buildAgentGraphTimeline」是个坏建议，因为它带着我警告过你的同一个门禁。readAgentGraphTimelinePage 需要 graphId 并且硬断言（assertIdentity(input.graphId, 'graph id')），还需要 AgentGraphTimelineMetadataStore。普通聊天会话没有 graph，所以拿 timeline 做骨架会让 Inspector 变成 graph-only。那就是 AgentGraphPanel 的失败模式，只是下沉了一层。你要的 per-session 读边界是 RuntimeEventStore.readSessionRuntimeEvents(sessionId)（packages/core/src/runtime-event-store.ts:42），timeline 自己也是消费它的。

   你「compose, not merge」的高度论证依然成立。timeline 不携带任何 intra-turn step 类型，它只是不适合当骨架。

   逐步归因不在账本里

   这一条影响范围。packages/runtime/src/ai-sdk-backend.ts:1591-1598 的注释原文：

   The send-level owner is completedStepUsage, which spans every Runtime loop step and retry.

   而 RuntimeEventTokenUsage.runtimeSteps 的文档是 "Number of provider runtime/tool-loop steps represented by this usage"。

   一次 send 一个聚合用量事件，还带一个字段声明它代表几步。RFC 的头号卖点，逐步延迟与成本归因，无法从 RuntimeEvent 账本推导。这个粒度从来没有被记录在那里。

   绕法也不成立

   逐次尝试的数据确实存在。recordProviderRequestAttempt（packages/runtime/src/agent-run.ts:292）把 provider_request_attempt_recorded 写进 AgentRun 账本，带 step、attempt、用量、latencyMs、timeToFirstTokenMs。今天就能通过 listSessionRuns + readEvents 读出来，context-diagnostics.ts 已经在这么做。我围绕它设计了一版 overlay，然后放弃了。理由值得记下来，免得有人再提。

   写入路径是会无声丢数据的。runtime-kernel.ts:2462-2477 里，隔两行，recordProviderRequestCapture 是 if (!run) return Promise.reject(...)，而 recordProviderRequestAttempt 是 run?.recordProviderRequestAttempt(attempt)。turn 拆除后的 late finalize 不留痕迹地消失，而那个 best-effort 的失败记录本身也可能写不进去。

   缺口于是集中在最值得检查的 turn 上。model-adapter.ts:105："Main-agent provider-call tracker. Auxiliary model calls intentionally omit it." 压缩的模型调用按设计不产生 attempt 记录，所以一个很长、压缩过、很贵的 turn 会渲染出看起来完整、却静默漏掉压缩开销的逐步明细。

   「没有」可检测，「残缺」不可检测。没有任何地方记录期望的 attempt 数量，而 capture 按 (step, requestHash) 跨重试去重，也当不了下界。一个只在明细完全缺失时才触发的覆盖旗标，漏掉的正是危险的那种情况。

   AgentGraphTimelineCoverageLimitation 不是我建议的那种先例。它是一组静态、总是全量列出的设计性结构缺口，而且旁边就放着 runtimeRecords: 'complete'。attempt 账本永远无法诚实地断言那个对应物。

   逐步成本无论如何拿不到：ProviderRequestUsage 只有 token 字段，没有 costUsd（provider-request-telemetry.ts:9-19）。

   一个在最贵的 turn 上低报、且无法自证在低报的逐步视图，比什么都不做更糟。

   建议的 PR 1 修订范围：按轮次

   骨架用 readSessionRuntimeEvents，按 turn 分组。有两件事无论有没有 overlay 都得处理：一个 turn 可能带多个聚合用量事件，以及 kernel 级压缩会写出一个 input: 0, output: 0 的幻影 turn（runtime-kernel.ts:816-822）。

   成本：直接展示用量事件上已经存好的 costUsd，并彻底去掉 pricing 参数。你不需要它，PricingSnapshot 住在 @maka/storage，而后者只是 @maka/runtime 的 devDependency，而且用当前快照重算会和调用时存下的值漂移。注意 costUsd 是可选的，pricing 查不到时 computeTokenUsageCostUsd 返回 undefined（ai-sdk-backend.ts:1979-1997），pi 和 fake backend 从不产生它。这种情况渲染成「成本不可用」，绝不能显示 $0。

   关于 totals，我在另一个方向上也错了。账本不是多算，是少算。provider_retry 被禁止写入账本（agent-run.ts:482），语义压缩往账本写 0/0 却往 telemetry 记真实用量（runtime-kernel.ts:816-822 对 ai-sdk-compaction.ts:1150）。所以 retries 无法由纯 RuntimeEvent 投影得出，tokens 应该标成「有已知缺口的账本视图」而不是账本权威。

   如果你确实想把 attempts 放进 PR 1，就做成一个单独的列表，标注「observed provider attempts（覆盖度未经验证）」，绝不汇总进 turn 的成本或延迟。join 用 refs.providerRequestTraceId（runtime-event.ts:322），这个字段正是为把聚合用量连到物理尝试而定义的。不要用 turnId，那会把主请求的尝试算到压缩调用头上。

   你的问题 2，telemetry 的 per-session 入口

   没有。UsageQuery（packages/core/src/usage-stats/types.ts:5-12）没有 sessionId，telemetry-repo.ts 的谓词只过滤时间、slug、provider、model 和 status。行上确实带 sessionId，存储是文件加内存过滤，所以加这个过滤不需要迁移。但它改的是 Usage 权威的查询契约，应该归那个面而不是 Inspector 的 PR。

   从 telemetry 读 per-session 成本还有个更大的问题：pi backend 完全不记。pi-agent-backend.ts 里 recordLlmCall 零命中。那类会话在账本里有 token，在 telemetry 里是 $0。在任何一个面承诺 per-session 成本数字之前，这个得先定下来。

   以上都不妨碍你建分支。纯投影、契约放 packages/core、builder 放 packages/runtime 依然是对的形态。需要下调的是粒度承诺。
6. ARE404 commented on Jul 31, 2026

   

   ARE404

   on Jul 31, 2026

   ContributorAuthor

   More actions

   Status update, and a change of plan for this issue.

   Following up on the metering discussion: I'm pausing the pure-RuntimeEvent PR 1 described above. The conclusion I came to while mapping the ledgers is the one you'd already reached — the per-attempt truth isn't in RuntimeEvent, and a per-turn projection over it would ship a view that's knowingly incomplete on exactly the turns worth inspecting. Rather than build that and label around it, I'd rather fix the metering source first.

   So the order becomes: canonical model-call accounting first, Inspector second as a pure read model / UI over the new records. I'll open a separate issue for the accounting work since its scope is well past this one, and link it here — this issue stays as the Inspector surface and becomes its downstream consumer.

   One concrete finding from mapping the call paths, which supports routing all four through one seam. callKind today only has two values:

   // packages/core/src/usage-stats/types.ts:52, :97
   callKind?: 'main' | 'semantic_compact';

   provider retry and history compact have no kind at all. History compaction does make a real billed model call — history-compact-summarizer.ts calls generateText against the session model — so today it's spend that no ledger attributes. Four real call paths, two of them accounted. That gap is part of what the canonical record needs to close.

   One coordination question before I write the contract, and it's the sequencing one: #1649 changes persistence for these stores. It would be wasteful for that to land a schema and then have the accounting work change it, so I'd like to freeze the ModelCallAttempt contract first and let #1649 implement the final shape once. @Astro-Han — does that ordering work against #1649's timeline, or is it far enough along that I should target the schema it lands with instead?

   I'll keep the accounting issue scoped to the metering seam and the Usage authority it feeds, and stay off Runtime Host ownership/election/admission and the Desktop embedded writer path.
7. Astro-Han commented on Jul 31, 2026

   

   Astro-Han

   on Jul 31, 2026

   Contributor

   More actions

   The reordering makes sense to me. Fixing the metering source first and letting the Inspector be a pure read model over the new records is a better sequence than labeling around gaps, and keeping this issue as the surface consumer is a clean split. Two useful things before you write the contract: an update on the callKind state, and an answer to your sequencing question.

   callKind update

   Good news on one of the four paths: main moved recently under this discussion. #1637 (merged July 30) added history_compact as a third kind, and history-compact-summarizer.ts:172-176 now records its generateText call to telemetry with callKind: 'history_compact' and a stable callId. So history compaction is attributed now, at least on the telemetry side.

   That leaves the gap list for the accounting issue looking like this:

   • Provider retries still carry no kind anywhere. The main path writes one send-level telemetry record per call, aggregated across retries (ai-sdk-backend.ts:1826), and provider_retry is barred from the AgentRun ledger (agent-run.ts:482).
   • The RuntimeEvent ledger still attributes no compaction cost — the 0/0 phantom turn from the last round is unchanged.
   • The pi backend records no telemetry at all; recordLlmCall has zero references in pi-agent-backend.ts.

   Smaller than "four paths, two accounted", but all three are worth closing.

   Sequencing against #1649

   Short version: you don't need to wait, and #1649 doesn't need to wait for you.

   #1649 is a persistence cutover, not a schema change — the execution sequence proposed there keeps public store APIs and semantics stable through the migration. Its first step (#1655, runtime events) is already merged, and the AgentRun stores are planned for a later step.

   So the two efforts compose:

   • If ModelCallAttempt becomes a new AgentRunEvent type — extending the seam provider_request_attempt_recorded already lives in — there's nothing to coordinate. Freeze it whenever it's ready; the migration carries whatever event types exist.
   • If it needs its own table or store, then it's worth syncing up so it only gets implemented once. In that case let's just talk timing on the accounting issue when you open it.

   Either way, I'd suggest cc'ing @likun666661 there, who is driving the #1649 sequence, so the two tracks can see each other.

   One thing worth folding into the contract while you're at it: the current write path uses run?.recordProviderRequestAttempt(attempt) (runtime-kernel.ts:2462-2477), which drops late writes silently. The new contract is a good opportunity to make attempt writes fail loudly instead.

   And agreed on staying off Runtime Host ownership and the Desktop writer path — that boundary is right.

   中文对照

   调序我觉得合理。先修计账源头，让 Inspector 作为新记录之上的纯读模型，好过围着缺口贴标签；这个 issue 保留为消费方也是干净的切分。写契约之前同步两件事：callKind 的最新状态，和你排序问题的回答。

   callKind 的最新状态

   四条路径里有一条有好消息：main 最近正好动了这块。#1637（7 月 30 日合并）把 history_compact 加成了第三个 kind，history-compact-summarizer.ts:172-176 现在会把它的 generateText 调用记入 telemetry，带 callKind: 'history_compact' 和稳定的 callId。所以 history compaction 现在至少在 telemetry 侧是有归属的。

   这样记账 issue 的缺口清单变成：

   • Provider retry 仍然在任何地方都没有 kind。主路径每次调用只写一条跨重试聚合的 send 级 telemetry 记录（ai-sdk-backend.ts:1826），provider_retry 被禁止写入 AgentRun 账本（agent-run.ts:482）。
   • RuntimeEvent 账本仍不归属任何压缩成本——上一轮说的 0/0 幻影 turn 原样未动。
   • pi backend 完全不记 telemetry；pi-agent-backend.ts 里 recordLlmCall 零命中。

   比「四条路径只记两条」小，但三个都值得关掉。

   与 #1649 的排序

   短版本：你不用等它，它也不用等你。

   #1649 是持久化 cutover，不是 schema 变更——那边提出的执行序列会在迁移全程保持 store 公共 API 和语义稳定。它的第一步（#1655，runtime events）已经合并，AgentRun 相关 store 排在后面的步骤。

   所以两边可以组合：

   • 如果 ModelCallAttempt 做成新的 AgentRunEvent 类型——扩展 provider_request_attempt_recorded 已经在用的那道缝——那就没什么要协调的。什么时候定稿都行，迁移会带上当时存在的所有事件类型。
   • 如果它需要独立的表或 store，那值得对一下时间，让它只被实现一次。这种情况就等你开记账 issue 之后在那边聊时机。

   无论哪种，建议在记账 issue 上 cc @likun666661——#1649 的序列是他在推——让两条线互相可见。

   顺手值得并进契约的一件事：现在的写入路径是 run?.recordProviderRequestAttempt(attempt)（runtime-kernel.ts:2462-2477），会把迟到的写入无声丢掉。定新契约正好是把 attempt 写入改成大声失败的机会。

   另外同意不碰 Runtime Host 归属和 Desktop writer 路径——这条边界是对的。
8. ARE404 commented on Jul 31, 2026

   

   ARE404

   on Jul 31, 2026

   ContributorAuthor

   More actions

   Accounting issue is open: #1679 — canonical ModelCallAttempt per real provider request, feeding the Host Usage authority. This issue stays as the Inspector surface and consumes those records once they land.
9. ARE404 commented on Aug 3, 2026

   

   ARE404

   on Aug 3, 2026

   ContributorAuthor

   More actions

   Picking this back up now that the accounting work it was waiting on is merged (#1687, #1755, #1877). Short version of what changed and the revised PR 1 shape, since the reason this issue was paused no longer holds.

   What the pause was about, and what closed it

   Your finding last round killed the RFC's headline: one aggregate usage event per send, runtimeSteps counting how many steps it stood for, so per-step latency and cost were not derivable from RuntimeEvent. That is fixed at the source rather than labeled around. ModelCallAttempt now records one canonical record per physical provider request, carrying step, attempt, latencyMs, timeToFirstTokenMs, usage with a usageBasis, and costUsd with a costBasis frozen at call time plus the pricingRates it was computed against.

   Three of the four gaps you listed are closed:

   • Provider retries — each attempt is its own record with an attempt ordinal sharing one logicalCallId, so retries are visible rather than aggregated away.
   • Compaction cost — history_compact and semantic_compact both settle through the same seam (feat(metering): route compaction model calls through the canonical seam #1877), so the 0/0 phantom turn is no longer the whole story of a compacted turn.
   • Silent write loss — AgentRun.recordModelCallAttempt rejects on failure instead of resolving-and-reporting, and the Usage projection is written only after the authority holds the record.

   Still open, and I am not proposing to close them here: goal_evaluation still writes only the frozen table (it has no run or turn identity at the Host layer — needs its own design), and the pi backend produces no ModelCallAttempt at all — zero references, it emits token_usage events only.

   Revised PR 1

   Same discipline as agreed: pure projection, contract in packages/core, builder in packages/runtime, no I/O, node-tested. Two changes from the shape we settled in July:

   The read boundary is the AgentRun stream, not the Usage ledger. listSessionRuns(sessionId) + readEvents(sessionId, runId) filtered to model_call_attempt_recorded, joined with readSessionRuntimeEvents(sessionId) for causal structure (tool dispatch, permission decision, error, compaction boundary). The Usage table is a range-queried read model with no session predicate; the stream is the authority and is already shaped per session and run. context-diagnostics.ts reads it the same way.

   No pricing parameter, and no second cost authority. Cost comes off the record as stored, with its costBasis. Unpriced renders "cost unavailable", never $0 — that rule is enforced in the contract already. Because Settings → Usage now aggregates these same records, a per-session total is the same authority rather than a competing one. That is the part I would most like you to confirm, since it is the inversion of your earlier warning: summing a session's attempts and showing that as its cost is now, as far as I can tell, reading one authority at a different scope rather than creating a second. If you see a path where the two still disagree, I would rather know before I build the totals.

   Coverage is stated, not implied. A session whose backend emits no canonical records is detectable at the session level — zero attempts alongside non-zero token_usage events means "this backend does not report per-call detail", which is exactly the pi case. The projection reports that as a coverage fact rather than rendering an empty timeline that looks like an idle session. Per-turn ledger facts (duration, compaction boundaries, tool structure) still render for those sessions.

   Tests: step and attempt grouping, retry ordinals sharing a logicalCallId, compaction attribution inside a turn, failure attribution across failed-tool → overflow → compact, unpriced and missing-usage records, and a pi-shaped session with no canonical records at all.

   Still no searchTrace in the contract, still a fifth SessionWorkbar tab for PR 2, still off Runtime Host ownership and the Desktop writer path.

   I will start on the contract and builder now; the one thing I would hold for your answer is the per-session cost total above, which is easy to leave out of the first cut if you would rather it wait.
10. ARE404 commented on Aug 3, 2026

    

    ARE404

    on Aug 3, 2026

    ContributorAuthor

    More actions

    PR 1 is up: #1956 — the pure trace projection (contract in packages/core, builder in packages/runtime, 9 node tests, no UI).

    The per-session cost question from my last comment is called out in the PR description as the one thing I'd rather have confirmed than assert. Dropping TraceTotals.costUsd if you disagree touches one field and one test.
11. ARE404

    added 2 commits that reference this issue on Aug 3, 2026

    feat(inspector): project a per-session causal trace over both ledgers

    ...

    77afd2c

    feat(inspector): project a per-session causal trace over both ledgers

    ...

    d44c2f6
12. Astro-Han

    added a commit that references this issue on Aug 3, 2026

    feat(inspector): project a per-session causal trace over both ledgers (…

    ...

    302c54d
13. ARE404

    added a commit that references this issue on Aug 3, 2026

    feat(inspector): surface the session trace as a workbar tab

    ...

    bc5e6ee
14. Astro-Han

    added a commit that references this issue on Aug 3, 2026

    feat(inspector): surface the session trace as a workbar tab (#2018)

    ...

    1a508c3
15. ARE404 commented on Aug 4, 2026

    

    ARE404

    on Aug 4, 2026

    ContributorAuthor

    More actions

    PR 3 is up: #2115 — the filter half (free text, step kind, failed-only, cost floor), pure predicates over the projected trace with no query API on the contract, per the shape we settled here.

    Replay step-through is deliberately not in it. It needs a decision about what "reconstructed state at a boundary" means against the ledger as it stands, and what a read-only surface may claim about it — I'd rather settle that here first than guess at it.
16. ARE404

    added a commit that references this issue on Aug 4, 2026

    feat(inspector): filter the trace by what a reader actually asks

    ...

    fcba431
17. Astro-Han

    added a commit that references this issue on Aug 4, 2026

    feat(inspector): filter the trace by what a reader actually asks (#2115)

    ...

    0e9fa90
18. Astro-Han

    added

    enhancementNew feature or requestNew feature or request

    on Aug 20, 2026
19. Astro-Han

    added the

    Feature

    issue type on Aug 29, 2026