Skip to content

feat: model-call cost and token usage tracking rolled into vouch metrics #326

Description

@plind-junior

right now vouch has no way to answer "what did this kb spend on model calls, and where?". kb.synthesize is deterministic today (src/vouch/synthesize.py), but the wire shape reserves an opt-in generative backend (llm=True currently raises), and kb.reindex_embeddings already drives embedder passes (src/vouch/embeddings/base.py). the moment any of those calls touch a paid model, an operator wants token counts and cost broken down by operation over a time window — the same audit-derived surface that shipped in #192, where vouch metrics already reports approval rate, lag percentiles and staleness (src/vouch/metrics.py). this issue adds per-call usage recording and folds it into that existing surface. it is a distinct axis from #101 (approval/citation/staleness rates): those measure review-gate health, this measures model spend. no retrieval behavior changes.

proposed surface

record one telemetry record per model call as an audit event whose event verb is usage.<op> (e.g. usage.synthesize, usage.reindex_embeddings), carrying the counts in the existing AuditEvent.data dict — {op, model, input_tokens, output_tokens, cost_usd, ms}. this reuses the append-only .vouch/audit.log.jsonl that metrics.compute already streams; no new on-disk state, no schema migration, no new artifact type.

  • callers emit the record where the call happens: the (future) generative branch of synthesize() and each embedder batch in the reindex path. a deterministic kb.synthesize (llm=False) records nothing.
  • extend metrics.compute to fold usage.* events into a new usage block in the stable Metrics.to_dict() schema: totals plus a by_op breakdown ({op: {calls, input_tokens, output_tokens, cost_usd}}), all windowed by since/until like every other audit-derived metric.
  • vouch metrics --json gains the usage block; render_prometheus gains vouch_usage_cost_usd and vouch_usage_tokens gauges labelled by op and model. bump SCHEMA_VERSION (currently 1) since this is an additive-but-versioned field, and document the block in docs/metrics.md.
  • config: metrics.cost.<model> = <usd per 1k input>/<usd per 1k output> in .vouch/config.yaml so cost derives from operator-supplied rates, not hardcoded values. an absent rate yields a null cost_usd and only token counts are reported.

this is telemetry, not a new kb.* method — it reads through vouch metrics, so no new registration across server.py / jsonl_server.py / capabilities.METHODS / cli.py is required. if a dedicated read method (kb.usage) is later wanted, that is a separate issue and would need all four sites plus tests/test_usage.py.

review gate & scope

usage records are pure observation of model calls; they create no claims, pages, entities or relations, so they neither route through proposals.approve() nor need to. no propose_* is emitted and nothing is approved — the review gate governs kb-artifact writes, and this writes none. the recording is a data-only audit event, so storage.py stays pure i/o and the fold-up logic lives in metrics.py alongside the other derived rollups. everything stays local-first: rates come from local config, records live in the local audit log, and the rollup is computed on read.

acceptance criteria

  • a usage.<op> audit event is written with {op, model, input_tokens, output_tokens, cost_usd, ms} in data at each model-call site (generative synthesize branch, reindex embedder batches).
  • deterministic kb.synthesize (llm=False) records no usage event.
  • metrics.compute folds usage.* events into a windowed usage block with totals and a by_op breakdown; the block respects since/until.
  • Metrics.to_dict() exposes the usage block, SCHEMA_VERSION is bumped, and docs/metrics.md documents the new fields.
  • render_prometheus emits vouch_usage_cost_usd and vouch_usage_tokens gauges labelled by op and model.
  • cost derives from metrics.cost.<model> config; a missing rate yields null cost_usd with token counts still present.
  • tests cover the fold-up (windowing, by-op split, null-cost path) and assert no retrieval or synthesize output changes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    embeddingsembedding-backed retrievalenhancementNew feature or requestsize: S50-199 changed non-doc lines

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions