You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
right now vouch has no way to answer "what did this kb spend on model calls, and where?". kb.synthesize is deterministic today (src/vouch/synthesize.py), but the wire shape reserves an opt-in generative backend (llm=True currently raises), and kb.reindex_embeddings already drives embedder passes (src/vouch/embeddings/base.py). the moment any of those calls touch a paid model, an operator wants token counts and cost broken down by operation over a time window — the same audit-derived surface that shipped in #192, where vouch metrics already reports approval rate, lag percentiles and staleness (src/vouch/metrics.py). this issue adds per-call usage recording and folds it into that existing surface. it is a distinct axis from #101 (approval/citation/staleness rates): those measure review-gate health, this measures model spend. no retrieval behavior changes.
proposed surface
record one telemetry record per model call as an audit event whose event verb is usage.<op> (e.g. usage.synthesize, usage.reindex_embeddings), carrying the counts in the existing AuditEvent.data dict — {op, model, input_tokens, output_tokens, cost_usd, ms}. this reuses the append-only .vouch/audit.log.jsonl that metrics.compute already streams; no new on-disk state, no schema migration, no new artifact type.
callers emit the record where the call happens: the (future) generative branch of synthesize() and each embedder batch in the reindex path. a deterministic kb.synthesize (llm=False) records nothing.
extend metrics.compute to fold usage.* events into a new usage block in the stable Metrics.to_dict() schema: totals plus a by_op breakdown ({op: {calls, input_tokens, output_tokens, cost_usd}}), all windowed by since/until like every other audit-derived metric.
vouch metrics --json gains the usage block; render_prometheus gains vouch_usage_cost_usd and vouch_usage_tokens gauges labelled by op and model. bump SCHEMA_VERSION (currently 1) since this is an additive-but-versioned field, and document the block in docs/metrics.md.
config: metrics.cost.<model> = <usd per 1k input>/<usd per 1k output> in .vouch/config.yaml so cost derives from operator-supplied rates, not hardcoded values. an absent rate yields a null cost_usd and only token counts are reported.
this is telemetry, not a new kb.* method — it reads through vouch metrics, so no new registration across server.py / jsonl_server.py / capabilities.METHODS / cli.py is required. if a dedicated read method (kb.usage) is later wanted, that is a separate issue and would need all four sites plus tests/test_usage.py.
review gate & scope
usage records are pure observation of model calls; they create no claims, pages, entities or relations, so they neither route through proposals.approve() nor need to. no propose_* is emitted and nothing is approved — the review gate governs kb-artifact writes, and this writes none. the recording is a data-only audit event, so storage.py stays pure i/o and the fold-up logic lives in metrics.py alongside the other derived rollups. everything stays local-first: rates come from local config, records live in the local audit log, and the rollup is computed on read.
acceptance criteria
a usage.<op> audit event is written with {op, model, input_tokens, output_tokens, cost_usd, ms} in data at each model-call site (generative synthesize branch, reindex embedder batches).
deterministic kb.synthesize (llm=False) records no usage event.
metrics.compute folds usage.* events into a windowed usage block with totals and a by_op breakdown; the block respects since/until.
Metrics.to_dict() exposes the usage block, SCHEMA_VERSION is bumped, and docs/metrics.md documents the new fields.
render_prometheus emits vouch_usage_cost_usd and vouch_usage_tokens gauges labelled by op and model.
cost derives from metrics.cost.<model> config; a missing rate yields null cost_usd with token counts still present.
tests cover the fold-up (windowing, by-op split, null-cost path) and assert no retrieval or synthesize output changes.
right now vouch has no way to answer "what did this kb spend on model calls, and where?".
kb.synthesizeis deterministic today (src/vouch/synthesize.py), but the wire shape reserves an opt-in generative backend (llm=Truecurrently raises), andkb.reindex_embeddingsalready drives embedder passes (src/vouch/embeddings/base.py). the moment any of those calls touch a paid model, an operator wants token counts and cost broken down by operation over a time window — the same audit-derived surface that shipped in #192, wherevouch metricsalready reports approval rate, lag percentiles and staleness (src/vouch/metrics.py). this issue adds per-call usage recording and folds it into that existing surface. it is a distinct axis from #101 (approval/citation/staleness rates): those measure review-gate health, this measures model spend. no retrieval behavior changes.proposed surface
record one telemetry record per model call as an audit event whose
eventverb isusage.<op>(e.g.usage.synthesize,usage.reindex_embeddings), carrying the counts in the existingAuditEvent.datadict —{op, model, input_tokens, output_tokens, cost_usd, ms}. this reuses the append-only.vouch/audit.log.jsonlthatmetrics.computealready streams; no new on-disk state, no schema migration, no new artifact type.synthesize()and each embedder batch in the reindex path. a deterministickb.synthesize(llm=False) records nothing.metrics.computeto foldusage.*events into a newusageblock in the stableMetrics.to_dict()schema: totals plus aby_opbreakdown ({op: {calls, input_tokens, output_tokens, cost_usd}}), all windowed bysince/untillike every other audit-derived metric.vouch metrics --jsongains theusageblock;render_prometheusgainsvouch_usage_cost_usdandvouch_usage_tokensgauges labelled byopandmodel. bumpSCHEMA_VERSION(currently 1) since this is an additive-but-versioned field, and document the block indocs/metrics.md.metrics.cost.<model> = <usd per 1k input>/<usd per 1k output>in.vouch/config.yamlso cost derives from operator-supplied rates, not hardcoded values. an absent rate yields a nullcost_usdand only token counts are reported.this is telemetry, not a new
kb.*method — it reads throughvouch metrics, so no new registration acrossserver.py/jsonl_server.py/capabilities.METHODS/cli.pyis required. if a dedicated read method (kb.usage) is later wanted, that is a separate issue and would need all four sites plustests/test_usage.py.review gate & scope
usage records are pure observation of model calls; they create no claims, pages, entities or relations, so they neither route through
proposals.approve()nor need to. nopropose_*is emitted and nothing is approved — the review gate governs kb-artifact writes, and this writes none. the recording is a data-only audit event, sostorage.pystays pure i/o and the fold-up logic lives inmetrics.pyalongside the other derived rollups. everything stays local-first: rates come from local config, records live in the local audit log, and the rollup is computed on read.acceptance criteria
usage.<op>audit event is written with{op, model, input_tokens, output_tokens, cost_usd, ms}indataat each model-call site (generative synthesize branch, reindex embedder batches).kb.synthesize(llm=False) records no usage event.metrics.computefoldsusage.*events into a windowedusageblock with totals and aby_opbreakdown; the block respectssince/until.Metrics.to_dict()exposes theusageblock,SCHEMA_VERSIONis bumped, anddocs/metrics.mddocuments the new fields.render_prometheusemitsvouch_usage_cost_usdandvouch_usage_tokensgauges labelled byopandmodel.metrics.cost.<model>config; a missing rate yields nullcost_usdwith token counts still present.