Skip to content

Record what each autonomous run consumes, privately - #9

Merged
drevendev merged 1 commit into
masterfrom
chore/usage-telemetry
Sep 3, 2026
Merged

drevendev merged 1 commit into
masterfrom
chore/usage-telemetry

Conversation

@drevendev

Copy link
Copy Markdown
Owner

Closes #8

Achieved outcome

Each author and acceptor run now writes one JSON record of its own consumption into
the private repository drevendev/zen-telemetry. The public repository stores no
usage data at all.

Changed artifacts

Artifact Change
scripts/record_usage.py Reads the execution file, extracts usage, writes one record to the ledger
scripts/tests/test_record_usage.py Negative controls for the extraction and the failure paths
.github/workflows/zendev-author.yml id: claude on the action step, plus a recording step that always runs
.github/workflows/zendev-acceptor.yml The same

The ledger repository was created separately and already holds its README, the rollup
script, and a daily workflow that aggregates records into summaries/ and a monthly
Issue.

Acceptance criteria

  • Each run writes exactly one record, including failed runs — the step is if: always()
  • Unknown usage is null and marked usage_source: unavailable, never zero
  • A missing token or unreachable ledger warns and exits 0
  • Nothing about usage is written to this public repository
  • Tests cover result extraction, the summed fallback, the unknown case, and three malformed-input paths

Checks

Check Outcome Evidence
python -m unittest discover -s scripts/tests passed 14 tests, 0 failures
Workflow YAML parse passed all four workflows parse
build-and-test, policy-guard see checks tab
Recording a real run end to end not_run requires an armed loop; no author or acceptor run has ever executed

Not checked

The whole path. No model run has happened yet, so the shape of the execution file is
taken from the action source, not observed. If the extraction is wrong, the first
armed run will record usage_source: unavailable rather than silently reporting
plausible-looking numbers — that is the failure mode this was designed for.

Assumptions and unknowns

  • Established: the action sets execution_file to a JSON array of messages, and
    the formatter in its source reads usage.input_tokens,
    usage.cache_creation_input_tokens, usage.cache_read_input_tokens and
    usage.output_tokens.
  • Assumption: the final result message carries cumulative usage. If it does not,
    the summed fallback covers it, and usage_source says which was used.
  • Unknown: whether total_cost_usd is populated at all under subscription
    authentication. If absent it is recorded as null and excluded from totals.

Highest-risk area for review

The cross-repository token. A write token for another repository now exists in the
environment of a public repository. It is scoped to one private repository that holds
no product; policy-guard refuses to let a workflow change ride along inside a
product diff; and the acceptor may not merge workflow changes that widen authority.
Review whether that is a trade you want before merging.

Remaining gate

Requires the secret TELEMETRY_PAT (fine-grained, only drevendev/zen-telemetry,
Contents read and write). Until it exists the step warns and records nothing, which is
the intended degraded behavior rather than a failure.

🤖 Generated with Claude Code

One JSON record per run into a private ledger repository. The public repository keeps
no usage data, because its Actions logs and artifacts are public with it.

Unknown usage is written as null and marked, so the rollup can exclude it rather than
reporting a run that appeared to cost nothing. The step never fails a run: a loop that
stops over bookkeeping is worse than a gap in the bookkeeping.

Refs #8

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@drevendev drevendev added priority:normal Default planned priority type:process Development workflow, governance, or coordination change area:tooling CI, scripts, guards, developer tooling labels Sep 3, 2026
@drevendev
drevendev merged commit e8f8296 into master Sep 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:tooling CI, scripts, guards, developer tooling priority:normal Default planned priority type:process Development workflow, governance, or coordination change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Record what each autonomous run consumes, privately

1 participant