audit: add first-run audit report and e2e test suite#591
Open
ojongerius wants to merge 4 commits into
Open
Conversation
…test Time-boxed adoption audit of the Python SDK first run from user-facing docs only. Adds a structured paper-cut report, a self-contained pytest e2e test codifying the happy path (in-process create/sign/verify + chain, emitter round-trip vs a live daemon, no-daemon silent-drop contract), and a drafted CI workflow example. No SDK or product code changed. https://claude.ai/code/session_01DHDfJc5djJxshX8bxuxvha
Re-ran the first-run scenario after the weekend releases. WAL gap (#8) is fixed for the remote/HTTP path via the new emitters package (WalEmitter, verified retain+replay). But the socket emitter was renamed Emitter -> DaemonEmitter and the top-level Emitter is now a Protocol, breaking 0.9.0 call sites and every daemon-setup doc example; PyPI still serves 0.9.0. Updates the report with a fixed/open/new delta, fixes the e2e test to the 0.10.0 API (7 tests incl. WAL retain/replay + runtime_checkable footgun), and corrects the CI example. No SDK/product code changed. https://claude.ai/code/session_01DHDfJc5djJxshX8bxuxvha
0.10.0 is published and a fresh pip install resolves to it by default; the earlier 0.9.0 reading was a stale local pip cache. This makes the broken daemon-setup Emitter(socket_path=...) examples live-facing. https://claude.ai/code/session_01DHDfJc5djJxshX8bxuxvha
ojongerius
force-pushed
the
claude/wizardly-clarke-P0G9k
branch
from
4725821 to
a0d15ff
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add a comprehensive first-run audit of the Python SDK (v0.9.0 → v0.10.0) documenting the user experience for a brand-new adopter, including:
audit/AUDIT_REPORT.md: detailed audit findings across install, local collector (daemon), and CI paths, with a re-audit delta for v0.10.0 covering fixes, regressions, and open issuesaudit/test_first_run_e2e.py: 7 reproducible tests codifying the documented paths (in-process happy path, 2-link chain, daemon round-trip, silent-drop contract, WAL durability, and protocol footguns)audit/agent-ci-example.yml: drafted GitHub Actions workflow showing how to stand up the daemon and verify receipts in CI (not wired into.github/per repo safety rules)Why
The audit surfaces critical gaps in the first-run experience:
Breaking changes not flagged: v0.10.0 renamed
Emitter→DaemonEmitterand made top-levelEmitteran un-instantiable Protocol; 0.9.0 code now raisesTypeError. The CHANGELOG buries this under Added rather than Breaking Changes.Docs lag the code: daemon-setup docs still show
Emitter(socket_path=…)(now broken) and reference v0.8.0a2; PyPI README never mentions the daemon or emitter exist, forcing adopters to read source.Silent data loss on local path:
DaemonEmitteris fire-and-forget with no buffering, retention, or signal when the daemon is down — a first-run dev emitting before the daemon starts loses all receipts with zero indication.WAL fix incomplete: v0.10.0's new
WalEmitterfixes remote (HTTP) delivery durability but cannot wrapDaemonEmitterdue to aruntime_checkableProtocol signature mismatch, leaving the local path unprotected.No CI recipe: adopters must invent daemon orchestration themselves, rediscovering the
/tmpsafe-set guard and other sharp edges.The audit codifies these findings as reproducible tests and provides a concrete CI example, enabling the team to track regressions and prioritize fixes.
Checklist
Security
https://claude.ai/code/session_01DHDfJc5djJxshX8bxuxvha