Skip to content

product: ship a representative coding-agent evidence fixture #250

Description

@Siddhant-K-code

Goal

Ship one versioned, non-sensitive coding-agent session fixture that exercises AgentTrace's core review workflow and serves as a common regression input for setup, evidence health, replay, UI, sharing, and export.

Problem

The repository has many unit fixtures and can freeze session behavior, but it lacks one canonical end-to-end evidence record that a contributor or user can inspect and reproduce.

Without it, setup claims, screenshots, provider matrices, UI behavior, and backend mappings can each be demonstrated with different data and drift apart.

Required scenario

The fixture must contain:

  • a clear task and session boundary;
  • file reads and writes;
  • shell commands;
  • at least one test run;
  • a tool failure;
  • a retry and recovery;
  • a human annotation or correction;
  • provider/source attribution;
  • redaction markers without real secrets;
  • parent/child relationships where supported;
  • one deliberately missing or ambiguous event; and
  • a documented final effect.

Use synthetic or repository-owned sample content only.

Artifacts

  • source event fixture in the canonical AgentTrace format;
  • expected session metadata;
  • expected evidence-health result from product: calculate and surface per-session evidence health #240;
  • expected replay/timeline/tree structure;
  • expected minimized share output;
  • expected OTLP mapping assertions;
  • a deterministic generator or refresh procedure; and
  • a short factual walkthrough explaining what is observed, derived, missing, and provider-specific.

Consumers

The same fixture must be usable by:

Boundaries

  • No customer or maintainer-private trace data.
  • No hidden reasoning or “complete ground truth” claim.
  • Do not hand-edit generated expected files without updating their source.
  • Provider-specific variants remain explicit rather than flattened.

Acceptance criteria

  • The fixture and generation/refresh process are committed and documented.
  • It includes success, failure, retry, recovery, effect, correction, redaction, and a known capture gap.
  • Expected source, health, relationship, and export assertions are machine-tested.
  • CLI replay and the local review UI render the same ordered facts.
  • The fixture contains no real secrets, personal data, or customer content.
  • Screenshots and walkthroughs state the exact AgentTrace version and fixture version.
  • CI detects incompatible fixture or schema changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions