TraceSight

English · 中文

Offline retry-safety analyzer for AI/platform engineers who already have incident traces. Import two recorded attempts, apply a small declarative side-effect policy, get a deterministic HIGH / MEDIUM / SAFE verdict when the retry may have repeated the same business write.

Not a trace store, agent runtime, or eval suite. Analysis never executes models or tools from the imported trace. HIGH is risk, not proof of a double commit. Package 0.4.0 is engineering complete; external product validation is still open (validation/PROTOCOL.md).

---

Try it

From a clone. The package is not published as tracesight on PyPI:

pip install -e ".[web]"

tracesight analyze-retry \
  examples/otlp_retry_attempt_1.json \
  examples/otlp_retry_attempt_2.json \
  --policy examples/failures/otlp-retry-policy.json

Expect HIGH (POSSIBLE_DUPLICATE_SIDE_EFFECT) on the public duplicate-refund fixture (examples/failures/DUPLICATE_REFUND.md). That is a risk finding from trace evidence — not a claim that money moved twice.

Windows desktop lab (optional, after the same install): start.vbs or start.bat. Electron owns tracesight serve and kills the API on quit.

---

What it decides

An operation fingerprint is SHA-256 over the canonical tool name plus the policy-selected identity fields. Arguments the policy does not name do not affect the fingerprint. No LLM fills evidence gaps.

Earlier evidence	Later evidence	Verdict
Output missing / error without commit evidence	Same operation executes	HIGH
Successful / committed output	Same operation executes	HIGH
Explicit pre-commit failure	Same operation executes	SAFE
Any earlier outcome	Retry returns already_processed	SAFE
Required identity field missing	Cannot correlate	MEDIUM
Read-only tool or different fingerprint	No duplicate evidence	SAFE

HIGH is risk, not proof that an external system committed twice. Recommended action: reconcile + enforce idempotency. Wording stays “possible duplicate side effect.”

{
  "schema_version": "tryloop.retry-policy/v1",
  "tools": {
    "issue_refund": {
      "side_effect": true,
      "identity_fields": ["order_id"]
    }
  }
}

Wire JSON may still say tryloop.* for artifact compatibility. The product name is TraceSight.

Import (fail closed otherwise): OTLP / OpenInference, OpenAI Agents SDK, Langfuse Observations API v2.

tracesight import-trace examples/openinference_trace.json \
  --output runs/support-incident.json --require-ready
tracesight inspect-incident runs/support-incident.json

analyze-retry exits 3 when the configured risk threshold is met (CI gate); usage 2; runtime 1. Use --fail-on never for an exploratory report.

Aliases tracesight, harness, agent-eval-harness, and historical tryloop hit the same CLI.

---

What it is not

• Not a Langfuse / Phoenix / LangSmith replacement. Those store spans. This answers one retry-safety question after you already have traces.
• Not Neural-Cost Recorder (model-swap Decision Brief) and not TokSight (re-run your workflow N times). Input here is incidents that already happened.
• Not validated on a real customer incident yet. Do not quote the historical v0.3 AUC 0.894 as TraceSight’s current score.
• Not a cloud SaaS or “New Relic for agents.”
• Batch Run N of user workflow code is a separate, incomplete trust boundary (batch_runner.py). Do not confuse it with offline import analysis.

Contract: docs/product/MVP_PRD.md · docs shell: frontend/public/docs/viewer.html.

---

Local Lab

Electron 1280×800 frameless window (Vite · React 19 · TypeScript · Tailwind v4). Routes: Home · Analyze · Batch inspect · Docs.

Dev split (optional):

tracesight serve --no-static --port 8765
cd frontend && npm run dev

---

Tests

python -m pytest -q
python -m ruff check src tests verification
cd frontend && npm run build

test_live_agent may need optional OTel extras. Do not hard-code a test count.

---

Historical: v0.3 structural evaluator

Retained under verification/v0.3_meta_eval/ as engineering evidence. Not the 0.4 acceptance metric.

v0.3 measured ROC-AUC 0.894 on a 100-entry corpus that is 85% producer-designed, and honest-FAILed its own ≥ 0.9 gate. That is self-consistency of a rule set against the corpus it was written for — not an external catch rate. See verification/v0.3_meta_eval/REPORT_V3.md.

The old harness (verify_trajectory, agent-eval-harness run --corpus …) remains importable for that reproduction.

---

License

MIT © 2026 TraceSight.

Do not commit customer traces, PII, or API keys. runs/ and verification/**/llm_logs/ are gitignored for a reason.