Skip to content

reviewer-pack.md

Latest commit

 

History

History
138 lines (106 loc) · 10.5 KB

reviewer-pack.md

File metadata and controls

138 lines (106 loc) · 10.5 KB
Raw

Reviewer Pack

This is the top-level reviewer pack for telemetry-lab.

Use it as a short, no-guessing path through the repository. The project is a local, file-based detection workflow lab: it is not a SIEM, not a dashboard, and not a production monitoring platform.

What to Review

Start with the stable demo matrix in docs/reviewer-path.md. It maps common review questions to the demo and artifacts that answer them:

Review question Demo Primary evidence
How are raw events converted to alert features? telemetry-window-demo data/processed/features.csv, data/processed/alerts.csv, data/processed/summary.json, data/processed/run_manifest.json
How is AI constrained? ai-assisted-detection-demo demos/ai-assisted-detection-demo/artifacts/case_summaries.json, demos/ai-assisted-detection-demo/artifacts/audit_traces.jsonl, guardrails in demos/ai-assisted-detection-demo/README.md
How are duplicate alerts reduced? rule-evaluation-and-dedup-demo demos/rule-evaluation-and-dedup-demo/artifacts/rule_hits_before_dedup.json, demos/rule-evaluation-and-dedup-demo/artifacts/rule_hits_after_dedup.json, demos/rule-evaluation-and-dedup-demo/artifacts/dedup_explanations.json
How are risky config changes investigated? config-change-investigation-demo demos/config-change-investigation-demo/artifacts/investigation_hits.json, demos/config-change-investigation-demo/artifacts/investigation_report.md
How are IAM changes investigated from CloudTrail-like events? cloud-iam-change-investigation-demo demos/cloud-iam-change-investigation-demo/artifacts/investigation_signals.json, demos/cloud-iam-change-investigation-demo/artifacts/investigation_report.md

Architecture

See docs/architecture.md for the repository-level workflow diagram.

The important shape is:

  1. committed sample inputs and configs
  2. deterministic local pipelines
  3. bounded AI drafting only where explicitly marked
  4. committed or regenerated artifacts
  5. reviewer inspection through docs, tests, and reports

Artifact Naming Contract

The current artifact names are reviewer-facing contracts for the v1 reviewer contract stabilization phase.

  • Keep the demo matrix stable unless a demo is intentionally retired.
  • Prefer additive artifacts over renaming existing reviewer-visible outputs.
  • Use docs/v0.6-to-v1-artifact-diff.md for the concrete fourth-to-fifth-demo compatibility baseline.
  • If an artifact must be renamed, update the README, reviewer path, this reviewer pack, demo README, tests, and committed sample outputs in the same change.
  • For each release, add the diff described in docs/reviewer-artifact-diff.md, using no-artifact-change when reviewer-facing artifacts are unchanged.
  • Do not introduce platform-style names that imply alert routing, case management, real-time ingestion, or autonomous response.

Evidence Pipeline Contract

See docs/evidence-pipeline-contract.md for the v1 JSON schema contract covering reviewer-facing JSON and JSONL evidence artifacts across the five-demo matrix.

See docs/schema-compatibility-matrix.md for schema versions, artifact paths, and v1.1-to-v1.2 compatibility labels.

See docs/vocabulary.md for the cross-demo meaning of event, signal, hit, finding, case_bundle, summary, report, and audit_trace.

In this repo, bounded correlation means evidence is attached only within fixed time windows, fixed entity or scope keys, and fixed event families or rule-local family sets. It is not cross-host, cross-account, or cross-source global attribution.

The current schema contract covers:

  • schemas/rule_hits.schema.json
  • schemas/case_bundles.schema.json
  • schemas/case_summaries.schema.json
  • schemas/ai_audit_traces.schema.json
  • schemas/dedup_rule_hits.schema.json
  • schemas/dedup_explanations.schema.json
  • schemas/config_change_events.schema.json
  • schemas/config_investigation_hits.schema.json
  • schemas/investigation_summary.schema.json
  • schemas/cloudtrail_normalized_events.schema.json
  • schemas/cloud_iam_findings.schema.json
  • schemas/cloud_iam_summary.schema.json
  • schemas/telemetry_summary.schema.json
  • schemas/run_manifest.schema.json

Stable Reviewer-Visible Artifacts

Area Stable artifact paths
Default telemetry sample data/processed/features.csv, data/processed/alerts.csv, data/processed/summary.json, data/processed/run_manifest.json, data/processed/event_count_timeline.png, data/processed/error_rate_timeline.png, data/processed/alerts_timeline.png
Richer telemetry sample data/processed/richer_sample/features.csv, data/processed/richer_sample/alerts.csv, data/processed/richer_sample/summary.json, data/processed/richer_sample/run_manifest.json, data/processed/richer_sample/event_count_timeline.png, data/processed/richer_sample/error_rate_timeline.png, data/processed/richer_sample/alerts_timeline.png
AI-assisted detection demo demos/ai-assisted-detection-demo/artifacts/rule_hits.json, demos/ai-assisted-detection-demo/artifacts/case_bundles.json, demos/ai-assisted-detection-demo/artifacts/case_summaries.json, demos/ai-assisted-detection-demo/artifacts/case_report.md, demos/ai-assisted-detection-demo/artifacts/audit_traces.jsonl, demos/ai-assisted-detection-demo/artifacts/run_manifest.json
Rule dedup demo demos/rule-evaluation-and-dedup-demo/artifacts/rule_hits_before_dedup.json, demos/rule-evaluation-and-dedup-demo/artifacts/rule_hits_after_dedup.json, demos/rule-evaluation-and-dedup-demo/artifacts/dedup_explanations.json, demos/rule-evaluation-and-dedup-demo/artifacts/dedup_report.md, demos/rule-evaluation-and-dedup-demo/artifacts/run_manifest.json
Config-change investigation demo demos/config-change-investigation-demo/artifacts/change_events_normalized.json, demos/config-change-investigation-demo/artifacts/investigation_hits.json, demos/config-change-investigation-demo/artifacts/investigation_summary.json, demos/config-change-investigation-demo/artifacts/investigation_report.md, demos/config-change-investigation-demo/artifacts/run_manifest.json
Cloud IAM change investigation demo demos/cloud-iam-change-investigation-demo/artifacts/normalized_cloudtrail_events.json, demos/cloud-iam-change-investigation-demo/artifacts/investigation_signals.json, demos/cloud-iam-change-investigation-demo/artifacts/investigation_summary.json, demos/cloud-iam-change-investigation-demo/artifacts/investigation_report.md, demos/cloud-iam-change-investigation-demo/artifacts/run_manifest.json

v1 Readiness Gate

Before treating the repo as v1-style consolidated:

  • Use docs/v1-contract-freeze.md as the v1.0 five-demo contract freeze checklist.
  • Use docs/v1-readiness-gate.md as the v1.0 readiness gate for fixed inputs, fixed outputs, schema validation, artifact regeneration, and test pass.
  • Keep the five-demo matrix stable, or document any intentional retirement in the README, reviewer path, reviewer pack, roadmap, and tests.
  • Keep reviewer-visible artifact names stable, or update the artifact naming contract and committed sample outputs in the same change.
  • Keep docs/reviewer-path.md, this reviewer pack, docs/architecture.md, and demo READMEs aligned with actual CLI behavior.
  • Keep README, package metadata, and repository metadata aligned with the local, file-based detection workflow lab framing.
  • Run python scripts/regenerate_artifacts.py --check to verify committed evidence artifacts match fresh pipeline output.
  • Regenerate and inspect committed artifacts after behavior changes.
  • For every release, include a reviewer-facing artifact diff with added fields, removed fields, semantic changes, and compatibility notes, or an explicit no-artifact-change statement.
  • Run pytest and confirm reviewer-doc tests still cover the demo matrix, artifact contract, and architecture boundaries.
  • Do not add SIEM, dashboard, alert routing, case-management, real-time ingestion, autonomous response, or final-verdict claims.

Fast Verification

From the repository root:

python -m pip install -e ".[dev]"
telemetry-lab run window --config configs/default.yaml
telemetry-lab run ai-assisted
telemetry-lab run dedup
telemetry-lab run config-change
telemetry-lab run cloud-iam
python scripts/regenerate_artifacts.py --check
pytest

Use the same Python interpreter for install, tests, and demo commands.

Review Anchors

Boundaries

  • No production monitoring claims
  • No real-time ingestion or streaming state
  • No dashboard, alert routing, or case-management service
  • No autonomous response actions
  • No final incident verdicts from the AI-assisted demo