P006 - Tool Side-Effect Sandbox & Idempotency Verifier

P006 proves that agent/tool workflows can prevent unsafe, duplicate, or non-idempotent side effects under retries, partial failures, ambiguous action plans, and approval boundaries.

Problem

AI-assisted workflows can now propose operational actions such as refunds, account updates, CRM writes, ticket escalations, and fulfillment changes. The dangerous failure is not just a bad answer. It is a duplicated or unsafe side effect caused by a retry, timeout, malformed action plan, missing approval, or partial failure.

P006 focuses on the pre-commit safety boundary: a proposed side-effecting action cannot mutate external state until typed contracts, policy rules, idempotency checks, and execution-mode rules have produced durable decisions.

This is not a chatbot, generic agent platform, dashboard, payment app, claims system, or live integration demo. All data is synthetic.

What V1 Proves

V1 is a deterministic Python safety layer for one synthetic customer-support refund workflow. It proves:

• typed action contracts before execution
• protected-field hashing and idempotency keys before commit
• policy and approval gates before side effects
• dry-run vs commit separation
• duplicate retry resolution without a second external write
• append-only action-ledger evidence
• file-backed simulator state with observable write counts and receipts
• compensation-required state after partial failure
• generated reports from real ledger events and simulator observations
• a CLI safety gate that fails on unsafe evidence

No real payments, real customer records, real claims systems, real external APIs, live providers, auth system, vector database, or LangGraph runtime are used.

Architecture

The narrow v1 flow is:

1. Load a synthetic support case and synthetic action plan.
2. Parse the proposed action into Pydantic models.
3. Promote valid proposals into an ActionContract.
4. Evaluate deterministic refund policy.
5. Reserve or resolve an idempotency key from protected fields.
6. Record each safety decision in an append-only JSONL ledger.
7. For dry-run, predict the write without mutating simulator state.
8. For commit, mutate the file-backed simulator only after safety gates pass.
9. For duplicate retry, resolve to the existing committed ledger entry.
10. For partial failure, record compensation_required with the external receipt.
11. Generate Markdown reports from ledger replay and simulator state.

Key modules:

• models.py: Pydantic contracts, decisions, ledger entries, execution results, reports
• hashing.py: canonical JSON and protected-field hashes
• policy.py: deterministic policy and approval checks
• idempotency.py: pre-commit reservation, retry resolution, conflict blocking
• ledger.py: append-only JSONL evidence and replay
• simulator.py: deterministic file-backed external-system simulator
• executor.py: orchestrates validation, policy, idempotency, ledger, simulator boundaries
• reports.py: structured and Markdown safety reports
• gate.py / cli.py: seeded scenario runner and safety gate

flowchart TD
    proposal["Synthetic action plan"]
    contract["Typed contract"]
    policy{"Policy gate"}
    idem{"Idempotency gate"}
    commit{"Commit boundary"}
    simulator["File-backed simulator"]
    ledger["Action ledger"]
    report["Generated report"]
    gate["CLI safety gate"]

    proposal --> contract --> policy
    policy -- "blocked" --> ledger
    policy -- "allowed" --> idem
    idem -- "duplicate retry" --> ledger
    idem -- "new reserved key" --> commit
    commit -- "dry-run" --> ledger
    commit -- "commit" --> simulator
    simulator --> ledger
    ledger --> report
    simulator --> report
    report --> gate

Loading

The simulator is the first mutation-capable component. The ledger is evidence, not permission.

Demo

Generated ledgers, reports, and simulator state are reproducible and intentionally kept out of git by default. Use the ignored .demo/ directory:

PYTHONPATH=src python3 -m p006 gate --output-dir .demo

Expected output:

PASS gate scenarios=5
- safe_refund: committed writes=1 passed=true
- duplicate_retry: retry_resolved writes=1 passed=true
- unsafe_amount: blocked writes=0 passed=true
- ambiguous_missing_fields: rejected_before_execution writes=0 passed=true
- partial_failure: compensation_required writes=1 passed=true

Useful focused commands:

PYTHONPATH=src python3 -m p006 run --case duplicate_retry --output-dir .demo
PYTHONPATH=src python3 -m p006 report --case partial_failure --output-dir .demo

Generated artifacts appear under:

• .demo/ledgers/
• .demo/reports/
• .demo/simulator_state/

Evidence Excerpts

Duplicate retry report:

action_status: retry_resolved
external_write_count: 1
duplicate_prevented: true
decision_path: contract_validated -> policy_decided -> idempotency_decided -> retry_resolved

The duplicate retry ledger shows the first action committing once and the retry resolving to that committed entry:

req_duplicate_retry_001: committed, receipt=receipt:req_duplicate_retry_001:000001
req_duplicate_retry_002: retry_resolved, existing_ledger_entry_id=duplicate_retry:000005

Partial failure report:

action_status: compensation_required
external_write_count: 1
compensation_required: true
decision_path: contract_validated -> policy_decided -> idempotency_decided -> commit_started -> compensation_required

Unsafe amount report:

action_status: blocked
external_write_count: 0
unsafe_action_blocked: true
decision_path: contract_validated -> policy_decided

Ambiguous action report:

action_status: rejected_before_execution
external_write_count: 0
decision_path: contract_rejected

Evaluation

Run the test suite:

python3 -m pytest

Current coverage asserts:

• valid action contracts parse
• malformed actions reject before execution
• protected-field hashes are stable
• idempotency keys survive retry metadata changes
• protected-field changes alter the key
• safe policy decisions allow eligible refunds
• unsafe amounts block before simulator mutation
• duplicate retries do not call the simulator twice
• dry-run does not mutate simulator state
• partial failure creates compensation-required state
• ledger replay reconstructs decision paths
• reports are generated from real ledger and simulator evidence
• the CLI gate exits nonzero on failed safety verdicts

Status

V1 implementation is functional for the seeded synthetic refund workflow. The command-only demo path is the default: generated evidence is reproducible from code and fixtures, not checked in as hand-authored proof.

For the fuller public narrative, see docs/case_study.md.

For the complete documentation map, see docs/index.md.

Next useful improvement: add screenshots or selected generated report snippets after the public presentation style stabilizes.