feat: add durable PostgreSQL orchestration workflows

Author: O2eg

README.md

@@ -8,8 +8,8 @@ experiments.
   `pg_stand` plan.
 - Keeps human-facing component CLIs small while exposing a typed MCP workflow
   to an AI agent.
-- Records immutable plan hashes, run state, component versions, and artifact
-  hashes.
+- Records immutable plan hashes, durable run state, append-only events,
+  component versions, and artifact hashes.
 - Validates and compares `pg_diag` JSON reports without treating a partial
   collection as a successful complete run.
 - Plans and runs `pg_perf_bench` only against an explicitly selected disposable
@@ -62,7 +62,7 @@ The control layers are deliberately separate:
 agent skills          workflow and interpretation rules
       |
       v
-pg-play-mcp           thirteen typed, high-level operations
+pg-play-mcp           twenty-six typed, high-level operations
       |
       v
 pg_play core          validation, planning, state, comparison
@@ -221,8 +221,12 @@ The `pg-play` CLI contains only complete experiment operations:
 | `capabilities` | Read installed component contracts |
 | `validate MANIFEST` | Validate the manifest and non-mutating component inputs |
 | `plan MANIFEST` | Calculate the current read-only plan and its hash |
-| `run MANIFEST --plan-hash HASH --run-id ID` | Execute exactly that plan |
-| `status MANIFEST --run-id ID` | Read durable run state |
+| `start MANIFEST --plan-hash HASH --run-id ID` | Start exactly that plan in a detached worker and return immediately |
+| `status MANIFEST --run-id ID` | Read durable state and detect a lost worker |
+| `events MANIFEST --run-id ID [--after-sequence N]` | Read ordered durable events, with cursor pagination |
+| `cancel MANIFEST --run-id ID [--reason TEXT]` | Request cooperative cancellation |
+| `resume MANIFEST --plan-hash HASH --run-id ID` | Verify and resume a failed, cancelled, or interrupted attempt |
+| `run MANIFEST --plan-hash HASH --run-id ID` | Execute synchronously for compatibility |
 | `inspect-report REPORT.json` | Validate and summarize one diagnostic artifact |
 | `compare-reports BASELINE.json CANDIDATE.json` | Produce deterministic summary deltas |
 | `inspect-benchmark-report REPORT.json` | Validate and summarize one benchmark artifact |
@@ -241,17 +245,22 @@ pg-workload init --directory workload
 pg-play validate experiment.yaml
 pg-play plan experiment.yaml > plan.json
 
-# Copy plan_hash from the reviewed plan.
-pg-play run experiment.yaml \
+# Copy plan_hash from the reviewed plan. This command returns after spawning
+# the durable worker; it does not wait for the experiment to finish.
+pg-play start experiment.yaml \
   --plan-hash sha256:... \
   --run-id baseline-001
 
 pg-play status experiment.yaml --run-id baseline-001
+pg-play events experiment.yaml --run-id baseline-001 --after-sequence 0
 pg-play inspect-report .pg_play/experiments/pg18-mixed-baseline/baseline-001/report.json
 ```
 
-`run` recalculates the plan and rejects a stale hash. A run id is immutable;
-retry a changed or failed experiment under a new id.
+`start` recalculates the plan and rejects a stale hash. A `run_id` remains
+permanently bound to the same manifest and plan. Use `resume` with that same id
+for a verified failed, cancelled, or interrupted attempt; use a new id when the
+manifest or plan changes. `run` retains the old synchronous behavior for
+scripts that explicitly need it.
 
 ## MCP server
 
@@ -264,11 +273,28 @@ pg-play-mcp
 Configure an MCP client to launch that executable with no shell wrapper. The
 server exposes only:
 
+See [Agent client setup](https://github.com/O2eg/pg_play/blob/main/docs/agent-client-setup.md)
+for concise MCP and Agent Skills configuration examples for Codex CLI, Claude
+Code, Hermes Agent, Kimi Code CLI, Gemini CLI, and OpenCode.
+
 - `component_capabilities`
+- `plan_live_diagnostics`
+- `start_live_diagnostics`
+- `live_diagnostics_status`
+- `live_diagnostics_events`
+- `cancel_live_diagnostics`
+- `plan_configuration_review`
+- `collect_configuration_facts`
+- `generate_configuration_candidate`
+- `compare_configuration_candidate`
 - `validate_experiment`
 - `plan_experiment`
+- `start_experiment`
 - `run_experiment`
 - `experiment_status`
+- `experiment_events`
+- `cancel_experiment`
+- `resume_experiment`
 - `inspect_diagnostic_report`
 - `compare_diagnostic_reports`
 - `inspect_benchmark_report`
@@ -279,23 +305,115 @@ server exposes only:
 - `teardown_experiment`
 
 An agent should validate, plan, show the mutation to the user, and then call
-`run_experiment` with the unchanged hash and a new run id. The implementation
-uses the stable [MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk)
-1.x line and intentionally excludes the 2.x prerelease API.
+`start_experiment` with the unchanged hash and a new run id. It can poll
+`experiment_status` and page through `experiment_events` without holding one
+long MCP request open. The implementation uses the stable
+[MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk) 1.x line
+and intentionally excludes the 2.x prerelease API.
+
+The durable schemas are also exposed as `pgplay://run-state-schema` and
+`pgplay://run-event-schema`.
 
 ## Agent skills
 
-The wheel contains two optional workflow skills under `pg_play/skills/`:
+The wheel contains five optional workflow skills under `pg_play/skills/`:
 
-- `run-postgres-experiment` — validate, plan, execute, and recover a run;
-- `analyze-postgres-experiment` — inspect reports, compare controlled runs,
-  and design the next single-variable iteration.
+- `run-postgres-experiment` — validate, plan, start, and monitor a new run;
+- `recover-postgres-experiment` — inspect, cancel, and safely resume an
+  existing durable run;
+- `diagnose-live-postgres` — capture a bounded, read-only diagnostic window on
+  an existing PostgreSQL host and interpret the validated report;
+- `review-postgres-configuration` — inspect an existing server with a minimal
+  read-only diagnostic set, generate a candidate, and compare changed settings;
+- `analyze-postgres-experiment` — inspect reports, compare or join controlled
+  runs, and design the next single-variable iteration.
 
 Skills contain procedural guidance; they do not reimplement orchestration or
 invoke raw component commands. Skill registration is agent-runtime-specific,
 so installing a Python wheel does not automatically activate them in every
 agent product.
 
+## Live-server incident diagnostics
+
+`plan_live_diagnostics` accepts PostgreSQL and SSH access references, one of
+the reviewed intents (`performance`, `locks`, `io`, or `cpu`), and a bounded
+snapshots window. It returns missing inputs, the exact versioned item allowlist,
+safety limits, and a content hash without connecting to the target. Passwords,
+private-key contents, arbitrary SQL, shell commands, tags, and caller-selected
+item ids are not accepted.
+
+`start_live_diagnostics` stores the unchanged plan in a new immutable capture
+directory and starts a detached worker. The worker invokes `pg_diag snapshots`
+in full remote mode, writes portable JSON and HTML, validates the JSON artifact,
+and records ordered events and terminal state. MCP disconnection does not stop
+the worker. Use `live_diagnostics_status` and `live_diagnostics_events` to
+observe it and `cancel_live_diagnostics` for cooperative cancellation.
+
+Capture duration is limited to 30–900 seconds, interval to 5–60 seconds, and
+the schedule to at most 121 snapshots. A lost worker produces `interrupted`;
+the existing evidence is retained and a new time window requires a new capture
+id rather than a misleading resume or merge.
+
+## Existing-server configuration review
+
+`pg_play` exposes a deliberately read-only review workflow for an existing
+PostgreSQL server. `plan_configuration_review` reports missing database, SSH,
+and tuning-intent inputs. `collect_configuration_facts` runs one bounded
+`pg_diag one-shot` collection containing only server version, effective
+`pg_settings`, database size, CPU, RAM, filesystem, mount, disk, and extension
+inventory items. It stores both the original diagnostic report and a compact
+`pg_diag/configuration-facts-v1` artifact.
+
+`generate_configuration_candidate` combines those observed facts with the
+explicit database duty, storage class, replication mode, and PITR intent. The
+result remains a `pg_configurator/v1` candidate. Finally,
+`compare_configuration_candidate` writes JSON and Markdown containing only
+changed or unobserved parameters, including current source, apply mode,
+pending-restart state, calculation rule, and warnings.
+
+The workflow never applies configuration, reloads PostgreSQL, restarts a
+service, or treats a candidate as a benchmark-proven optimum. During collection,
+`pg_diag` remote mode opens a bounded dynamic local SSH forward to the requested
+PostgreSQL endpoint and closes it with the collection session. Passwords and
+private-key contents are not accepted; callers provide passfile, key, and strict
+known-hosts paths.
+
+## Recovery model
+
+Each run directory contains `state.json`, the original `experiment.yaml`, the
+reviewed `plan.json`, `postgresql-parameters.json`, append-only `events.jsonl`,
+and `worker.log`. `start_experiment` launches a detached worker with its own
+session, so closing or crashing the MCP stdio process does not terminate the
+experiment. If the worker itself disappears, the next `experiment_status`
+changes an active run to `interrupted`.
+
+Cancellation is a durable request in `cancel.request.json`. The worker checks
+it before every step and while a component is running. A running component is
+started in its own process group; before pg_play signals a process left behind
+by a crashed worker, it verifies the recorded PID start time, operating-system
+user, and executable identity.
+
+Resume is intentionally conservative:
+
+| Step | Safe resume rule |
+| --- | --- |
+| stand | Recalculate a read-only pg_stand plan and reconcile only the reviewed desired configuration |
+| benchmark | Retry only through pg_perf_bench's idempotent dedicated-database reset |
+| prepare database | Retry the declarative database preparation operation |
+| install workload | Retry the idempotent profile installation operation |
+| start/stop workload | Reconcile the requested running/stopped state |
+| diagnostics | Retry the read-only collection operation |
+
+Before a resume, pg_play verifies the manifest hash, plan hash, configuration
+artifact, installed component versions, the exact allowlisted resume policy
+for every recorded step, and all recorded artifact paths, sizes, and SHA-256
+hashes that are available.
+
+Completed benchmark and diagnostic steps must have report artifacts. A valid
+completed step is reused; an incomplete step or a step with missing or changed
+artifacts is rerun only when its operation is in the allowlist above. Any
+unknown step or changed core artifact blocks recovery.
+
 ## Determinism and safety
 
 - Every component returns the exact `pg_play/component/v1` envelope in hidden
@@ -310,12 +428,14 @@ agent product.
   component contract does not yet provision a client certificate for the
   dedicated workload role. Direct TLS use of each component remains available.
 - `pg_stand apply` verifies its component plan hash; `pg_play run` verifies the
-  combined plan hash. Machine-mode `pg_perf_bench benchmark` independently
+  combined plan hash. `start` and `resume` enforce the same hash. Machine-mode
+  `pg_perf_bench benchmark` independently
   verifies a content-sensitive benchmark plan hash before resetting its database.
 - Subprocesses receive argument arrays with `shell=False`.
 - Password-bearing CLI arguments and password-bearing machine output are
   rejected.
-- Background workload stop verifies PID ownership before signaling a process.
+- Detached component cleanup verifies PID start time, process ownership, and
+  executable identity before signaling a process group.
 - A failed cleanup cannot hide the original collection failure; a cleanup
   failure after successful diagnostics marks the run partial.
 - `pg_diag` partial artifacts are retained and explicitly marked partial.
@@ -331,9 +451,11 @@ run evidence.
 The first contract covers one configuration candidate, one managed stand,
 an optional controlled benchmark, selected workload profiles, a one-shot or
 snapshots report, and deterministic diagnostic and benchmark comparison.
-Automatic extraction of OS facts and generation or
-application of TuneD/systemd artifacts remains a roadmap item; it is not
-silently approximated by the current implementation.
+Existing-server incident capture is separately limited to the four reviewed
+read-only profiles and one bounded snapshots window per immutable capture id.
+Configuration review extracts the bounded CPU, RAM, filesystem, mount, and disk
+facts listed above. Generation or application of TuneD/systemd artifacts remains
+a roadmap item; it is not silently approximated by the current implementation.
 
 ## Development