feat: fold chaos scenarios into long_horizon phase types (#12 partial, #13)

[hardbyte]

Summary

Consolidate the legacy chaos.py runner into the long_horizon.py phase substrate. Chaos scenarios become named phase compositions and the chaos-specific orchestration moves into phase-type hooks that share the sample stream, wait-event sampler, and per-phase aggregator with every other scenario. Closes #13. Partial-closes #12 — the chaos.py half is gone; cross-system chaos runs still need to land scenario-by-scenario via the new phase types.

New phase types (bench_harness/phases.py + hooks.py)

Phase type	Behaviour
postgres-restart	docker compose stop postgres for the first half of the duration, docker compose start for the rest. Drives the harness-managed compose lifecycle via callbacks the orchestrator stashes in runtime.state.
pg-backend-kill(rate=N)	Admin connection runs pg_terminate_backend(pid) against the SUT's datname every 1/N seconds (filtered to active / idle-in-tx). Uses datname rather than application_name because adapters don't uniformly set it.
pool-exhaustion(idle_conns=N)	Holds N idle connections to the SUT's database for the duration; releases on phase end. Best-effort if max_connections is below the request — the chaos point is "what happens under pressure," not "fail the run if PG can't fit."
repeated-kill(instance=I,period=Ns)	Periodic SIGKILL + auto-restart of replica I every period, composing the existing kill_worker / start_worker pool methods. Re-uses parse_duration for the period spec.

Existing kill-worker / start-worker are unchanged.

New scenarios

chaos_crash_recovery, chaos_postgres_restart, chaos_repeated_kills, chaos_pg_backend_kill, chaos_pool_exhaustion — all warmup → baseline → chaos → recovery shapes, ~4 minutes each.

Aggregator

bench_harness/writers.py::compute_summary now derives, for any clean / recovery phase that immediately follows a chaos / lifecycle span:

• jobs_lost = ∫enqueue_rate − ∫completion_rate over the chaos+recovery span (rate × window_s integrated).
• chaos_recovery_time_s = elapsed_s from end of chaos until completion_rate first re-attains 90% of the baseline median.
• baseline_completion_rate_median (sanity-check field).

These attach to the recovery phase block in summary.json. Duplicate-completions tracking (chaos.py originally hashed per-job idempotency keys) is out of scope for this PR — the harness doesn't currently track per-job ids; tracked as follow-up.

Pending (NOT migrated)

The following chaos.py scenarios stay in the deprecated runner because they are system-specific:

• leader_failover (awa-only — depends on awa's leader election semantics)
• retry_storm (needs adapter-driven retry-storm shape)
• priority_starvation (needs priority-aware adapter producer)

chaos.py is not deleted. It carries a deprecation banner pointing at the new scenarios and emits a DeprecationWarning on import.

Smoke run

$ uv run python long_horizon.py run --scenario chaos_crash_recovery \\
    --systems awa --replicas 2 --worker-count 16 --producer-rate 1000 \\
    --producer-mode depth-target --target-depth 1000

[awa] phase warmup (warmup) for 30s
[awa] phase baseline (clean) for 60s
[awa] phase kill (kill-worker) for 60s
[awa] phase restart (start-worker) for 60s
[harness] launching awa replica 0: ...
[awa] phase recovery (clean) for 60s
[harness] results at: results/chaos_crash_recovery-20260501T193816Z-5a7fda

summary.json recovery block:

{
  "jobs_enqueued": 2359542.83,
  "jobs_completed": 150448.27,
  "jobs_lost": 2209094.56,
  "chaos_recovery_time_s": 4.62,
  "baseline_completion_rate_median": 457.20
}

(The absolute jobs_enqueued is inflated because awa's depth-target producer reports very large rates while the kill phase has zero workers; the metric mechanics are correct but the awa-side instrumentation in that mode is something to revisit. chaos_recovery_time_s of 4.6 s is the trustworthy number — completion rate came back to ≥90% of baseline within one sample window after the restart.)

index.html, plots/, and per-phase wait-event histograms all render (wait-event sampler runs by default, untouched).

Tests

5 new tests: scenario resolution for each new chaos scenario + a synthetic-csv test that exercises the writer's chaos aggregator end-to-end.

$ uv run pytest tests/ -q
95 passed in 6.09s

Test plan

• pytest tests/ -q — 95 pass (was 85; +10 new)
• Smoke chaos_crash_recovery against awa with 2 replicas — emitted jobs_lost, chaos_recovery_time_s, rendered index.html
• Cross-system smoke (awa + procrastinate + river) on each chaos_* scenario — follow-up
• Migrate leader_failover / retry_storm / priority_starvation — follow-up issue

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b9f4deede9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Keep pg-backend kill loop reconnecting after query errors

When cur.execute raises inside _pg_backend_kill_loop, the handler immediately breaks out of the loop, which ends the background thread for the rest of the phase. In practice, a transient DB error (for example during a restart or brief connection interruption) will disable backend-kill injection entirely, so chaos_pg_backend_kill runs can silently devolve into a mostly clean phase and corrupt the experiment signal.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Compute recovery time even when chaos has no completion samples

chaos_recovery_time_s is gated on having at least one chaos-phase completion_rate sample, because chaos_end_t is derived only from completion_rate rows. For severe outages, adapters can skip rate samples during chaos, and then this path leaves chaos_recovery_time_s as None even if recovery samples clearly show when throughput returns, dropping a key metric in the exact failure modes this aggregation is meant to cover.

Useful? React with 👍 / 👎.