docs(deploy): document JetStream SyncAlways durability vs throughput

CHANGELOG.md

@@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **"Durability & Storage" operations guide** (`docs/src/content/docs/durability.md` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/reverse-proxy.mdx`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/deployment.md`): documents #84. A new Operations page making the embedded-JetStream durability contract explicit before the docs site publishes: a `200` from `POST /v1/ingest` means the event has been `fsync`'d to disk on the node (the server runs with `SyncAlways: true` in `internal/mq/embedded.go`), which makes the storage substrate's `fsync` tail the ingest latency floor. Covers the contract (and how it differs from JetStream's default page-cache-then-periodic-sync mode), why a slow `fsync` tail manifests as `create stream: ... context deadline exceeded` and `503` backpressure, a where-it's-cheap-vs-expensive substrate table (managed cloud block storage and PLP NVMe vs. ZFS-without-SLOG / qcow2-on-`ext4` / spinning disks), an `fio` recipe + verdict bands to measure your own storage (with the macOS `F_FULLFSYNC` honesty caveat), and the symptom checklist. Forward-references the configurable group-commit interval (`mq.sync_interval`, [#139](https://github.com/Wave-RF/WaveHouse/issues/139)) and the planned `wavehouse storage-check` preflight ([#84](https://github.com/Wave-RF/WaveHouse/issues/84)) without claiming either exists yet. Cross-linked from Configuration (Message Queue), Deployment (Persistent Storage), and the Ingest Pipeline's worker-side ack section; no code changes.
 - **"Behind a reverse proxy" deployment guide** (`docs/src/content/docs/reverse-proxy.mdx` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/deployment.md`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/api.md`, `internal/api/stream.go`, `internal/api/stream_test.go`): closes #241. A new Operations page for the common "WaveHouse behind nginx / Caddy / Cloudflare Tunnel" setup, since several behaviors only matter behind a proxy and weren't documented together. Covers: TLS termination (WaveHouse serves plain HTTP and manages no certs); the request-body size limits and the division of responsibility (WaveHouse ships fixed in-code memory-safety backstops — 1 MiB control / 16 MiB ingest — while the proxy is the tunable *outer* limit, so a missing/loose proxy limit can't OOM the server); Server-Sent Events buffering + idle-timeout tuning (WaveHouse sends one `: connected` comment then **no periodic heartbeat** yet, [#226](https://github.com/Wave-RF/WaveHouse/issues/226), so quiet streams rely on the proxy not closing idle connections and on `EventSource` auto-reconnect); the `?token=` / `since` / `Last-Event-ID` forwarding streams need; `X-Forwarded-For` trust (don't expose `:8080` directly — it's honored, so a direct client could spoof it); and which health paths to expose (`/livez`/`/readyz` internal-optional, `/v1/health` must stay public). Ships full example nginx, Caddy, and Cloudflare-Tunnel configs, and is cross-linked from Deployment, Configuration, and the API reference. One small code change lands with it: the SSE endpoint (`GET /v1/stream`) now sets `X-Accel-Buffering: no` so nginx-class proxies stream events without buffering out of the box (nginx strips the header before the client sees it; Caddy/Cloudflare ignore it). The health-probe guidance is also upgraded from "optional" to a recommendation — keep the bare `/livez`/`/readyz`/`/healthz` paths internal (a public `/readyz` turns each hit into a ClickHouse `Ping`) and expose only `/v1/health` publicly.
 - **Coverage publishing — a self-hosted Go coverage README badge and GitHub Code Quality PR comments** (`.github/workflows/ci.yml`, `.github/actionlint.yaml` (new), `scripts/cov/main.go`, `scripts/ci/publish-badge.sh` (new), `.testcoverage.yml`, `go.mod`/`go.sum`, `README.md`, `AGENTS.md`, `docs/src/content/docs/development.md`): closes #133, now that the repo is public. Two published surfaces, both **non-gating** — `make cov`'s thresholds stay the only merge gate. (1) **README badge**: a new `cov badge` subcommand renders a [shields.io endpoint](https://shields.io/endpoint) JSON for the merged Go total using the *exact* number `threshold.total` gates (same `.testcoverage.yml` excludes), and a new non-gating `badge` job — the sole holder of `contents:write`, running only on trusted main — publishes it to an orphan `badges` branch via `scripts/ci/publish-badge.sh`, which the README reads over `raw.githubusercontent.com` (unrestricted for a public repo). (2) **PR comments**: the `coverage` job converts the merged Go profile to Cobertura (`go tool gocover-cobertura`, a new pinned Go `tool` dependency, with `-ignore-dirs` mirroring the YAML's global excludes) and uploads it to GitHub Code Quality via `actions/upload-code-coverage` (`code-quality: write`); the `github-code-quality[bot]` posts the aggregate + per-file diff-vs-`main` comment. The upload is `continue-on-error` so this public-preview GitHub feature can never red CI, and fork PRs skip it (no `code-quality` token, per GitHub's own guard). `actionlint` doesn't recognize the preview `code-quality` permission scope yet, so a new `.github/actionlint.yaml` suppresses only that one message. Requires the repo's *Settings → Code quality* enablement for the comments to render. Full design in `.github/workflows/README.md` §"Coverage publishing".
 

docs/src/config/sidebar.ts

@@ -46,6 +46,7 @@ export const sidebar: StarlightUserConfig["sidebar"] = [
     items: [
       { label: "Configuration", slug: "configuration" },
       { label: "Deployment", slug: "deployment" },
+      { label: "Durability & Storage", slug: "durability" },
       { label: "Behind a reverse proxy", slug: "reverse-proxy" },
     ],
   },

docs/src/content/docs/configuration.mdx

@@ -64,6 +64,8 @@ The server speaks **plain HTTP** — there is no inbound-TLS setting. Terminate
 | `mq.gap_window_minutes` | `WH_MQ_GAP_WINDOW_MINUTES` | `15` | How many minutes of messages to retain in NATS for SSE gap-fill. The Active Sweeper will not purge messages newer than this window. |
 | `mq.max_bytes_gb` | `WH_MQ_MAX_BYTES_GB` | `50` | Maximum NATS JetStream stream size in GB. When full, new publishes are rejected with `DiscardNew` policy, triggering 503 backpressure on the ingest endpoint. |
 
+**Durability.** The embedded server runs with JetStream `SyncAlways`, so every event is `fsync`'d to disk before `POST /v1/ingest` returns `200`. This makes your storage's `fsync` latency your ingest latency floor — see [Durability & Storage](/durability) to check whether your substrate can sustain it. There is no knob to relax this today ([#139](https://github.com/Wave-RF/WaveHouse/issues/139) tracks a configurable group-commit interval).
+
 ### Deduplication
 
 | YAML Key | Env Var | Default | Description |

docs/src/content/docs/deployment.md

@@ -171,6 +171,8 @@ In a Docker / Podman / Kubernetes deployment, **`data_dir` must resolve to a hos
 
 If `data_dir` resolves into the container's writable overlay layer instead, **JetStream state is wiped on every restart**: in-flight events are lost, gap-fill stops bridging restarts, and disk usage accumulates inside `/var/lib/docker` instead of the volume the operator chose.
 
+Beyond persistence, the *speed* of that volume matters: JetStream `fsync`s every event to `<data_dir>/nats` before the ingest endpoint returns `200`, so the volume's `fsync` latency is your ingest latency floor. Managed cloud block storage handles this without thinking; commodity or virtualized substrates (ZFS without a SLOG, qcow2-on-`ext4`, spinning disks) can stall ingest with multi-second `fsync` tails. See [Durability & Storage](/durability) to measure yours before going live.
+
 WaveHouse runs a simple existence check on startup and logs a `WARN` if `<data_dir>/nats` (or `<data_dir>/pebble` when dedupe is on) is missing or empty:
 
 ```text wrap=false

docs/src/content/docs/durability.md

@@ -0,0 +1,103 @@
+---
+title: "Durability & Storage"
+description: "What a WaveHouse ingest ack guarantees, why embedded JetStream fsyncs every publish, and how to tell whether your storage substrate can sustain it."
+sidebar:
+  order: 11
+---
+
+WaveHouse buffers every ingested event in embedded NATS JetStream before the [ingest worker](/ingest-pipeline) drains it into ClickHouse. That buffer lives on disk at `<data_dir>/nats`, and **WaveHouse runs JetStream in its strictest durability mode**: every publish is `fsync`'d to non-volatile storage before the producer is acknowledged.
+
+This is a deliberate, strong guarantee — but it makes your ingest latency a direct function of your storage's `fsync` latency. On managed cloud block storage that is effectively free; on some commodity or virtualized substrates the `fsync` tail balloons into seconds and ingest visibly suffers. This page explains the contract, where it is cheap versus expensive, and how to measure your storage before you trust it.
+
+## The durability contract
+
+The embedded server is started with `SyncAlways: true` (`internal/mq/embedded.go`). Concretely:
+
+> When a client receives `200` from `POST /v1/ingest`, the event has already been `fsync`'d to disk on the WaveHouse node.
+
+Ingestion is still [asynchronous](/architecture) end-to-end — the `200` means *durably buffered in JetStream*, not yet *written to ClickHouse* (the worker flushes to ClickHouse later, and the [worker's own ack](/ingest-pipeline#backpressure-and-durability-knobs) is what records "now in ClickHouse"). But the buffering step itself is hard-durable: an event that got a `200` survives an immediate, uncontrolled power loss on the node.
+
+This is the strongest mode JetStream offers. It is stronger than the default, where a publish is acked once the write reaches the OS page cache and the data is flushed to disk later by a periodic background sync — fast, but a hard crash can lose the not-yet-flushed window.
+
+| Mode | Ack means | Crash exposure | Throughput |
+| --- | --- | --- | --- |
+| **`SyncAlways` (WaveHouse today)** | data is `fsync`'d to disk | none for acked events | bounded by `fsync` latency |
+| Periodic group commit (default JetStream) | data is in the OS page cache | up to one sync interval of acked-but-unflushed events | bounded by memory/CPU |
+
+WaveHouse does not currently expose a knob to relax this — `SyncAlways` is always on. Exposing a configurable group-commit interval (`mq.sync_interval`) is tracked in [#139](https://github.com/Wave-RF/WaveHouse/issues/139).
+
+## Why the fsync tail is your ingest floor
+
+Because the publish blocks on `fsync`, **your typical ingest latency is your storage's typical `fsync` latency, and your worst-case publish is your storage's worst-case `fsync`.** When that tail is healthy (sub-millisecond to single-digit milliseconds) the guarantee is essentially free. When it is not, the same code path that handles every production message stalls:
+
+- Publishes block for the duration of the `fsync`, so a multi-second `fsync` tail is a multi-second ingest tail.
+- The embedded server's stream/consumer setup and every publish run under the JetStream client's request timeout; a slow-enough substrate makes them exceed it. The boot-time symptom is `create stream: ... context deadline exceeded`.
+- If the worker cannot drain to ClickHouse faster than producers publish, the stream fills toward `mq.max_bytes_gb` and the API returns `503` ([backpressure by construction](/ingest-pipeline#backpressure-and-durability-knobs)).
+
+## Where `SyncAlways` is cheap vs. expensive
+
+The strict guarantee translates well to managed cloud infrastructure — the presumed production target — and to enterprise-grade local disks. It is the commodity and virtualized substrates that bite.
+
+**Healthy — `SyncAlways` is effectively free:**
+
+| Substrate | Why |
+| --- | --- |
+| Cloud block storage (gp3/io2 EBS, GCP pd-ssd, Azure Premium SSD) | Battery-backed cache acks sync writes from non-volatile DRAM, not NAND. Sub-millisecond at typical load. |
+| Enterprise NVMe with power-loss protection (Optane, Samsung PM-series, Solidigm D7) | The PLP capacitor lets the controller ack a sync write from DRAM — the `fsync` ≈ `memcpy`. |
+| Local `ext4` on consumer NVMe | Single-device journal commit, 1–10 ms typical. Tail spikes under heavy concurrent dirty data, but bounded. |
+
+**Problematic — measure before you trust it:**
+
+| Substrate | Failure mode |
+| --- | --- |
+| ZFS without a SLOG, consumer NVMe | Every sync write hits the ZIL, gated by transaction-group commit cadence that serializes across all pool consumers. Single-digit-ms idle, **5–25 s under concurrent load**. |
+| Loopback / qcow2 on `ext4` inside a VM | Stacks a second journaling layer; often 10× slower than direct `ext4` and highly variable. |
+| Spinning disks | Mechanical seek on the NAND-equivalent program path: multi-millisecond baseline, multi-second tail. |
+
+The tell for a commit-cadence problem (ZFS-without-SLOG, noisy-neighbor VM host) is that a single-threaded benchmark looks fine while a concurrent one is far worse — so always benchmark with multiple writers, and benchmark the guest **and** the host if virtualized.
+
+## Check your storage before you trust it
+
+Replicate JetStream's exact pattern — a 4 KiB write followed by a flush, in a tight loop — and report the percentiles. The numbers that matter are **p99** and **max**: those are your worst-case publish latency.
+
+On Linux, [`fio`](https://fio.readthedocs.io/) (packaged on every distro) is the honest, standard tool. Point it at the volume that backs `<data_dir>/nats`, ideally before WaveHouse is running:
+
+```bash
+# 8 concurrent writers — the variant that surfaces commit-cadence problems
+fio --name=jetstream-fsync --directory=/var/lib/wavehouse/nats \
+    --rw=write --bs=4k --size=64M --fsync=1 --runtime=30 --time_based \
+    --numjobs=8 --group_reporting
+```
+
+Run it under representative load, not on an idle box — idle benchmarks understate real-world tails.
+
+Read the measured p99 against these bands, which track WaveHouse's `SyncAlways` default:
+
+| p99 `fsync` | Verdict for `SyncAlways: true` |
+| ---: | --- |
+| < 1 ms | **Ideal** |
+| 1–5 ms | **Good** |
+| 5–50 ms | **Workable** — watch bursty load |
+| 50 ms – 1 s | **Marginal** — relax durability once `mq.sync_interval` ([#139](https://github.com/Wave-RF/WaveHouse/issues/139)) lands, or move to faster storage |
+| > 1 s | **Broken** — `create stream` will time out under load; fix the storage substrate |
+
+:::caution[macOS `fsync` lies by default]
+A plain `fsync()` on macOS returns once data is in the drive's volatile cache — it does **not** force a flush to NAND; only `fcntl(fd, F_FULLFSYNC)` does (NATS, Postgres, and SQLite all use it). On a Mac, any per-flush number under ~1 ms is almost certainly not a real flush — the gap between plain `fsync()` and `F_FULLFSYNC` can be ~180× on the same consumer NVMe. `fio` on macOS calls plain `fsync()`, so don't trust Mac `fio` numbers for tail-latency planning. This mostly matters when benchmarking a dev machine; production WaveHouse runs on Linux, where `fio` is honest.
+:::
+
+A self-contained `wavehouse storage-check` preflight subcommand that bakes this measurement and verdict into the binary — including the per-platform honest flush — is tracked in [#84](https://github.com/Wave-RF/WaveHouse/issues/84).
+
+## Symptoms of storage that can't keep up
+
+If you see any of these, benchmark the `<data_dir>/nats` volume as above:
+
+- `create stream: ... context deadline exceeded` at startup.
+- Ingest p99 latency in the seconds, or occasional `200`s that take multiple seconds to return.
+- Intermittent `503 Service Unavailable` from `/v1/ingest` when ClickHouse is healthy (the worker can't drain fast enough because acking is `fsync`-bound).
+- Flaky CI or load tests that pass on fast storage and fail on a shared/virtualized host.
+
+## See also
+
+- [Configuration → Message Queue (NATS)](/configuration#message-queue-nats) — the `mq.*` knobs (`gap_window_minutes`, `max_bytes_gb`).
+- [Deployment → Persistent Storage](/deployment#persistent-storage-required-for-containers) — `data_dir` must resolve to a host-backed volume.
+- [Ingest Pipeline → Backpressure and durability knobs](/ingest-pipeline#backpressure-and-durability-knobs) — the worker-side ack cost and the in-flight backpressure layers.

docs/src/content/docs/reverse-proxy.mdx

@@ -2,7 +2,7 @@
 title: "Behind a reverse proxy"
 description: "Running WaveHouse behind nginx, Caddy, or a Cloudflare Tunnel: TLS, request-body limits, SSE buffering and timeouts, header/auth forwarding, and health probes."
 sidebar:
-  order: 11
+  order: 12
 ---
 
 import { Tabs, TabItem } from "@astrojs/starlight/components";