WAL fills the default 10Gi PVC during bulk ingest: no backpressure, boot deadlock on full volume, liveness kills recovery

[xe-nvdk]

During sustained bulk ingest (snapshot backfill at ~100 MB/s), the write-ahead log can grow faster than it flushes. On a Kubernetes deployment with an S3/object-storage backend, the local PVC holds only the WAL and cache, and the chart's default persistence.size: 10Gi (helm/arc/values.yaml) fills in minutes at that rate. The resulting failure chain is worse than a full disk:

1. WAL fills the volume with no backpressure. Writes keep being accepted while the WAL grows unboundedly until the volume is full.
2. Boot deadlock. After the writer goes down, it cannot start again: creating the initial WAL file on startup requires free space on the already-full volume, so the pod can never come back without manual intervention (growing the PVC or deleting WAL segments by hand).
3. Liveness kills recovery. Once space is available, WAL replay after an unclean stop can exceed the liveness budget in helm/arc/templates/deployment.yaml (periodSeconds: 10, failureThreshold: 3, no startupProbe), so the pod is killed mid-recovery and crash-loops repeatedly before it manages to drain.

Observed in the field during a TB-scale migration; the operator's workaround was ARC_WAL_ENABLED=false for the migration window (throughput was unchanged, WAL stayed flat), which works but gives up durability for live writes during that window.

Proposed fixes

• Backpressure (core): when WAL disk usage crosses a high-water mark, reject writes (503, or a typed backpressure error clients can retry) instead of growing until the volume is full. Reserve enough headroom that the writer can always create its initial WAL file on boot, so the deadlock in (2) becomes impossible.
• startupProbe (chart): add a startupProbe with a generous failureThreshold × periodSeconds window (minutes, not seconds) so WAL recovery is never killed by the liveness probe; keep the current livenessProbe for steady state.
• Sizing guidance and defaults (chart/docs): document WAL volume sizing for object-storage backends (roughly: peak ingest rate × worst-case flush lag, plus cache), call out that 10Gi is a dev-scale default, and consider raising it or adding a loud comment in values.yaml.

Related (probably a separate issue)

With very wide rows (tens of KB of line protocol per row), the ingest buffer's row-count bound (100k rows) and the WAL's payload byte cap (WAL payload exceeds maximum allowed size, 104,857,600 bytes) cannot be reconciled by tuning: a full flush of wide rows exceeds the cap, so those flushes never reach the WAL at all — the WAL directory holds only a 7-byte header file while ingest runs. Chunking WAL record writes, or bounding the ingest buffer by bytes as well as rows, would fix it. Happy to split this into its own issue.

[xe-nvdk]

Status: the liveness prong is fixed. #700 (merged, lands in 26.09.2) adds a five-minute startup probe to the OSS Helm chart so WAL replay is no longer killed mid-recovery, and documents the 10Gi PVC default as development-scale with sizing guidance for object-storage deployments.

Still open from this issue:

1. WAL backpressure in core: reject writes (503 or a typed backpressure error) when WAL disk usage crosses a high-water mark, instead of growing until the volume is full.
2. Boot deadlock on a full volume: startup cannot create the initial WAL file with zero free space, so a node cannot recover without manual intervention. A reserved headroom file or a pre-flight check with a clear error would both work.

Keeping this open for those two.

[efegokdemir]

I'd like to work on the two remaining core items: WAL disk-usage backpressure and safe startup when the volume is full. Before coding, could you confirm the preferred API/behavior—HTTP 503 versus a typed retryable error, and reserved headroom versus a startup preflight check? These choices affect write and recovery semantics, so I want to align with maintainers first.