fix(helm): protect WAL recovery with a startup probe

[atirna]

Summary

• Add a five-minute startup probe before the OSS chart's liveness probe starts, so WAL replay can finish without a crash loop.
• Document the 10Gi PVC as development-scale and describe the WAL sizing inputs for object-storage deployments.
• Check the installed Deployment's startup-probe contract in the existing Helm integration job.

Refs #676

Test plan

• helm lint ./helm/arc
• helm template arc ./helm/arc checks the default /health startup probe and the unchanged liveness/readiness probes.
• helm template arc ./helm/arc --set startupProbeFailureThreshold=45 checks the grace-period override.
• The startup-probe assertion fails against the pre-fix chart and passes after restoring the chart change.

[github-actions]

All contributors have signed the CLA. Thanks!
_{Posted by the CLA Assistant Lite bot.}

[xe-nvdk]

This lands the right fix for the liveness prong of #676, and the design checks out at every level I could test. The code itself is ready; two administrative items block the merge, listed at the end.

What I verified:

Source ordering. The probe comment's claim is accurate: WAL recovery runs synchronously in main() (RecoverWithOptions, cmd/arc/main.go) well before api.NewServer is constructed and Fiber binds the listener, so /health cannot answer until replay finishes. The startup probe therefore measures exactly the window that needs protecting.

Static checks. helm lint clean, helm template renders the probe correctly at the default and with --set startupProbeFailureThreshold=45, scripts/test_startup_probe.sh passes on this branch and fails against the pre-fix chart (regression coverage confirmed), and shellcheck reports nothing.

Live cluster A/B. I installed both charts side by side on a local k3s cluster with the container command wrapped in a 120-second sleep before exec, simulating a long replay ahead of the listener bind:

• Pre-fix chart: liveness killed the container on a ~70s cycle ("Container arc failed liveness probe, will be restarted"), 4 restarts in 5 minutes, never Ready. This reproduces the #676 crash loop exactly.
• This branch: the startup probe absorbed 12 failed checks against its budget of 30, zero restarts, pod Ready at 2m32s.

Optional polish, fine to fold into the rebase or skip:

• startupProbe inherits timeoutSeconds: 1 while liveness uses 5. During replay the port refuses instantly so it rarely matters, and a slow first /health right after the listener binds would only cost one 10s period. Setting 5 for consistency is cheap.
• check_probe 30 passes --set with the default value, so the values.yaml default itself is never exercised; one run without --set would pin it.
• The assertions exit silently on failure under set -e; an error echo would make a CI failure readable.
• The CI call templates helm/arc while the job installs dist/arc-*.tgz; pointing the script at the tgz would test the exact artifact being released.

Blocking for merge, both quick:

1. CLA. The CLA check is failing; signing is now required on every PR (#699). Please sign per the bot's instructions in this thread and comment recheck.
2. Rebase. RELEASE_NOTES_2026.09.2.md conflicts with main (the Bug fixes section gained the #697 entry this morning; that is the only conflicting file). Please rebase and keep your entry at the top of the section.

[atirna]

rebased onto main and kept the #676 entry at the top of Bug fixes.

[atirna]

thx for the review. I pushed the test follow-up: the default chart render is now checked without an override, the 45 override stays covered, and assertion failures name the missing probe field. helm lint ./helm/arc and the startup-probe script pass.

kept timeoutSeconds and the packaged-chart check as-is for this chart-only test follow-up.

[atirna]

I have read the CLA Document and I hereby sign the CLA

[xe-nvdk]

CLA is green and the rebase is clean: the chart, workflow, and release-notes content match what was verified in the first round (including the live cluster A/B), with your entry at the top of the Bug fixes section. The new test commit improves on two of the optional notes: the default case now templates without --set, so the values.yaml default is actually exercised, and the assertions report what failed instead of exiting silently. I re-ran the script against a sabotaged threshold to confirm the failure message fires. Merging.