danielgaskins · Oct 6, 2026 · Oct 7, 2026
@@ -7,8 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- The PostgreSQL ImageVolume policy exception now includes the `<clusterName>-restore` recovery Pods and Jobs alongside the source Cluster.
+
 ### Added
 
+- An opt-in PostgreSQL backup verification runbook and suspended cnpg-drill values example for the kagent API v2 database.
 - **`make verify-kagent-netpol` asserts the atenet router admits the kagent controller** (giantswarm/agent-platform#827): in both flavours, the 8080 rule of `substrate-atenet-router` admits exactly the controller's selector labels from `kagent.namespaceOverride`, and the assertion refuses a render whose rule names the release namespace instead. `docs/substrate-security.md` names the namespace relation and why a port-forward does not prove the path.
 - **Preset field `requirements.residentWeightsGiB`: the weights a runtime holds in memory when it keeps part of the checkpoint on disk** (giantswarm/model-manager#240). It is optional, at most `weightsGiB` (the render fails above it) and defaults to `weightsGiB`. On a unified-memory node, model-manager's fit check needs the `--gpu-memory-utilization` claim to hold the resident weights plus `overheadGiB`. `qwen3-8-flash-next-nvfp4` declares 74 GiB resident (the n-gram table is read from disk), so 74 + 20 GiB fit within its 0.80 claim. `nemotron-3-super-nvfp4` declares 12 GiB of overhead instead of 30, so 75 + 12 GiB fit within its 0.72 claim: the CUDA-graph pool already counts inside the claim.
 - **`docs/model-serving-image-pulls.md`: the runtime's download at boot and where a cold node's time goes** (giantswarm/agent-platform#812). A GPU pool node created by cluster-manager (0.26.0) on gpu-node-pool 0.8.0 fetches the pre-pull's images (`modelServing.prepull.images`) into containerd while it joins, so the pre-pull, gated on the GPU, only unpacks. On a cold g6.xlarge node Ready → predictor Ready went from 471–482 s to 419–437 s. The page records the variants measured: unpacking at boot as well slows the GPU operator's and the node's DaemonSet pulls and is slower overall. What remains is the unpack of the 15 GB runtime.

@@ -4263,3 +4263,8 @@ verify-all: ## Run every verify-* target of this file, the set CI runs, each wit
 		echo "<==== $$target: $$(( ($$(date +%s%N) - start) / 1000000 )) ms"; \
 	done
 	@echo "all $(words $(VERIFY_TARGETS)) verify targets passed."
+
+.PHONY: verify-postgres-restore
+verify-postgres: verify-postgres-restore
+verify-postgres-restore: ## Assert the ImageVolume exception covers the source and its exact restore name.
+	@HELM="$(HELM)" python3 tests/verify-postgres-restore.py $(CONNECTIVITY_DIR)
@@ -727,6 +727,7 @@ muster builds `read-only`, `none` and `full` in. An installation adds its own (`
 - `postgres.backup.crossplane.*` provisions the store as Crossplane managed resources (S3 bucket + IRSA role on AWS, Storage Account + Container on Azure, with a PrivateEndpoint on private installations) and derives the path and credentials for the `ObjectStore`. Installations without Crossplane set `destinationPath` and one credential source themselves.
 - `postgres.backup.method: volumeSnapshot` takes CSI snapshots instead (needs a `VolumeSnapshotClass`; no WAL archive).
 - Restore is a second Cluster bootstrapped with `recovery` from the same `ObjectStore` (`externalClusters[].plugin` with `serverName` = the source Cluster's name), never the live one. Name it exactly `<clusterName>-restore`: the Cluster's network policy selects that name too (the recovery Job needs the Kubernetes API and the store), and the AWS role trusts `<clusterName>-restore*` ServiceAccounts.
+- For a one-off backup verification Job, see the [read-only restore check](docs/postgres-backup-verification.md). It checks recovered kagent API v2 migration records, saved conversation contexts and pgvector, then verifies cleanup.
 - Turning the block on for a Cluster that already runs rolls its instance pods once (the plugin sidecar). The `immediate` first `Backup` fires before the sidecar exists and fails with `requested plugin is not available`; the next scheduled run, or a manual `Backup` with `method: plugin` and `pluginConfiguration.name: barman-cloud.cloudnative-pg.io`, succeeds. A Cluster created with the block does not hit this.
 
 ```bash

@@ -0,0 +1,31 @@
+# Optional cnpg-drill chart 0.1.6 values, not agent-platform chart values.
+# See ../postgres-backup-verification.md before running.
+cluster: kagent-pg
+drillClusterName: kagent-pg-restore
+recoveryObjectStore: kagent-pg-recovery-readonly
+suspended: true
+retainOnFailure: false
+timeoutSeconds: 1800
+maxBackupAgeSeconds: 90000
+# Supply a separate read-only role here only when the recovery ObjectStore
+# uses workload identity. Do not copy the source's writer role.
+recoveryServiceAccountAnnotations: {}
+checks:
+  # These checks target the kagent API v2 database, not the retained 0.10 DB.
+  # Update database/schema names if the installation overrides them.
+  - name: core-migrations
+    database: kagent_v2
+    query: SELECT count(*) > 0 FROM public.schema_migrations WHERE version_id > 0 AND is_applied
+    expected: "t"
+  - name: vector-migrations
+    database: kagent_v2
+    query: SELECT count(*) > 0 FROM public.vector_schema_migrations WHERE version_id > 0 AND is_applied
+    expected: "t"
+  - name: saved-conversation-context
+    database: kagent_v2
+    query: SELECT count(*) > 0 FROM public.a2a_context
+    expected: "t"
+  - name: vector-library
+    database: kagent_v2
+    query: SELECT '[1,0,0]'::public.vector <-> '[1,0,0]'::public.vector
+    expected: "0"
@@ -0,0 +1,179 @@
+# Verify a PostgreSQL backup with a disposable restore
+
+This is an optional one-off check for installations using
+`postgres.backup.method: plugin` with an S3-backed Barman Cloud ObjectStore.
+It uses [cnpg-drill](https://github.com/danielgaskins/cnpg-drill) chart 0.1.6 to
+restore an eligible completed backup, run read-only SQL assertions, and remove
+the recovery Cluster and PVCs. The release stays suspended afterward.
+
+## Before running
+
+- Confirm the CNPG operator, Barman Cloud plugin, source Cluster and archive are
+  healthy, with a completed plugin Backup within the required recovery window.
+  The example uses 25 hours for a daily backup; choose the actual requirement.
+- Check capacity for another database copy. Review the source storage class,
+  image pull Secrets and extension images. ImageVolume extensions need a
+  compatible Kubernetes/container runtime and PostgreSQL image. Confirm the
+  source and `<source>-restore` ImageVolume policy exception is installed on
+  clusters enforcing volume restrictions. For source-specific affinity or node
+  placement, use the native recovery procedure with that placement; cnpg-drill
+  does not copy those settings.
+- Provision a separate identity that can list the archive and read base backups
+  and WAL, including KMS decryption if needed. Verify reads succeed and uploads
+  and deletes are denied. Place its credentials in a Secret in the source
+  namespace with keys `accessKey` and `secretKey`; use the installation's Secret
+  manager. The commands below never print Secret values.
+- Review [the values example](examples/postgres-restore-check-values.yaml).
+  Its `kagent_v2`/`public` queries target the API v2 Goose migration histories,
+  not the retained 0.10 database. Adjust names for the installation. The saved
+  conversation check intentionally fails for an empty database: choose a known
+  application-data invariant appropriate to the backup instead of removing it.
+  The migration checks establish initialized histories, not an exact schema
+  version. Add version or data assertions for the recovery requirement.
+- Reserve `<sourceCluster>-restore` for one scheduler. This is the name selected
+  by the existing CNPG network policies. Do not run this Job alongside another
+  native or scheduled restore using that name. A leftover Cluster stops the
+  drill; inspect it before retrying.
+
+Use Bash, kubectl, Helm and jq. Set the readable kubeconfig used by both clients
+and the installation's context. Run the following steps in the same shell:
+
+```bash
+set -euo pipefail
+export KUBECONFIG="$HOME/.kube/config"
+export DRILL_CONTEXT="REPLACE_WITH_CONTEXT"
+export DRILL_NAMESPACE=kagent
+export SOURCE_CLUSTER=kagent-pg
+export RECOVERY_STORE=kagent-pg-recovery-readonly
+export RECOVERY_SECRET=kagent-pg-recovery-s3
+export DRILL_RELEASE=pg-backup-check
+export DRILL_JOB="pg-backup-check-$(date -u +%Y%m%d%H%M%S)"
+export DRILL_VALUES=docs/examples/postgres-restore-check-values.yaml
+```
+
+Keep the release name short and dedicated. Use a new Job name for a new drill.
+For installations with namespace-wide egress restrictions, allow the drill Job
+access to the Kubernetes API. The recovery Pods need API and archive access
+under the existing CNPG policies. Do not add general network access.
+
+## 1. Create a read-only recovery ObjectStore
+
+Discover the source's writer store and copy its archive configuration, replacing
+its credential references. This example uses static S3 keys. It does not change
+the source Cluster, writer store, retention policy, or writer identity.
+
+```bash
+set -euo pipefail
+WRITER_STORE=$(kubectl --context "$DRILL_CONTEXT" -n "$DRILL_NAMESPACE" \
+  get clusters.postgresql.cnpg.io "$SOURCE_CLUSTER" -o json |
+  jq -er '[.spec.plugins[]? | select(.name == "barman-cloud.cloudnative-pg.io" and .enabled != false)] |
+    if length == 1 then .[0].parameters.barmanObjectName else error("Expected one Barman plugin") end')
+[ "$RECOVERY_STORE" != "$WRITER_STORE" ]
+kubectl --context "$DRILL_CONTEXT" -n "$DRILL_NAMESPACE" \
+  get objectstores.barmancloud.cnpg.io "$WRITER_STORE" -o json |
+  jq -e --arg ns "$DRILL_NAMESPACE" --arg name "$RECOVERY_STORE" --arg secret "$RECOVERY_SECRET" '
+    if (.spec.configuration.destinationPath | startswith("s3://")) then
+      {apiVersion, kind, metadata: {name: $name, namespace: $ns},
+       spec: {configuration: (.spec.configuration |
+         del(.s3Credentials, .azureCredentials, .googleCredentials) |
+         .s3Credentials = {
+           accessKeyId: {name: $secret, key: "accessKey"},
+           secretAccessKey: {name: $secret, key: "secretKey"}})}}
+    else error("This example requires an S3 archive") end' > recovery-objectstore.json
+kubectl --context "$DRILL_CONTEXT" -n "$DRILL_NAMESPACE" \
+  apply --dry-run=server -f recovery-objectstore.json
+kubectl --context "$DRILL_CONTEXT" -n "$DRILL_NAMESPACE" \
+  apply -f recovery-objectstore.json
+```
+
+For IRSA, provision a separate read-only role trusted by
+`system:serviceaccount:<namespace>:<sourceCluster>-restore`. Set the recovery
+store's `s3Credentials` to `inheritFromIAMRole: true` and supply that role's ARN
+in `recoveryServiceAccountAnnotations.eks.amazonaws.com/role-arn` in the values
+file. The Crossplane-generated writer role also trusts restore names, but its
+write permissions make it unsuitable for this read-only check. Do not copy it.
+Verify the recovery role's access before running. AWS authentication has not
+been exercised by the local test described below.
+
+## 2. Preview and install the suspended check
+
+Render chart 0.1.6 and review its image digest, namespace Role, SQL assertions,
+source Cluster and recovery store. Then install the dedicated release:
+
+```bash
+set -euo pipefail
+helm template "$DRILL_RELEASE" oci://ghcr.io/danielgaskins/charts/cnpg-drill \
+  --version 0.1.6 --namespace "$DRILL_NAMESPACE" --values "$DRILL_VALUES" \
+  --set-string cluster="$SOURCE_CLUSTER" \
+  --set-string drillClusterName="${SOURCE_CLUSTER}-restore" \
+  --set-string recoveryObjectStore="$RECOVERY_STORE" --set suspended=true \
+  --set retainOnFailure=false > recovery-check.yaml
+kubectl --context "$DRILL_CONTEXT" -n "$DRILL_NAMESPACE" \
+  apply --dry-run=server -f recovery-check.yaml
+helm upgrade --install "$DRILL_RELEASE" oci://ghcr.io/danielgaskins/charts/cnpg-drill \
+  --version 0.1.6 --namespace "$DRILL_NAMESPACE" --values "$DRILL_VALUES" \
+  --set-string cluster="$SOURCE_CLUSTER" \
+  --set-string drillClusterName="${SOURCE_CLUSTER}-restore" \
+  --set-string recoveryObjectStore="$RECOVERY_STORE" --set suspended=true \
+  --set retainOnFailure=false --kube-context "$DRILL_CONTEXT" --wait --timeout 5m
+kubectl --context "$DRILL_CONTEXT" -n "$DRILL_NAMESPACE" \
+  get cronjob "${DRILL_RELEASE}-cnpg-drill" -o json |
+  jq -e '.spec.suspend == true and .spec.concurrencyPolicy == "Forbid"'
+```
+
+## 3. Run and validate one restore
+
+```bash
+set -euo pipefail
+kubectl --context "$DRILL_CONTEXT" -n "$DRILL_NAMESPACE" \
+  create job "$DRILL_JOB" --from="cronjob/${DRILL_RELEASE}-cnpg-drill"
+drill_deadline=$((SECONDS + 2400))
+while true; do
+  drill_state=$(kubectl --context "$DRILL_CONTEXT" -n "$DRILL_NAMESPACE" \
+    get job "$DRILL_JOB" -o json | jq -r '
+      if any(.status.conditions[]?; .type == "Complete" and .status == "True") then "complete"
+      elif any(.status.conditions[]?; .type == "Failed" and .status == "True") then "failed"
+      else "pending" end')
+  [ "$drill_state" = pending ] || break
+  [ "$SECONDS" -lt "$drill_deadline" ] || { echo "Drill timed out; inspect the Job and recovery resources" >&2; exit 1; }
+  sleep 5
+done
+kubectl --context "$DRILL_CONTEXT" -n "$DRILL_NAMESPACE" \
+  logs "job/$DRILL_JOB" -c drill > recovery-result.json
+jq -e -s 'length == 1 and (.[0] |
+  .source == (env.DRILL_NAMESPACE + "/" + env.SOURCE_CLUSTER) and
+  .drillCluster == (env.SOURCE_CLUSTER + "-restore") and
+  .status == "passed" and .cleanup == "cluster-and-pvcs-deleted" and
+  (.checks | length > 0) and all(.checks[]; .passed == true))' recovery-result.json
+```
+
+A pass covers the selected backup, archive path and configured assertions. It
+does not establish regional recovery, restore all Substrate snapshot content,
+or compare every application row. The report hashes query results. Save it
+with the backup ID, source UID, recovery time and any PITR target.
+
+## 4. Inspect cleanup and keep the report
+
+Confirm `<sourceCluster>-restore`, its Pods and PVCs are gone. Inspect backing
+PVs or cloud disks under the storage reclaim policy, and recheck source health.
+Keep the Job until its report is saved. The dedicated release remains suspended;
+remove it with `helm uninstall` if no longer needed. Do not unsuspend it until
+the data checks, credential restrictions, storage cleanup and costs are reviewed.
+
+If the Job fails or times out, save its logs and treat the drill as failed.
+`archive_access_denied` and `wal_unavailable` need archive investigation. A
+missing report after termination is incomplete evidence. Inspect leftovers
+before retrying; remove only confirmed recovery resources, never the source.
+A leftover Cluster contains a copy of application data. Reusing a fixed name
+before the previous run's storage cleanup finishes can fail that cleanup check.
+
+## Validation scope
+
+The commands and values were tested with a local PostgreSQL fixture on Kubernetes
+1.35.8, CNPG 1.30.1, Barman Cloud plugin 0.15.0, PostgreSQL 18.6, pgvector 0.8.2
+and RustFS 1.0.0. The fixture applied the API v2 baseline SQL, seeded migration
+tracking records and a synthetic conversation context; it did not run a kagent
+controller. The published cnpg-drill chart 0.1.6 restored that data and removed
+the temporary resources. Giant Swarm's IAM, Cilium policies, admission rules,
+private image registry and cloud storage were not tested live. Repeat the first
+run in the installation before relying on this result.
@@ -11,8 +11,9 @@
 # Cluster never reaches an active instance. kagent's controller then crashloops on
 # its pgvector migration check, because there is no database to connect to.
 #
-# Scope: the cnpg.io/cluster label the operator sets on this Cluster's instance
-# pods and on its own bootstrap/join Jobs. Both carry the ImageVolume. The rule
+# Scope: the source and <clusterName>-restore labels the operator sets on
+# instance pods and bootstrap/recovery Jobs. The restore needs the same
+# extension image as the source. The rule
 # is restrict-volume-types' restricted-volumes (kyvernoPolicies.rules).
 #
 # PolicyExceptions must live in policy-exceptions, giantswarm, flux-giantswarm or
@@ -36,7 +37,11 @@ spec:
         namespaces:
         - {{ $ns }}
         selector:
-          matchLabels:
-            cnpg.io/cluster: {{ $clusterName }}
+          matchExpressions:
+          - key: cnpg.io/cluster
+            operator: In
+            values:
+            - {{ $clusterName }}
+            - {{ $clusterName }}-restore
 {{- end }}
 {{- end }}
@@ -0,0 +1,46 @@
+#!/usr/bin/env python3
+"""Check ImageVolume admission selectors for source and recovery workloads."""
+
+import os
+import re
+import subprocess
+import sys
+from pathlib import Path
+
+chart = sys.argv[1]
+helm = os.environ.get("HELM", "helm")
+values = Path(chart) / "ci/test-postgres-backup-aws-values.yaml"
+
+
+def render(*overrides):
+    return subprocess.check_output(
+        [helm, "template", "t", chart, "-f", str(values),
+         "--set", "kyvernoPolicies.enabled=true", *overrides],
+        text=True,
+    )
+
+
+for name, namespace in (("kagent-pg", "kagent"), ("custom-pg", "kagent")):
+    output = render("--set", f"postgres.clusterName={name}",
+                    "--set", f"postgres.namespace={namespace}",
+                    "--set", f"kagent.namespaceOverride={namespace}")
+    exception = next(part for part in output.split("\n---")
+                     if "kind: PolicyException" in part
+                     and f"name: {name}-image-volume" in part)
+    assert re.search(r"kinds:\s*\n\s*- Pod\s*\n\s*- Job", exception)
+    assert re.search(rf"namespaces:\s*\n\s*- {namespace}\s*\n", exception)
+    assert "key: cnpg.io/cluster" in exception
+    assert "operator: In" in exception
+    selector = exception.split("key: cnpg.io/cluster", 1)[1]
+    allowed = re.findall(r"^\s*- ([a-z0-9-]+)\s*$", selector, re.M)
+    for candidate, expected in ((name, True), (name + "-restore", True),
+                                (name + "-restore-other", False),
+                                ("unrelated", False)):
+        assert (candidate in allowed) == expected, (candidate, allowed)
+    assert set(allowed) == {name, name + "-restore"}
+
+output = render("--set", "postgres.vector.enabled=false")
+assert "-image-volume" not in output
+output = render("--set", "postgres.vector.extensionImage.reference=")
+assert "-image-volume" not in output
+print("ok: source/recovery ImageVolume selectors, unrelated-name rejection, and extension gating")