Upgrade fails when per-pod PVCs remain Pending (block-node StatefulSet) — upgrade/installation rollback

[mahmoudian1]

Bill of Materials, Application, or CLI Version

sudo solo-provisioner block node install --profile=local --chart-version=0.26.2 --values=/path/to/values-override.yaml

Describe the bug

When running solo-provisioner to install/upgrade the block node (chart v0.26.2), the StatefulSet creates per-pod PVCs (e.g. live-storage-block-node-block-node-server-0) that remain Pending because no matching PVs exist. The Pending PVCs prevent pod scheduling, Helm times out waiting for the pod to become Ready, and because Helm runs with atomic=true the release is rolled back and the upgrade fails.

This issue groups the relevant troubleshooting logs and requests clearer documentation and/or a migration approach so upgrades/installations do not fail in this way.

Describe the expected behavior

This issue shouldn't happen and the dependent resources should be created. The installer/upgrade should detect missing per-pod PVs and emit a clear, actionable error pointing to the Pending per-pod PVC names and recommended remediation steps.

To Reproduce

Prepare a host machine and run:

sudo solo-provisioner block node install --profile=local --chart-version=0.26.2 --values=/path/to/values-override.yaml
(Or run upgrade similarly:)

sudo solo-provisioner block node upgrade --profile=local --chart-version=0.26.2 --with-reset --values=values-override.yaml
Watch the block-node namespace resources:

kubectl -n block-node get pods
kubectl -n block-node get pvc
kubectl -n block-node get pv

Observed behavior (logs / excerpts)
Pod pending due to unbound PVCs:

weaver@bn-lfh01-previewnet:~$ kubectl -n block-node get po
NAME                             READY   STATUS    RESTARTS   AGE
block-node-block-node-server-0   0/1     Pending   0          11s
Pod describe shows Pending + unbound PVCs:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  22s   default-scheduler  0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims.
PVCs show Pending (per-pod PVCs), while other PV/PVC entries show some PVs already Bound:

kubectl -n block-node get pvc
NAME                                                  STATUS    VOLUME                    CAPACITY
archive-storage-block-node-block-node-server-0        Pending
archive-storage-pvc                                   Bound     archive-storage-pv        5Gi
live-storage-block-node-block-node-server-0           Pending
live-storage-pvc                                      Bound     live-storage-pv           5Gi
logging-storage-block-node-block-node-server-0        Pending
logging-storage-pvc                                   Bound     logging-storage-pv        5Gi
verification-storage-block-node-block-node-server-0   Pending
verification-storage-pvc                              Bound     verification-storage-pv   5Gi

Solo-provisioner / Helm timed out waiting and rolled back due to atomic=true:

2026-02-25T21:12:44Z DBG Patch StatefulSet "block-node-block-node-server" ...
2026-02-25T21:12:44Z DBG error updating the resource "...": cannot patch "block-node-block-node-server" with kind StatefulSet: StatefulSet.apps "block-node-block-node-server" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
...
2026-02-25T21:12:45Z ERR Helm upgrade failed error="helm.upgrade_failed: failed to run upgrade action, cause: release block-node failed, and has been rolled back due to atomic being set: cannot patch "block-node-block-node-server" ...

Example values override used:

cat values-override.yaml
blockNode:
   config:
    BLOCK_NODE_EARLIEST_MANAGED_BLOCK: "270691"
    BACKFILL_START_BLOCK: "270691"

Additional Context

No response

[brunodam]

@mahmoudian1 , a few questions:

• what was the version of solo-provisioner used? v0.9.0-rc.1 or v0.8.0
• can you share values-override.yaml contents on both runs

[mahmoudian1]

@mahmoudian1 , a few questions:

• what was the version of solo-provisioner used? v0.9.0-rc.1 or v0.8.0
• can you share values-override.yaml contents on both runs

weaver@bn-rfh01-previewnet:~/0.27.0$ cat values-override.yaml
resources:
  requests:
    cpu: "8"
    memory: "12Gi"
  limits:
    cpu: "8"
    memory: "12Gi"

labels:
  app.kubernetes.io/instance: bn-previewnet-rfh01  # Unique per node
  nodeType: rfh
  nodeId: rfh01
  network: previewnet

blockNode:
  config:
    secretRef: "bn-previewnet-archiver-gcs-hmac"
    ARCHIVE_BUCKET_NAME: "bn-archive-previewnet"
    ARCHIVE_ENDPOINT_URL: "https://storage.googleapis.com"
    ARCHIVE_STORAGE_CLASS: "ARCHIVE"
    ARCHIVE_REGION_NAME: "us-central1"
    BLOCK_NODE_EARLIEST_MANAGED_BLOCK: "1000"
    BACKFILL_BLOCK_NODE_SOURCES_PATH: /opt/hiero/block-node/backfill/block-node-sources.json
    BACKFILL_FETCH_BATCH_SIZE: "20"
    BACKFILL_START_BLOCK: "1000"
    JAVA_OPTS: '-Xms8G -Xmx8G -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/tmp/dump.hprof"'
    BACKFILL_GREEDY: "true"
  backfill:
    # The path to the block-node-sources.json file must match the path property in
    # blockNode.config.BACKFILL_BLOCK_NODE_SOURCES_PATH
    path: /opt/hiero/block-node/backfill
    filename: block-node-sources.json
    sources:
      - address: lfh01.previewnet.blocknode.hashgraph-devops.com
        port: 40840
        priority: 1

weaver@bn-rfh01-previewnet:~/0.27.0$ sudo solo-provisioner --version
2026-02-27T03:16:52Z INF Workflow report is saved pid=152712 report_path=/opt/solo/weaver/logs/setup_report_20260227_031652.yaml
version: |
    0.8.0
commit: b55ec0e45108ecbbbd55e473f1105b64066303a7
go: go1.25.2