hyperdisk-balanced-high-availability, GKE storage class and ReadOnlyMany

[GregoireW]

I'm in a situation where I use a regional persistant disk on GKE (storage class, PV, PVC ) and on PV / PVC I can use the ReadOnlyMany so that any pod on my (n1/n2/n2d) node can read the data I got in this disk.

my GKE is on 2 zones, my nodes are on those 2 zones, my regional-pd is on the 2 zones, there is no issue, I can attach this disk on multiple node and be used by multiple pods.

I try to switch to node that only support hyperdisk (n4/c3/c3d...) So I created an hyperdisk of type hyperdisk-balanced-high-availability, and with the access mode 'multiple VMs read write'. (I would be happy with a read only on multiple VM but it doesn't exist so I switch to that. )

On GKE, I created a storage class like:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-hyperdisk-balanced-high-availability
parameters:
  type: hyperdisk-balanced-high-availability
  provisioned-throughput-on-create: "250Mi"
  provisioned-iops-on-create: "7000"
provisioner: pd.csi.storage.gke.io
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

and I try to create a PV / PVC like

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv-hyperdisk
spec:
  storageClassName: "csi-hyperdisk-balanced-high-availability"
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
                - my-region-xxx-a
                - my-region-xxx-c
            - key: topology.kubernetes.io/region
              operator: In
              values:
                - my-region-xxx
  csi:
    driver: "pd.csi.storage.gke.io"
    readOnly: false
    volumeHandle: "projects/my-proj/regions/my-region-xxx/disks/my-hyperdisk"

pvc is referencing the PV with the same access mode.

My issue is when I try to set a pod with this volume, I got an error because of the access mode:

• If I set ReadWriteMany => the controller set an error ( I think this one:

  gcp-compute-persistent-disk-csi-driver/pkg/gce-pd-csi-driver/utils.go

  Line 136 in c08bad5

  if mnt != nil && mod == csi.VolumeCapability_AccessMode_MULTI_NODE_MULTI_WRITER {

  )
• If I set ReadWriteOnce => I can only mount 1 node ( Multi-Attach error for volume "xxxx" Volume is already used by pod(s) ... )
• If I set ReadOnlyMany => I got an error about the disk not able to be mount in read only

So my question is: What can I do to make this work ? How can I mount my disk multiple time on different node on different zone from my GKE cluster ?

[muffl0n]

How did you manage to get hyperdisk-balanced-high-availability to work in the first place? Even when I'm adding the argument --allow-hdha-provisioning to the DaemonSet, I'm getting the error

failed to extract parameters: parameters contain invalid disk type hyperdisk-balanced-high-availability

I'm on GKE 1.32.x with gcp-compute-persistent-disk-csi-driver 1.16.1.

[GregoireW]

I create disk beforehand so that I get a clean name

[lindeskar]

How did you manage to get hyperdisk-balanced-high-availability to work in the first place?

I think we have related issues, so I created #2090 with this description:

I created a bunch of PVCs using a StorageClass with hyperdisk-balanced-high-availability when running 1.31.6-gke.1064001.

After an upgrade to 1.32.2-gke.1297002 the same StorageClass now gives "invalid disk type" for newly created PVCs.

gcp-compute-persistent-disk-csi-driver:v1.14.2-gke.8 was running in 1.31.6 while we now see gcp-compute-persistent-disk-csi-driver:v1.16.1-gke.0 in 1.32.2.

GKE release notes from yesterday say the disk type is available in 1.33 - but I have clear evidence it's also available in 1.31! 🤯

Please help me understand why we see this regression from 1.14.2. 🙏