Creation of additional storage classes with dynamic volume provisioning: impossible or documentation lacking?

[victor-sudakov]

Running minikube version: v1.20.0, commit: c61663e

There is only one storage class created when starting a minikube:

$ kubectl get sc
NAME                 PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   k8s.io/minikube-hostpath   Delete          Immediate           false                  18h

Dynamic Provisioning in this storage class works fine.

I would like to add more storage classes with dynamic volume provisioning, so that I could test my PersistentVolumeClaim manifests (written for EKS) on Minikube without modifications. These storage classes need not be different physically or programmatically from the standard Minikube sc, but I need to test manifests for other clusters without modifications. Those manifests have different "storageClassName" attributes, like "gp3", "sc1" etc.

If the creation of additional storage classes with dynamic volume provisioning is possible, I cannot find it anywhere if it's documented for Minikube.

[ilya-zuyev]

@victor-sudakov in general, adding a new storage class to minikube is the same process as for regular kubernetes cluster.
https://kubernetes.io/docs/concepts/storage/storage-classes/

I guess, there's some specific for some SCs, related to minikube internal environment. If you have any issues, please feel free to report it here.

➜  minikube git:(master) ✗ kubectl get sc
NAME                 PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   k8s.io/minikube-hostpath   Delete          Immediate           false                  19h

➜  minikube git:(master) ✗ kubectl apply -f - <<EOF 
heredoc> apiVersion: storage.k8s.io/v1
heredoc> kind: StorageClass
heredoc> metadata:
heredoc>   name: local-storage
heredoc> provisioner: kubernetes.io/no-provisioner
heredoc> volumeBindingMode: WaitForFirstConsumer
heredoc> EOF
storageclass.storage.k8s.io/local-storage created

➜  minikube git:(master) ✗ kubectl get sc
NAME                 PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-storage        kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  4s
standard (default)   k8s.io/minikube-hostpath       Delete          Immediate              false                  19h

[victor-sudakov]

@ilya-zuyev I have a question. What is with this provisioner: kubernetes.io/no-provisioner parameter in your example? Should not it be provisioner: k8s.io/minikube-hostpath ? Will your storage class even work (allocate storage on the host) that way?

[victor-sudakov]

OK, I have created a SC like you advised

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp2
provisioner: k8s.io/no-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

The storage class is there

$ kubectl get sc
NAME                 PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2                  k8s.io/no-provisioner      Delete          WaitForFirstConsumer   false                  2m21s

but my PVCs using that storage class are in the Pending state with the message "waiting for a volume to be created, either by external provisioner "k8s.io/no-provisioner" or manually created by system administrator"

Obviously I'm doing something wrong, or your advice is incomplete?

[victor-sudakov]

Maybe it's useful to provide the complete output:

$ kubectl describe persistentvolumeclaim/ebspod1
Name:          ebspod1
Namespace:     default
StorageClass:  gp2
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: k8s.io/no-provisioner
               volume.kubernetes.io/selected-node: minikube
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       diskopod1
Events:
  Type    Reason                Age                  From                         Message
  ----    ------                ----                 ----                         -------
  Normal  WaitForFirstConsumer  2m15s                persistentvolume-controller  waiting for first consumer to be created before binding
  Normal  ExternalProvisioning  2s (x11 over 2m15s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "k8s.io/no-provisioner" or manually created by system administrator
$ 

[victor-sudakov]

If I create a storage class with provisioner: k8s.io/minikube-hostpath hoping for dynamic volume provisioning to work:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: io1
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
provisioner: k8s.io/minikube-hostpath
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Then PVCs fail with the failed to get target node: nodes "minikube" is forbidden: User "system:serviceaccount:kube-system:storage-provisioner" cannot get resource "nodes" in API group "" at the cluster scope error:

Events:
  Type     Reason                Age                  From                                                                    Message
  ----     ------                ----                 ----                                                                    -------
  Normal   WaitForFirstConsumer  2m15s                persistentvolume-controller                                             waiting for first consumer to be created before binding
  Warning  ProvisioningFailed    30s (x4 over 2m15s)  k8s.io/minikube-hostpath_minikube_40f19f28-8735-4784-ab6d-ccca0a4d856a  failed to get target node: nodes "minikube" is forbidden: User "system:serviceaccount:kube-system:storage-provisioner" cannot get resource "nodes" in API group "" at the cluster scope
  Normal   ExternalProvisioning  7s (x11 over 2m15s)  persistentvolume-controller                                             waiting for a volume to be created, either by external provisioner "k8s.io/minikube-hostpath" or manually created by system administrator

[victor-sudakov]

I have actually found a workaround. I've created many PVs, and several SCs named "gp2", "gp3" referring to those static PVs. So now I can use those SCs for testing purposes. But why cannot I have those PVs created dynamically for me by Minikube's dynamic storage controller?
stupid_workaround.yaml.txt

[victor-sudakov]

Attached
testcase.yaml.txt
is a complete testcase for the problem. Apply the manifest and watch the "diskopod1" pod never starting.

[victor-sudakov]

So, how can I make non-default SCs use the minikube-hostpath provisioner (in order to allocate PVs dynamically)?

[medyagh]

@victor-sudakov tthanks for providing your experience and sharing the work arround ! this could be a tutorial on our website I would accept a PR to add it to our website

and if you want you could make minikube storage provisioneer configurable

@victor-sudakov the storage provisioner is implemented as an addon itself so you should be able to disable the default one

$ minikube addons disable storage-provisioner

does that help ?

[medyagh]

/triage needs-information

[victor-sudakov]

the storage provisioner is implemented as an addon itself so you should be able to disable the default one

$ minikube addons disable storage-provisioner

does that help ?

Actually, the bundled storage provisioner is a very nice thing to have, I don't want to disable it, I would prefer for it to support arbitrary storage classes transparently.

[sharifelgamal]

Yeah this just isn't something our storage-provisioner/default-storageclass addons support currently, but there is no theoretical reason why we couldn't. We would happily accept a PR that does this.

[victor-sudakov]

Yeah this just isn't something our storage-provisioner/default-storageclass addons support currently, but there is no theoretical reason why we couldn't. We would happily accept a PR that does this.

I'm afraid I'm not qualified enough to offer a PR for the issue. Thanks for the response that this is currently technically not possible and not a misconfiguration on my part. Shall we close the issue?

[yayaha]

I'd like to contribute, but I need someone with enough knowledge to point me to the right direction. Thanks!

[medyagh]

@victor-sudakov there is a PR that says could fix this here is the link to the binary from a PR that I think might fix this issue, do you mind trying it out ? #12797

do u mind trying the binary of that PR and see if that fixes the issue?

http://storage.googleapis.com/minikube-builds/12797/minikube-linux-amd64
http://storage.googleapis.com/minikube-builds/12797/minikube-darwin-amd64
http://storage.googleapis.com/minikube-builds/12797/minikube-windows-amd64.exe

[victor-sudakov]

@victor-sudakov there is a PR that says could fix this here is the link to the binary from a PR that I think might fix this issue, do you mind trying it out ? #12797

do u mind trying the binary of that PR and see if that fixes the issue?

@medyagh Yes, with that minikube binary my test case (see above) is now working and pod running:

$ kubectl get sc,pv,pvc
NAME                                             PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/gp5                  k8s.io/minikube-hostpath   Delete          WaitForFirstConsumer   false                  3m25s
storageclass.storage.k8s.io/standard (default)   k8s.io/minikube-hostpath   Delete          Immediate              false                  5m8s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM           STORAGECLASS   REASON   AGE
persistentvolume/pvc-0cd25d47-1129-4cf6-81a0-0cdbdbf765a1   1Gi        RWO            Delete           Bound    test1/ebspod1   gp5                     3m24s

NAME                            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/ebspod1   Bound    pvc-0cd25d47-1129-4cf6-81a0-0cdbdbf765a1   1Gi        RWO            gp5            3m25s
$ 

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[victor-sudakov]

/remove-lifecycle stale

[cvetomir-todorov]

+1 from me.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[victor-sudakov]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[victor-sudakov]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[DerekTBrown]

I had the same issue/experience. Ultimately, here is what ended up working:

# tilt-resources.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: storage-provisioner-cluster-role-binding
  namespace: kube-system
subjects:
- kind: ServiceAccount
  name: storage-provisioner
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: my-storage-class
provisioner: k8s.io/minikube-hostpath
volumeBindingMode: WaitForFirstConsumer

# Tiltfile
k8s_yaml("./tilt-resources.yaml")

It seems like the minikube storage provisioner addon should grant the provisioner the appropriate permissions from the get-go.

[mindthecap]

To fix the cannot get resource "nodes" in API group "" at the cluster scope a better approach is to add minimal required rights for the user.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: storage-provisioner-minikube-volume-provisioner
  namespace: kube-system
subjects:
  - kind: ServiceAccount
    name: storage-provisioner
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: minikube-volume-provisioner
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: minikube-volume-provisioner
rules:
  - verbs:
      - get
      - list
      - watch
    apiGroups:
      - ""
    resources:
      - nodes

This will add a new ClusterRole for minikube volume provisioner and attaches the role to storage-provisioner. I don't know if the account names are persistent between Minikube versions, i'm stuck with v1.31.2 right now.