Hostpath PV is resized but PVC is not when resizing offline

[xing-yang]

When trying to expand a volume which is offline using the CSI hostpath driver, PV is expanded but PVC is not. There is probably a bug with CSI hostpath driver implementation, but external-resizer should do the check and prevent this from happening.

How to reproduce:

1. I deployed csi-driver-host-path for k8s 1.14 using the script in the following location.
   https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy/kubernetes-1.14

2. kubectl create -f https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/examples/csi-storageclass.yaml

3. kubectl apply -f https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/examples/csi-pvc.yaml

4. https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/examples/csi-app.yaml

5. Modified csi-pvc.yaml to increase the PVC size to 2Gi.
   external-resizer worked properly and expanded online PVC/PV size to 2Gi.

6. Deleted the pod created using csi-app.yaml. PVC is now offline.

7. Modified csi-pvc.yaml to change the PVC size to 3Gi.

8. PV was increased to 3Gi, but PVC remained at 2Gi.

root@k8s_ubuntu1:~/go/src/github.com/kubernetes-csi/csi-driver-host-path/examples# kubectl get pvc
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
csi-pvc   Bound    pvc-67c942df-df41-11e9-a872-000c296bb855   2Gi        RWO            csi-hostpath-sc   15h
root@k8s_ubuntu1:~/go/src/github.com/kubernetes-csi/csi-driver-host-path/examples# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS      REASON   AGE
pvc-67c942df-df41-11e9-a872-000c296bb855   3Gi        RWO            Delete           Bound    default/csi-pvc   csi-hostpath-sc            15h

[xing-yang]

I don't see PluginCapability_VolumeExpansion_ONLINE or PluginCapability_VolumeExpansion_OFFLINE returned by the CSI hostpath driver: https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/pkg/hostpath/identityserver.go#L60. Does the resize controller check this capability?

[msau42]

/assign @gnufied

[zhucan]

When trying to expand a volume which is offline using the CSI hostpath driver, PV is expanded but PVC is not. There is probably a bug with CSI hostpath driver implementation, but external-resizer should do the check and prevent this from happening.

How to reproduce:

1. I deployed csi-driver-host-path for k8s 1.14 using the script in the following location.
   https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy/kubernetes-1.14
2. kubectl create -f https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/examples/csi-storageclass.yaml
3. kubectl apply -f https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/examples/csi-pvc.yaml
4. https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/examples/csi-app.yaml
5. Modified csi-pvc.yaml to increase the PVC size to 2Gi.
   external-resizer worked properly and expanded online PVC/PV size to 2Gi.
6. Deleted the pod created using csi-app.yaml. PVC is now offline.
7. Modified csi-pvc.yaml to change the PVC size to 3Gi.
8. PV was increased to 3Gi, but PVC remained at 2Gi.

root@k8s_ubuntu1:~/go/src/github.com/kubernetes-csi/csi-driver-host-path/examples# kubectl get pvc
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
csi-pvc   Bound    pvc-67c942df-df41-11e9-a872-000c296bb855   2Gi        RWO            csi-hostpath-sc   15h
root@k8s_ubuntu1:~/go/src/github.com/kubernetes-csi/csi-driver-host-path/examples# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS      REASON   AGE
pvc-67c942df-df41-11e9-a872-000c296bb855   3Gi        RWO            Delete           Bound    default/csi-pvc   csi-hostpath-sc            15h

+1

[gnufied]

The external resize controller does not currently verify offline/online capabilities. We could but then we will also have to maintain a pod informer in external-resizer to check if volume is in-use or not.

But this bug seems to be unrelated to that - it looks like @xing-yang you are expecting to expand a offline PVC and then without creating a Pod, you expect the PVC to be fully expanded? Offline does not mean that PVC will be expanded fully if Pod is offline, the Pod still has to be created for expansion on the node to happen which will finish the resizing process.

The only way to skip expansion on the node is to return false from ControllerExpandVolume request

[xing-yang]

@gnufied I'm expecting PV should not be expanded at all because host path csi driver does not support offline expansion.

I think resizer should be checking if pvc is used by a pod. I opened another issue for that: #62. This current issue is to make sure host path driver itself is implemented correctly.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[xing-yang]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[xing-yang]

/remove-lifecycle stale