The provisioner exits after 30 minutes of idle.

[jsafrane]

What happened:

Automatic gRPC bump to 1.59.0 introduced a new gRPC behavior that closes idle connections after 30 minutes of inactivity. After 30 minutes of no provisioning / deletion, the connection to a CSI driver is silently closed. At the next provisioning / deletion, the provisioner realizes the connection is closed and exits with Lost connection to CSI driver, exiting. A new provisioner starts immediately, but it must wait for leader election to expire, which adds quite a long delay to volume provisioning (and our downstream e2e tests time out).

What you expected to happen:

The gRPC connection should not close because of inactivity.

How to reproduce it:
On a very quiet cluster (no provisioning/deletion), wait for 30 minutes after external-provisioner start and create a new PVC that should be dynamically provisioned.

[jsafrane]

I filed kubernetes-csi/csi-lib-utils#153 to disable autoclose.

[jsafrane]

/assign

[jsafrane]

/reopen
we still need to vendor new csi-lib-utils here

[k8s-ci-robot]

@jsafrane: Reopened this issue.

In response to this:

/reopen
we still need to vendor new csi-lib-utils here

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sathieu]

@jsafrane What is the status of this issue ?

[sathieu]

Maybe there is another issue around this: when the socket times out, the container fails without releasing the lease. Is this intended? Once restarted, the container doesn't recover the lease, we have to wait for the lease timeout (300s with vsphere-csi).