Refactor connection with retries and backoff

[vladimirvivien]

This PR attempts to fix #76
It retries to connect to the driver (within the same session) several times before giving up. The logic is as follows:

• Within apimachinery/wait.ExponentialBackoff condition do 5 times:
  • Attempt to connect with DialContext:
    • Block until a connection is successful
    • Give up if --connection-timeout reached
  • Evaluate any error from DialContext, if retryable:
    • Attempt to reconnect (based on error type)
  • If not, giveup right away
  • if number of attempts reached
    • Give up, stop
    • Else try again

Return successful connection or error if one was generated

Using flag --connection-timeout can be used to control how long a connection request lasts before the gives up and try again. Because this will retry right away, 5 times total, flag --connection-timeout should be set to a sensible value, something like 5 to 10 secs. For instance a 5 second time out will produce the following total attempt time:

total connection attempt time ≈ 5 sec * 5 attempts * backoffFactor if no connection is created within that time period, the code will stop.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vladimirvivien

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vladimirvivien]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[pohly]

It's not obvious to me at all how this will handle the situation that the CSI driver hasn't created the csi.socket yet. Does that go to the exponential backoff or will it be treated as a permanent error and the code returns immediately?

This probably needs a unit test.

[vladimirvivien]

@pohly thanks for taking a look.

Yeah I just added/fixed the test. If that is not enough I can add more.

So I did look into the grpc code trying to understand how grpc connection behave. When block is enabled and DialContext is used, the code will wait timeout time until (socket file is present) connection is established. The backoff strategy just retries several times until it gives up. Which forces k8s to relaunch the container again.

[vladimirvivien]

/hold

[saad-ali]

Let's move this PR to https://github.com/kubernetes-csi/node-driver-registrar since this repo is deprecated.

[lpabon]

🛂 ⛔️
Hi, this repo has been split up into two repos:

• cluster-driver-registrar: Registers the driver with Kubernetes by setting up a CRD
• node-driver-registrar: Registers the driver with the kubelet on the node.

This repo has been closed, and no new changes will be accepted. Please move your content to one of these repos.

Thank you,

[pohly]

Here's a renewed effort by @darkowlzz to get the gRPC code enhanced: kubernetes-csi/csi-lib-utils#8