Add timeout to Ceph GET API calls

[Madhu-1]

Describe the feature you'd like to have

Provide a way to configure the timeout for the ceph. Get API calls to avoid command stuck if there is some problem between the ceph cluster and csi driver (cluster health, slow ops, or short network connectivity problem)

What is the value to the end user? (why is it a priority?)

Currently, if ceph doesn't responds to any CSI calls the cephcsi will start throwing an operation already exists error message even if the ceph cluster is recovered, the only way to recover the csi driver is to restart the csi pods. Restarting csi driver pods is not an optimal solution in most of the production clusters.

How would the end user gain value from having this feature?

Avoid restarting csi pods in the production clusters even if any GET API call is stuck. The ask is to add timeout only to GET API calls not for any other operations to avoid stale resources in the cluster.

But again we need to consider all the corner cases carefully before doing this change.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

[Rakshith-R]

/assign @karthik-us

[nixpanic]

Maybe context.WithTimeout() can be used in the ConnPool.Get() function. The Get() could return an additional context, which would be cancelled after the timeout. The ConnPool.Put() should then mark the context as successful completed.

If the context times out, the ConnPool logic should remove the connection from the pool, and disconnect (which hopefully works, but may block indefinitely?) it.