Synchronous mode of operations

[oritnm]

The operations defined in the spec are sync mode. What is the expectation of the CO upon receiving a completion response ?

• Is it an acknowledgment of the SP of receiving the command and guaranteeing that it will be processed? OR
• Is it expect the operation to be completed by the SP?

If it is the latter, then an async mode of operation needs to be taken under consideration. There may be use-cases in which the SPs would not be able to complete a request within a timeout interval. Such a case for example could be delete of a volume - the operation may require time and if the CO counts on the capacity to be released by the time the completion is received then it is problematic.

[jieyu]

As it is currently, when the CO received a response from the SP and it's successful, the CO will assume the operation has been done. So SP shouldn't send a response until the operation is actually done (e.g., send the response after the volume has been deleted).

The timeout logic is entirely up to the CO, depending on how long the CO wants to wait. Given the operation has to be idempotent, the CO can always choose to re-issue the RPC after a timeout.

[oritnm]

The thing is that every SP may have a different underlying mechanism for an operation to get completed, different operations may take different times depending on the underlying implementation and size of volume , for example deleting a filesystem with billions of files underneath it or deleting a large iscsi volume with snapshots could affect the time it takes to complete the operations. Since obviously you don't want the CO to have a logic on a per operation per SP (and even per size) then I think it may be good to support async mode.
The retry mechanism could solve this issue but requires supporting some kind of an exponential backoff mechanism for a very long operation. These retries on a long operation though will be completed at the end will cost in "bothering" both the CO and the SPs and could lead to scalability issues.

[jieyu]

@oritnm I totally agree with you that CO should not have SP specific logic. With that in mind, sounds like there are two options:

1. Introduce async mechanism as you suggested. One async mechanism I can think of is to use stream response for certain calls. Do you have any suggestions? It'll complicate the CO logic I can imagine. Anyone has other thoughts?

2. CO waits forever on an operation. That eliminates the SP specific logics in the CO. I need to double check if gRPC will give you an error if the SP service is down.

You mentioned that retrying at the CO side will "bother" both SP and CO. My feeling is that SP needs to deal with idempotency anyway, so I don't see an increase of complexity there. For CO, retrying definitely increase the complexity, but shouldn't be that complex.

[oritnm]

SP must be able to deal with idempotency since its responses may get lost, however the likelihood of such cases is not not extremely frequent (although obviously not rare) but if long operation handling will use the timeout and retry mechanism as the mean to establish long operations, then in case of a (really) long operation the SP may receive several such retires which all should be identified as a retry of an operation in process and withdrawn (leaving always the last request so the response will be on this request).

[Jgoldick]

If one imagines that a NodePublishVolume involves a node executing a file system mount operation and a slow FSCK has to run first, this could take a while. If the CO times out the operation and retries the Publish on a new node, the FSCK could be interrupted with no progress actually being made...or worse having a permanently corrupt FS. While most file systems are quick recovery these days, that's not an explicit requirement. This case might raise the temperature of this issue.

[jieyu]

Raised this issue during the working group meeting. We all agree that's something we'll look into in the future, but probably not in v1. The simpler the first version is, the more likely that this spec will be successful.

If an operation timed out in the CO, the CO shouldn't assume that the operation is either successful or failed. For instance, if NodePublishVolume timed out, the CO should assume that it might be successful, therefore, shouldn't call NodePublishVolume in other nodes if its access mode is SINGLE_NODE_WRITER. The CO should either call NodeUnpublishVolume, or call NodePublishVolume again.

[oritnm]

@jieyu as noted above, while the sync mode seems easier at first look I think it is problematic, a long operation such as deleting a volume may take a long time which will need get timeouted many times unless your timeouts are in the order of mins as may take in some systems for large volume (order of several TB).
The complexity of sync mode in this case would be on the CO and on the SP, identifying the operation is just a retry and that the previous operation should be continued and not aborted. Take for example the complexity of such a request on SP as noted by @Jgoldick .

[jdef]

gRPC protocol also supports timeouts on both the client and server end https://grpc.io/docs/guides/wire.html

[rootfs]

If we decide to move async operation to v1+, we need a path forward to help migration. As for now, I am most concerned with the binary true/false from each operation. This makes async operations to be dedicated APIs.

If we have a meaningful return status to indicate the progress of the operation, we can support async by only adding a polling operation to get the latest status.

[childsb]

A couple ideas on async design:

• Include RPC call back endpoint in initial call
• Immediate synchronous response is only regarding the validity of the call (is call supported, is payload schema correct, etc..)
• Each call includes a UUID such that the response could be to a single shared RPC endpoint. The best design is such that the UUID is a hash of the payload and can be generated instead of tracked.
• The plugin can maintain state or not. The CO could periodically retry which would recover/restart any operation if plugin doesn't keep state.

Pros:

• Simplified plugin architecture. The sync model appears simple on the surface-- you execute the operation in the same thread as the call and quickly return a response-- but the proper design runs the operation on its own thread so if the RPC interrupts mid stream the operation continues and can be rejoined by another RPC.
• Timeouts are the devil. Many operations are fast with no system load, but slow with system load (cloud provider calls, recursive fs operations, provision, encrypt, deletion/'scrub') and you either sacrifice error response time or function calls failing that would eventually succeed
• Retry period instead of timeouts. Retries can be easily discarded by the plugin if the operation is ongoing.
• More natural way of error and status handling. Async calls return status/error/success when the action completes and not after a call timeout or RPC interrupt. Also any invalid call payload is identified quickly instead of (potentially) conflating it with the actual operation error/status.
• Simplified recovery logic.. With sync model, any ongoing operations that RPC call is terminated must be handled. You are forced to recover at the call termination, or have thread join and cleanup logic. In an async model the operation can continue indefinitely and any redundant requests to perform the operation can be discarded or the operation dies naturally.
• Early (initial) adoption of this model will save future legacy baggage (recovery, timeouts etc..)

Cons:

• bi-directional network traffic (async adds call from plugin to CO).. ((If network routing rules prevent plugin calling back to the CO, the CO can poll a plugin endpoint for status.))
• Failure of callback puts system in odd state. After CO failure, the CO may make a redundant call to the plugin which the plugin should positively respond to as if it had succeeded so the CO can be in proper state.

I wanted to discuss this more and agree on a direction before i do any PR against the current spec.

[oritnm]

@childsb, That's the same notion I was thinking about as well, other than the UUIDs which are based on contnent - the uuid must be unique for the CO and for the SP to uniquely identify the right request/ response on both ends. I think the UUID should include CO id, SP id and a request id between them

[saad-ali]

@jieyu as noted above, while the sync mode seems easier at first look I think it is problematic, a long operation such as deleting a volume may take a long time which will need get timeouted many times unless your timeouts are in the order of mins as may take in some systems for large volume (order of several TB).

The complexity of sync mode in this case would be on the CO and on the SP, identifying the operation is just a retry and that the previous operation should be continued and not aborted. Take for example the complexity of such a request on SP as noted by @Jgoldick .

I spoke with several of you offline about this. And we had a discussion about this during the CSI community meeting on 8/23/2017. We concluded that for 1.0 we will pursue only a synchronous API, because doing so lowers the bar for writing a plugin and the added complexity of a asynchronous calls wouldn't buy much.

First it's important to note that CSI only defines the interface between cluster orchestrators and storage plugins, not between volume plugins and storage control planes. Implementing synchronous calls between CO and SP does not prevent the SP from calling control plane async.

While we all agree that long running operations can be problematic, and that many of the volume operations like attach and format can take a long time to complete, asynchronous calls don't necessarily solve that. In particular, if an operation is taking a long time, weather the API between CO and SP is sync or async, eventually the CO must give up trying regardless of what the SP says (defensive design practice). Moreover, even if it were not going to stop waiting, the CO may crash and lose the state about the operation.

Fundamentally we all want these calls to be handled gracefully in error scenarios. CSI's error handling strategy for many calls (like create volume) for most long running calls is to have the caller (CO) retry. Therefore it strongly suggests that the calls an SP implements should be idempotent. And this is the key -- in order for COs to be able to reliably recover, SPs must implement idempotency. A particular call may be idempotent independent of whether it is synchronous or asynchronous.

Post-1.0 we can reconsider adding asynchronous if there is still a strong need.