Some commands can take longer in sync execution time than the MRP timeout

[bzbarsky-apple]

Problem

Some commands take longer than the MRP timeout to execute. Their execution then blocks standalone acks from being sent, and the sender keeps resending.

We have run into this with AddNOC/UpdateNOC (due to the crypto bits they have to do).

I believe @bluebin14 has also run into it for some commands that touched storage when storage was being (due to a bug) very slow. But in general, we don't have a good idea of how long sync storage might take.

Proposed Solution

For commands that involve known slow crypto, we should probably just add manual FlushAcks() calls.

For others, not clear.

[bzbarsky-apple]

Also, the newly added IM timeout of 2 seconds + MRP bits is probably too low for these too... @mrjerryjohns @erjiaqing

[tcarmelveilleux]

@turon

[erjiaqing]

Also, the newly added IM timeout of 2 seconds + MRP bits is probably too low for these too... @mrjerryjohns @erjiaqing

The caller can still set the timeout for each command they send.

The 2s timeout should cover 99% cases. Since only cryptographic bits and some async commands are using longer time.

[bzbarsky-apple]

The caller can still set the timeout for each command they send.

Yes, but it's not trivial. And in fact, the 2s change broke commissioning... Anyway, that's a separate thing from what we do about MRP for these commands.

[bluebin14]

Although the storage fragmentation bug has been fixed, back then only an MRP of 5s+2s on both client and server could make it reliably past commissioning (changing it on one side had no effect). Also, RemoveFabric was just as slow as AddNOC/UpdateNOC because it has to do a lot of cleanup.

[bzbarsky-apple]

This is going to somewhat change what's happening on the wire (not at the protocol level, but in terms of actual messages sent), when fixed...

[woody-apple]

SDK Review: Given this will make the test harness more resilient to a variety of DUTs, we believe this is appropriate for the SVE branch.