Duplicate ACK to same message id will cause receiver reject the message

[erjiaqing]

Problem

1. Some coding issue results in the code sending two ACKs to one message id.
2. The code might reject the message with payload if duplicate ACK is received.

Proposed Solution

Analysis the protocol and

1. Avoid sending duplicate ACK to the same message
2. Discuess whether to reject the message if duplicate ACK received.

[bzbarsky-apple]

Avoid sending duplicate ACK to the same message

#7935 will mostly do that. We could still run into it in the following way, with events listed in the order they happen, with time passing between the steps:

1. A sends a message to B.
2. B receives message from A.
3. B responds with an app-level response with a piggybacked ack (but this message is delayed in transit)
4. A resends the message.
5. B receives the resent message.
6. B responds with a standalone ack (as we always do for duplicates using MRP).
7. A receives the standalone ack.
8. A receives the response from step 3, which got delayed in the network somewhere.

This is a perfectly legitimate sequence of events, albeit one that is not going to happen in our CI as things currently stand, because our CI does not test interesting complicated network scenarios so far... I keep hoping we'll re-enable the cirque tests and use those to test this sort of thing.

Anyway, given the scenario above it seems clear to me that a message with a piggyback ack that is not itself a duplicate has to be processed properly even if the ack is for a message id that we don't think we need an ack for.

@erjiaqing @yufengwangca @turon

[bzbarsky-apple]

Or another scenaraio where we have the same problem:

1. A sends a message to B.
2. B receives message from A.
3. B responds with an app-level response with a piggybacked ack, but this response is lost.
4. A resends the message.
5. B receives the resent message.
6. B responds with a standalone ack (as we always do for duplicates using MRP).
7. A receives the standalone ack.
8. B resends its response.
9. A receives the resent response, which has an app-level message we need to process, and a piggybacked ack for a message id that has already been acked.

This scenario we could test in our existing CI, by just having our loopback-drops-messages stuff set up to strategically drop the right things (the message from step 3 in this case).