Should send_all automatically do wait_send_all_might_not_block after sending the data?

[njsmith]

Currently send_all blocks until the kernel accepts responsibility for the data, and then immediately returns. I'm wondering if it should instead block until the kernel has not only accepted responsibility, but also until the the kernel send buffer has some space in it (i.e., the socket becomes writable again).

Pros:

• It would make TCP_NOTSENT_LOWAT automatically work on MacOS. Currently it only works if you explicitly call wait_send_all_might_not_block, because the TCP_NOTSENT_LOWAT is only applied to select/kqueue type writability checks, not to send writability (discussion here: TCP_NOTSENT_LOWAT umbrella issue #76). This would also let us normalize TCP_NOTSENT_LOWAT semantics across MacOS and Linux (which does apply TCP_NOTSENT_LOWAT limits to send) -- in both cases send_all would wait to return until after the low water mark was reached. It would also open the door to a potential optimization on Linux: when a large buffer is passed to send_all, we could temporarily disable TCP_NOTSENT_LOWAT (so that the kernel accepts it in one big chunk instead of making us dribble it into the send buffer over multiple calls), and then re-enable it, and then wait for writability.

• It would allow us to remove wait_send_all_might_not_block, which adds significant surface area to the Stream API (it would go from 4 methods → 3 methods, 25% smaller, and testing this method in particular adds significant complications).

• It's generally consistent with the idea that send_all blocks until the data is sent -- not only do we hand it off to the kernel, we wait for the kernel to make some progress. (I guess this isn't so much a pro, as a counter to the possible "con" that send_all waiting is surprising.)

• It automatically gives better results for latency-sensitive applications, in particular those where you get better results when waiting as late as possible before committing to what you want to send (e.g. screen-sharing). These apps are why wait_send_all_might_not_block exists, but currently it's not clear they'll actually benefit because this is a bit obscure and you have to explicitly set up your code to use it.

Cons:

• Maybe it will kill throughput on super high bandwidth applications, or something? (Stuff like an echo server benchmark, but possibly also stuff like Antoine's worrying about here. Would want to check this. OTOH maybe it's fine because we'd end up replacing a loop like:

  while True:
      await wait_writable(socket)
      socket.send(...)

  with a loop like:

  while True:
      socket.send(...)
      await wait_writable(socket)

[njsmith]

Note: if we ever do drop wait_send_all_might_not_block from the SendStream interface, then we should update test_windows_pipes.py to add a "clogged pipes" test – right now it doesn't run the clogged tests because they test wait_send_all_might_not_block.