Send Deadlock from #4169

[nibanks]

Describe the bug

Stress has found (link) a send deadlock after #4169 was merged.

0b 00000098`287fe680 00007ffb`6af8dcc0     ntdll!KiUserExceptionDispatch+0x2e
0c 00000098`287fedb0 00007ffb`6af585b3     msquic!QuicSendFlush+0xa60 [D:\a\msquic\msquic\src\core\send.c @ 1442] 
0d 00000098`287ff3c0 00007ffb`6af7803c     msquic!QuicConnDrainOperations+0x3a3 [D:\a\msquic\msquic\src\core\connection.c @ 7629] 
0e 00000098`287ff430 00007ffb`6af76c3b     msquic!QuicWorkerProcessConnection+0x23c [D:\a\msquic\msquic\src\core\worker.c @ 505] 
0f 00000098`287ff5a0 00007ffb`6afcca75     msquic!QuicWorkerLoop+0x13b [D:\a\msquic\msquic\src\core\worker.c @ 659] 
10 00000098`287ff610 00007ffb`6afcc71f     msquic!CxPlatRunExecutionContexts+0x105 [D:\a\msquic\msquic\src\platform\platform_worker.c @ 395] 
11 00000098`287ff680 00007ffb`a2774cb0     msquic!CxPlatWorkerThread+0x19f [D:\a\msquic\msquic\src\platform\platform_worker.c @ 493] 
12 00000098`287ff860 00007ffb`a445e8ab     kernel32!BaseThreadInitThunk+0x10
13 00000098`287ff890 00000000`00000000     ntdll!RtlUserThreadStart+0x2b

This seems to be because the connection has the send flags to send QUIC_CONN_SEND_FLAG_APPLICATION_CLOSE, but apparently isn't able/allowed to send it right now. This results in the send path just spinning.

Affected OS

• Windows
• Linux
• macOS
• Other (specify below)

Additional OS information

No response

MsQuic version

main

Steps taken to reproduce bug

Run stress (spinquic) tests.

Expected behavior

Pass without issue.

Actual outcome

Generates an assert from the deadlock detection.

Additional details

No response

[nibanks]

After discussing with Radek over IM, I believe the problem is around the packet_builder code that tries to determine if we've build a packet already. It currently does this via checking the Datagram and DatagramLength parameters, but there must be cases where these are still expected to be NULL or 0.

Perhaps we just add a new flag to the packet builder that says "I have already written a close frame at a lower handshake level" and be explicit about it.