Streams not gracefully ending during high usage

[tegefaulkes]

Specification

While working on #14 we discovered that during tests with a large amount of streams and data being sent concurrently. Readable streams would fail to end. So far as we could tell, random streams would fail to process a message from streamRecv that had fin = true. In these cases other streams would send a fin frame, end and clean up on both sides, all after the problematic stream should've done the same.

Since then we found a stop-gap solution where we send a single null byte with the finish message. Given that, we're reasonably certain that the problem is related to 0-length messages not being sent or processed event though fin was set to true.

Normally a 0-length message is not sent but in the case of a finish message they should be. This functionality was added in this commit cloudflare/quiche@cc98af0.

Under normal conditions with low load. Sending fin frames with 0-length messages works fine. The problem only happens under higher loads during testing and even then, somewhat randomly. It's been hard to narrow down the specific conditions that cause the problem

To address this problem multiple things need to be done.

1. Create a pure rust example that can reproduce the problem. This will be useful for the upstream issue
2. Investigate the quiche source code for the problem. based on what I understand is happening. The problem is likely located on the receiving side with marking the stream as readable after it has received the 0-length finish frame. But only during message loads.
3. Post an upstream issue with the rust example and a possible solution. Sooner the better so they can address the problem and fix it ASAP.

For now we're using a work-around where we add data to the fin frame message. In this case a single null byte. This should mark the stream as readable regardless of an potential issues with 0-length fin frames.

Additional context

• Related Test connection multiplexing across multiple servers and clients #14
• cloudflare/quiche@cc98af0

Tasks

• 1. Create a pure rust example that can reproduce the problem. This will be useful for the upstream issue
• 2. Investigate the quiche source code for the problem. based on what I understand is happening. The problem is likely located on the receiving side with marking the stream as readable after it has received the 0-length finish frame. But only during message loads.
• 3. Post an upstream issue with the rust example and a possible solution. Sooner the better so they can address the problem and fix it ASAP.

[tegefaulkes]

Cool, progress. I have modified the server and client example and simplified them somewhat.

The server does 3 things.

1. Closes the sending side immediately, we only want to test a single direction.
2. Reads data from the streams and drops it, we don't care about contents.
3. Keeps a set tracking active streams. When a ID is read it is added to the set, when a fin frame is received it is removed. Every time we remove a stream from the set we print the number remaining.

The client does 2 things.

1. Reads data from the streams and drops them.
2. Creates X amount of streams and sends Y amount of messages per stream before ending with a fin message. X and Y are configurable to fine tune load.

What I'm trying to observe with this test is.

1. Do all of the streams finish on the server side? This is indicated by the output showing 0 streams left after streams have finished.
2. Can I affect the amount of streams that fail to close by adjusting the amount of streams and messages?

What I finding is,

1. The threshold for triggering the problem seems to be about 10000 streams and 200 messages ea. This causes 20-50% of the streams to not finish. More often then not it's just under 50%.
2. Increasing the load doesn't seem to affect the amount of failures. Hitting a threshold of load seems to be the main factor here.

I think at this stage I have enough evidence to post an upstream issue now. But first I'll confirm no other errors or factors are involved.

[tegefaulkes]

I changed the client code slightly and I can't trigger the problem anymore. The original failures may have been a problem with my example code?

So far there are no failures with 15000 streams at 1000 messages each. Back to the drawing board.

[CMCDragonkai]

What did you change?

[tegefaulkes]

It was a slight change to the logic for tracking the number of messages sent and when to send the fin frame.

[tegefaulkes]

Forcing congestion control by setting set_initial_max_stream_data_bidi_local and set_initial_max_stream_data_bidi_remote to a low value doesn't cause it.

[CMCDragonkai]

Can you roll back your change and replicate it?