HttpUpgradeStream doesn't have overloads for Memory<byte>

[buybackoff]

HttpUpgradeStream was omitted from #2333, while it's request and response streams do have those methods. Found it while investigating https://github.com/dotnet/corefx/issues/29951.

Also cannot understand yet if the request and response streams use non-allocating ValueTask from https://github.com/dotnet/corefx/issues/27445?

So far all performance benefits from 2.1 are not available for WebSockets via Microsoft.AspNetCore.WebSockets middleware.

[davidfowl]

We’ll need to do this in 2.2

[davidfowl]

@muratg is this something we can patch in 2.1.1 as well? There's no public API change here fortunately.

[buybackoff]

@davidfowl Finally found your commit dotnet/corefx@8958ef9 to understand if HttpUpgradeStream is the only blocker. And indeed it looks like a couple of #ifdefs should unlock the new IValueTaskSource feature.

[muratg]

@davidfowl I think we can. Can you send a bar check?

[halter73]

Also cannot understand yet if the request and response streams use non-allocating ValueTask from dotnet/corefx#27445?

For reads, we use the non-allocating ValueTask since it calls through to MessageBody.ReadAsync which awaits a PipeReader which in turn uses ValueTask<TResult> all the way through and saves an allocation in the event read data is already available at the time of the call.

For writes, it might look like we allocate a task since we call through the Task-returning IHttpResponseControl.WriteAsync method, but once again it doesn't allocate if the IO completes immediately (due to being able to fit in the PipeWriter buffer) due to this trick where we directly return Task.CompletedTask.

[buybackoff]

@halter73 yet on a synthetic test I see a lot allocations due to async machinery, and Pipe.FlushAsync is always on the path. Another huge source of allocations is ThreadPool, but options.ApplicationSchedulingMode = SchedulingMode.Inline; helps a lot in both allocations and throughput.

This picture below is for approximately 1 million messages sent from a single client: I fire 10 WS.SendAsync without awaiting, then await those 10 ValueTasks, then call WS.ReadAsync without awaiting and then await all 10. (the performance scales c.7 times from 1 task to 10 "concurrent" tasks). I could get c.86k messages per seconds in this benchmark with c.60% load of a core (on a 12 core machine, 5% of total). But adding 11th task throughput suddenly drops by 2+x . (A client build from ClientWebSocket on .NET 2.1. also has a similar picture - only async machinery allocates.)

Is it possible somehow to dedicate N threads for Send/Receive and do not use async machinery (e.g. like NetMQ does)? Or somehow to avoid those noise allocations? Or I should choose something more specialized for this? (BTW, @davidfowl's PlatformLevelTechempower has exactly the same numbers +-jitter)

P.S. Haven't measured latency percentiles yet, but throughput is definitely enough my current task. I just love the middleware pipeline vs. manual sockets on a TCP port.

[halter73]

@buybackoff When IO is actually async (because read data is not yet available or the response buffers are full), seeing async allocations is completely expected. Since we don't want to block a thread, we have to move the continuation data off the stack and onto the heap.

[buybackoff]

Interestingly, putting the following code at the end of the while(true) receiver loop reduces total memory consumption very significantly (to 1.5M, otherwise it grows to 200-300M) and avoids GC1 completely. Typical forced GC0 time is c.250 microsecs, with once per approx. 1M messages increasing to around c.1 millisec. Found it quite accidentally, but that is exactly what I needed - to understand that infrastructure adds max 1 millisec per 1 million messages. Without forced mode memory is increasing rapidly and GC1 causes delay in mid-to-upper single digit milliseconds. (Running with SustainedLowLatency)

if (count % 50000 == 0)
{
    var sw = new Stopwatch();
    sw.Start();
    GC.Collect(0, GCCollectionMode.Forced, false);
    sw.Stop();
    var micros = (sw.ElapsedTicks * 1000000) / Stopwatch.Frequency;
    if (micros > maxElapsed & count > 1000000)
    {
        maxElapsed = micros;
    }
    Console.WriteLine("GC time: " + micros + " ; max: " + maxElapsed);
    Console.WriteLine(GC.CollectionCount(0) + " | " + GC.CollectionCount(1) + " | " +
                      GC.CollectionCount(2));
    Console.WriteLine(GC.GetTotalMemory(false));
}

Docs about induced GC say:

A collection is performed as soon as possible. The Collect(0, Forced, False) method requests a background collection, but this is not guaranteed; depending on the circumstances, a blocking collection may still be performed. If a background collection is already in progress, the method returns immediately.

So hopefully not all threads stop for 250-1000 micros at once and each paused thread just doesn't initiate an AsyncRead while GC0-ing, giving other threads opportunity to take pending reads. Do you think this is the case?

---

Offtop, but:

Since we don't want to block a thread, we have to move the continuation data off the stack and onto the heap.

This exercise made crystal clear why Rust needed Pin<Self> to implement async/await. Basically Rust does forced GC0 per every call. Given that async/await is one of the main and powerful features of .NET it's interesting if garbage from async machinery could be put into a special "async object heap" (there is already LOH) and opposite to LOH, AOH could be cleared when every async state machine finishes...