Updating to .Net Framework / .Net Core 8 from .Net Framework .Net Core 6 seems to come with performance penalties for IO completions

[TimLovellSmith]

This is a somewhat anecdotal description of a performance issue, which I want to post in case anyone can help us 'track it down' based on pattern recognition.

Description

While migrating our app to .Net 8, AND doing other changes, we saw some increases in CPU performance costs, this is probably of magnitude 1-3% CPU overall, that we thought we could try to better track down.

These show where the Delta growths and reductions for us are mostly at:

From the new increase in usage, we can notice the semaphore is now being used in order to gate access to 'GetQueuedCompletionStatus', which seems like it was previously called more often from CompletionPortThreadStart.

I notice other nearby things that are in the new 'growth areas' (in cost) is the 'PortableThreadPool' and 'IOCompletionPoller'.

Also there seems to be an increase in SwapContext, which I guess is switching threads?

Configuration

I think this is Windows Server 2022 OS both before/after. Build may have changed.

Analysis

Looks to me, suspiciously like the main cause of increased CPU cost is something like

• reconfiguring the managed thread pool? or
• a new implementation of reading from IO completion ports?
• updating LIFO semaphore implementation?

When I zoom in on LowLevelLifoSemaphore usage before/after, I see

before: we mostly use 'Wait' and 'WaitForSignal', and its something like 0.1% of total CPU.
after: we mostly use 'Wait' and 'WaitForSignal' but skewed even more towards using 'Wait'. Also now we use 'Release' more than 'WaitForSignal' whereas before it was the other way around. And its something like 0.3% of total CPU.
So, this doesn't seem to explain the entire change in overall CPU cost, but it might be a meaningful part of it.

I see we had a 4x increase in the amount of time in 'SpinWait'. And all of that is from being LowLevelLifoSemaphore.Wait

That sounds possibly a bit like the observations about fairness and unfriendliness to background tasks discussed here, but seen through a different lens?
#103812

In general, our app tends to do a lot of IO processing, and is not usually CPU bound, with many background processes on the machine that occasionally need resources.

I wonder what actually changed to make this difference.
Are there any obvious ways we could get back to spending less CPU?

[TimLovellSmith]

I suppose using Windows Thread Pool instead is an option, no idea if better or worse yet..

[TimLovellSmith]

Possibly related to PR #64834 ?

[TimLovellSmith]

This setting also looks like an interesting one to experiment with.

The comment here seems to run very counter to the "received wisdom" I have, which says you, the author of the IO handling thread, should never hold the thread for long, but you should strive to release it back to the IO completion thread pool in a timely manner.

[I mean, it seems to think that onus should be moved from shoulders of the callback author, at some performance cost.]

        // Continuations of IO completions are dispatched to the ThreadPool from IO completion poller threads. This avoids
        // continuations blocking/stalling the IO completion poller threads. Setting UnsafeInlineIOCompletionCallbacks allows
        // continuations to run directly on the IO completion poller thread, but is inherently unsafe due to the potential for
        // those threads to become stalled due to blocking. Sometimes, setting this config value may yield better latency. The
        // config value is named for consistency with SocketAsyncEngine.Unix.cs.
        private static readonly bool UnsafeInlineIOCompletionCallbacks =
            Environment.GetEnvironmentVariable("DOTNET_SYSTEM_NET_SOCKETS_INLINE_COMPLETIONS") == "1";

[TimLovellSmith]

In fact, the more I think about it, the worse that seems.

If you stick to processing IO on the IO threads, that increases the chances, that by the time you finish processing the IO, more IO has already arrived. So you can keep processing IO without any yielding of the (IO) thread or context switching.

The converse - if you always hand IO off to non-IO threads for processing, you always incur thread switch cost to hand the work over, and you also increase the chance that you block, or spin, an IO thread waiting for the next bit of IO, since the elapsed time before you get back to waiting for IO is smaller.

[TimLovellSmith]

I also wonder why a bit why you should want four IO poller threads per CPU, or to raise their thread priority.

                    // Poller threads are typically expected to be few in number and have to compete for time slices with all
                    // other threads that are scheduled to run. They do only a small amount of work and don't run any user code.
                    // In situations where frequently, a large number of threads are scheduled to run, a scheduled poller thread
                    // may be delayed artificially quite a bit. The poller threads are given higher priority than normal to
                    // mitigate that issue. It's unlikely that these threads would starve a system because in such a situation
                    // IO completions would stop occurring. Since the number of IO pollers is configurable, avoid having too
                    // many poller threads at higher priority.
                    if (IOCompletionPollerCount * 4 < Environment.ProcessorCount)
                    {
                        _thread.Priority = ThreadPriority.AboveNormal;
                    }

Seems like an attempt to optimize for the 'hosed case', where the server is maxed out on CPU and we imagine that we somehow make things better by prioritizing IO.

But that itself, sounds like something that only sometimes does what you want... depending on the direction:

• prioritizing receiving requests: grows your current work backlog. (Except you kind of can't if you're maxed on CPU.) So maybe shouldn't be prioritized?

• prioritizing sending responses: closes/shrinks your current work backlog by finish responding to requests (with better latency?). So maybe is good to prioritize.

[TimLovellSmith]

Whoops, I'm reading that wrong, aren't I.
four IO poller threads per CPU? That'd be
(IOCompletionPollerCount < Environment.ProcessorCount *4)

[TimLovellSmith]

Well, the IOCompletionPollerCount code seems to be changed a bit since the original PR so maybe I shouldn't focus on the old code.

I do still wonder if having a count > 1 is likely going to lead to having lots of competing polling threads on small machines though, which sounds like inefficiency.

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[teo-tsirpanis]

c.c. @kouvel