Reduce the impact of blocking DNS calls on Unix

[davidfowl]

After helping a customer look at a thread pool starvation case on linux on .NET Core 3.1 I ended up here. After doing some research and with some discussion on Twitter, it turns out that getaddrinfo_a uses an internal thread pool and blocks on getaddrinfo and isn't doing any async IO. This change is an improvement over what we had before because our threadpool doesn't grow but I'm not sure this change is a net positive in the long run. The thread pool limits are controlled by compile time constants in glibc (essentially, another library is doing async over sync for us on a less controllable threadpool...).

I wonder if we're better off controlling this blocking code and maybe it should be possible to turn this off with a configuration switch.

The other improvement I was thinking about was only allowing one pending request to a specific host name concurrently. That would improve situations where DNS is slow and new blocking calls are issued for the same host name (which is the case the customer ran into) on thread pool threads.

cc @geoffkizer @stephentoub @scalablecory

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

After helping a customer look at a thread pool starvation case on linux on .NET Core 3.1 I ended up here. After doing some research and with some discussion on Twitter, it turns out that getaddrinfo_a uses an internal thread pool and blocks on getaddrinfo and isn't doing any async IO. This change is an improvement over what we had before because our threadpool doesn't grow but I'm not sure this change is a net positive in the long run. The thread pool limits are controlled by compile time constants in glibc (essentially, another library is doing async over sync for us on a less controllable threadpool...).

I wonder if we're better off controlling this blocking code and maybe it should be possible to turn this off with a configuration switch.

The other improvement I was thinking about was only allowing one pending request to a specific host name concurrently. That would improve situations where DNS is slow and new blocking calls are issued for the same host name (which is the case the customer ran into) on thread pool threads.

cc @geoffkizer @stephentoub @scalablecory

Author:	davidfowl
Assignees:	-
Labels:	area-System.Net, area-System.Threading, untriaged
Milestone:	-

[davidfowl]

Also, most other platforms have a similar issue here but usually offer the "built in OS version" and another version of the API. See node's excellent documentation here:

https://nodejs.org/docs/latest/api/dns.html#dns_implementation_considerations

[stephentoub]

After helping a customer look at a thread pool starvation case on linux on .NET Core 3.1 I ended up here.

Just to be clear, that implementation hasn't shipped yet in any supported .NET release; it was only merged in January for .NET 6.

The other improvement I was thinking about was only allowing one pending request to a specific host name concurrently. That would improve situations where DNS is slow and new blocking calls are issued for the same host name (which is the case the customer ran into) on thread pool threads.

This is largely orthogonal. Regardless of how it's implemented, a "coalesce all pending calls for a given address into one" can be added, layered on top of the underlying operation via a transient address->task cache (that's possible for pretty much any kind of async operation). It is more impactful, though, the more resources are consumed by a pending operation, and if that includes a blocked thread, that's relatively large.

This change is an improvement over what we had before because our threadpool doesn't grow but I'm not sure this change is a net positive in the long run.

I agree, though if the implementation is exactly as you describe, I'm not even sure it's a worthwhile improvement over what we had before. Such an implementation has two main benefits over using the same pool:

1. If the caller synchronously blocks on the async operation, this change eliminates the dependency within the same resource pool, reducing the chances of stalls.
2. It effectively changes prioritization by giving this set of operations a dedicated set of processors rather than being prioritized against all other works item in the pool.

But it also has cons over using the same pool, namely that those other-pool threads end up competing for resources with the main thread pool, which can't use or manage or scale them. It also further bifurcates our implementations, adding some non-trivial interop code that only works on Linux; on macOS, for example, we still just do our own async-over-sync. Interestingly, there's a follow-up PR in #47036 to add cancellation to that Linux code, but that PR calls out that all it supports is removing the pending entry from a queue, and once it's in flight, it can't be canceled: that fits with the implementation you describe. And if that's all it's doing, enabling that same form of cancellation with our own async-over-sync should be close to a one-line change (passing the cancellation token as an argument to the task:

runtime/src/libraries/System.Net.NameResolution/src/System/Net/Dns.cs

Line 635 in e5c7b03

Task.Factory.StartNew(func!, arg, CancellationToken.None, TaskCreationOptions.DenyChildAttach, TaskScheduler.Default);

).

If we rule out caring about callers doing sync-over-async with the resulting task (I'm not saying we can discard that, just that doing so is problematic code, anyway), then I'd expect just reverting back to what we had previously would be a better choice; in typical usage, it'll effectively become as if the call was just made sync, with the pool thread plucking the item it just queued out of its local queue. (And then consider the aforementioned coalescing, which could help in some circumstances but add overhead in others.)

cc: @gfoidl

[gfoidl]

Oh, that's interesting and makes #34633 less useful, as pointed out.
(It seems to be a bad glibc-API with room for improvement, also in that docs...)

A different way to mitigate this is to have a look at https://c-ares.haxx.se (pointed to from the twitter-discussion mentioned above).
I just had a quick look at license (MIT-based) and docs, and it looks like c-ares provides all the functionality we need (incl. cancellation). Additionally it's not glibc-based, so async DNS resolution could be used on more platforms.

(There are other libs listed but none of them seems fine to use -- either license or implementation).

If this is an option, I can try to make a prototype to see how things go. So the steps would be

1. revert NameResolutionPal.Unix enabled async name resolution #34633 and close the PR that would add cancellation
2. make the prototype that uses c-ares
3. productize the c-ares wrapper for libraries

[davidfowl]

@stephentoub

Just to be clear, that implementation hasn't shipped yet in any supported .NET release; it was only merged in January for .NET 6.

Yep, I'm aware.

This is largely orthogonal. Regardless of how it's implemented, a "coalesce all pending calls for a given address into one" can be added, layered on top of the underlying operation via a transient address->task cache (that's possible for pretty much any kind of async operation). It is more impactful, though, the more resources are consumed by a pending operation, and if that includes a blocked thread, that's relatively large.

Right. It is orthogonal but I think it would reduce the chance of starvation caused.

I agree, though if the implementation is exactly as you describe, I'm not even sure it's a worthwhile improvement over what we had before.

👍🏾 I think we should revert.

[davidfowl]

@gfoidl

A different way to mitigate this is to have a look at https://c-ares.haxx.se (pointed to from the twitter-discussion mentioned above).
I just had a quick look at license (MIT-based) and docs, and it looks like c-ares provides all the functionality we need (incl. cancellation). Additionally it's not glibc-based, so async DNS resolution could be used on more platforms.

Yes I think this might be worth looking at (it also has wide platform support). This is what nodejs uses under the covers for dns.resolve but dns.lookup uses getaddrinfo. I dug up the history of problems with the async over sync model in nodejs and it caused lots of problems which lead them to implement dns.resolve (though they are some subtle differences).

The other option is a fully managed version of DNS resolution. It would also help us improve our UDP stack. This is the approach that golang and mono took (cc @migueldeicaza).

I think no matter what we do, we'd need a way to "use OS defaults" in case people configure custom DNS servers and want their .NET apps to automagically use them.

[stephentoub]

This is the approach that golang and mono took

#25793

[davidfowl]

2 birds with one stone then! #19443

[geoffkizer]

I think that if we want to switch implementations of our DNS client, then a full-managed implementation is probably the best option -- as opposed to relying on a 3rd party native library. We could use the mono code as the basis for this (though I haven't looked at it and have no idea how complete/performant it is).

There are some other DNS-related features we could enable if we did that. One is retrieving TTL info. Another is discovering HTTPSVC records, see #43643.

[wfurt]

I think the biggest difficulty would be match OS behavior. e.g. I feel it would be be strange if OS resolution gives different answers than .NET.
DNS it self is not that hard IMHO. But on Unix name resolution can be configured by cascade of nss modules. There are additional directives in /etc/resolv.conf besides just name servers. On some systems, systemd can synthesize records. On macOS resolution can come form mDNS aka multicast.
Perhaps project for runtime labs?

[geoffkizer]

I think the biggest difficulty would be match OS behavior. e.g. I feel it would be be strange if OS resolution gives different answers than .NET.

Yeah, I agree. Golang lets you choose -- you can use OS resolution or you can use their custom name resolution. If we implemented our own resolver we would probably need to do the same, which isn't ideal.

Perhaps project for runtime labs?

Yeah, this would be a great project for runtime labs.