NetworkInformationException: Permission denied (.NET 8 + Docker +GCloud)

[christosk92]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Identical code (below) that works fine in .NET 7, fails with the following error in .NET 8:

Unhandled exception. System.Net.NetworkInformation.NetworkInformationException (13): Permission denied
2023-11-16 02:04:53.524 JST
   at System.Net.NetworkInformation.NetworkChange.CreateSocket()
2023-11-16 02:04:53.524 JST
   at System.Net.NetworkInformation.NetworkChange.add_NetworkAddressChanged(NetworkAddressChangedEventHandler value)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpConnectionPoolManager.StartMonitoringNetworkChanges()
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpConnectionPool.HandleAltSvc(IEnumerable`1 altSvcHeaderValues, Nullable`1 responseAge)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpConnectionPool.ProcessAltSvc(HttpResponseMessage response)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)

Trimmed and adjusted code (same result for ANY https call):

var httpClient = new HttpClient();
_ = await httpClient.GetAsync("https://www.google.com");

The peculiarity of this issue lies in the fact that the code executes fine on a local environment. But the moment the container is spun up in a google cloud run instance, it fails.

Another weird thing is that raw TCP connections work fine, we have a database connection that goes directly over TCP, rather than traditional HTTP(s).

I have spent hours debugging this issue, and sadly we have no other choice but to revert back to .NET 7 for the time being.

Expected Behavior

A Proper HTTPs connection

Steps To Reproduce

No response

Exceptions (if any)

NetworkInformationException

.NET Version

.NET 8

Anything else?

No response

[martincostello]

This issue looks like it needs to be transferred to dotnet/runtime.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Identical code (below) that works fine in .NET 7, fails with the following error in .NET 8:

Unhandled exception. System.Net.NetworkInformation.NetworkInformationException (13): Permission denied
2023-11-16 02:04:53.524 JST
   at System.Net.NetworkInformation.NetworkChange.CreateSocket()
2023-11-16 02:04:53.524 JST
   at System.Net.NetworkInformation.NetworkChange.add_NetworkAddressChanged(NetworkAddressChangedEventHandler value)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpConnectionPoolManager.StartMonitoringNetworkChanges()
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpConnectionPool.HandleAltSvc(IEnumerable`1 altSvcHeaderValues, Nullable`1 responseAge)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpConnectionPool.ProcessAltSvc(HttpResponseMessage response)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)

Trimmed and adjusted code (same result for ANY https call):

var httpClient = new HttpClient();
_ = await httpClient.GetAsync("https://www.google.com");

The peculiarity of this issue lies in the fact that the code executes fine on a local environment. But the moment the container is spun up in a google cloud run instance, it fails.

Another weird thing is that raw TCP connections work fine, we have a database connection that goes directly over TCP, rather than traditional HTTP(s).

I have spent hours debugging this issue, and sadly we have no other choice but to revert back to .NET 7 for the time being.

Expected Behavior

A Proper HTTPs connection

Steps To Reproduce

No response

Exceptions (if any)

NetworkInformationException

.NET Version

.NET 8

Anything else?

No response

Author:	christosk92
Assignees:	-
Labels:	area-System.Net, untriaged
Milestone:	-

[gfoidl]

the moment the container is spun up

Can you share which image the container is based?
Is it a non-root image?
Can you try a container that runs as root?

[wfurt]

That seems to come from HTTP/3 processing. What do you use as request version @christosk92?
I think the default is enabled in in .NET 8 but there should be ways how to suppress that.

https://learn.microsoft.com/en-us/dotnet/core/extensions/httpclient-http3

Now, if my theory is right, you should see same behavior with .NET 7 if you enable HTTP/3.
The NetworkChange.CreateSocket will attempt to create Nertlink socket and it seems like that is not permitted in your container. (you can create simple app to verify that)
Perhaps something HttpClient can handle better. I feel we should not fail if there is path forward with HTTP/1.1 or H/2.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Identical code (below) that works fine in .NET 7, fails with the following error in .NET 8:

Unhandled exception. System.Net.NetworkInformation.NetworkInformationException (13): Permission denied
2023-11-16 02:04:53.524 JST
   at System.Net.NetworkInformation.NetworkChange.CreateSocket()
2023-11-16 02:04:53.524 JST
   at System.Net.NetworkInformation.NetworkChange.add_NetworkAddressChanged(NetworkAddressChangedEventHandler value)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpConnectionPoolManager.StartMonitoringNetworkChanges()
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpConnectionPool.HandleAltSvc(IEnumerable`1 altSvcHeaderValues, Nullable`1 responseAge)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpConnectionPool.ProcessAltSvc(HttpResponseMessage response)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2023-11-16 02:04:53.524 JST
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)

Trimmed and adjusted code (same result for ANY https call):

var httpClient = new HttpClient();
_ = await httpClient.GetAsync("https://www.google.com");

The peculiarity of this issue lies in the fact that the code executes fine on a local environment. But the moment the container is spun up in a google cloud run instance, it fails.

Another weird thing is that raw TCP connections work fine, we have a database connection that goes directly over TCP, rather than traditional HTTP(s).

I have spent hours debugging this issue, and sadly we have no other choice but to revert back to .NET 7 for the time being.

Expected Behavior

A Proper HTTPs connection

Steps To Reproduce

No response

Exceptions (if any)

NetworkInformationException

.NET Version

.NET 8

Anything else?

No response

Author:	christosk92
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[ManickaP]

This is a general problem for Alt-Svc handling, the spec says you need to clear them on network change RFC 7838:

By default, cached alternative services will be cleared when the
client detects a network change. Alternative services that are
intended to be longer lived (such as those that are not specific to
the client access network) can carry the "persist" parameter with a
value "1" as a hint that the service is potentially useful beyond a
network configuration change.

We just happen to use Alt-Svc only for H/3. So disabling it is a workaround.

I agree with Tomas, we should fix this, at least catch the exception and continue business as usual. If we're not able to detect network changes due to access rights limitations we won't.

[rzikm]

As a workaround, setting DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_HTTP3SUPPORT env var to false should work.

[ManickaP]

Or programmatically with AppContext switch: "System.Net.SocketsHttpHandler.Http3Support".