HttpClient.DefaultProxy should respect IE proxy changes

[ahdung]

Env: Windows server 2008 R2 SP1

In .netfx, this API work fine, if turn on Fiddler capturing, it capturing.
But in .netcore (3.1+), this API always return HttpNoProxy, no longer respect the IE proxy settings.

How get the IE proxy settings? Please help.

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Env: Windows server 2008 R2 SP1

In .netfx, this API work fine, if turn on Fiddler capturing, it capturing.
But in .netcore (3.1+), this API always return HttpNoProxy, no longer respect the IE proxy settings.

How get the IE proxy settings? Please help.

Author:	ahdung
Assignees:	-
Labels:	area-System.Net, untriaged
Milestone:	-

[ahdung]

I think this is really a mistake:
https://source.dot.net/#System.Net.Requests/System/Net/WebRequest.cs,564

GetSystemWebProxy acquire from HttpClient.DefaultProxy, but the latter one is a settable Property, How can do this? the GetSystemWebProxy API should respect its name meaning.

[karelz]

HttpClient.DefaultProxy uses IE proxy, unless it is overridden by environment variables, or by the app itself (it is intentionally settable): https://github.com/dotnet/runtime/blob/master/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/SystemProxyInfo.Windows.cs

I am curious why that detection does not work on your machine. Are you able to enable logging in your environment, or even better, debug into the proxy initialization?

WebRequest.GetSystemWebProxy() intentionally uses HttpClient.DefaultProxy (see PR dotnet/corefx#41692) and allows application to overdrive/customize system proxy. It is intended for better flexibility and to have common proxy value in the application.
BTW: WebRequest class is obsolete and HttpClient is recommended to be used instead -- see docs

[ahdung]

Thanks for your response.
I'm not talking the HttpClient.DefaultProxy can't be settable, i'm talking WebRequest.GetSystemWebProxy() shouldn't use it, this API should always get the IE proxy settings, like netfx does. Think, if i set DefaultProxy to a custom proxy, how can call it system web proxy? unless the system web proxy not equals the IE proxy in .netcore context, but this means that an API with the same name has completely different meanings and behaviors in netfx and net[core], this is a horrible thing for migrate.

The another problem is the WebRequest.GetSystemWebProxy() is cached, when it gets the IE proxy settings at first time, it never change, but you should know, the IE proxy settings is a dynamic thing. look, in netfx, i use Fiddler capture the web requests, when i start capturing, it capturing, when i stop it, it stop, everything is controllable and expected. But now, if i start program with the capture turned off, it will never be able to capture, conversely, when i start the program with the capture turned on, it will always capture and cannot be stopped unless close program or fiddler. Basically, you did something like this:

// the right way:
DateTime GetNowTime() => DateTime.Now;

// your way
DateTime _now = DateTime.Now;
DateTime GetNowTime() => _now;

Finnally, i don't want to expand discussion of WebRequest and HttpClient, i just care How to get the IE proxy settings in real time? Thank you.

[karelz]

We do not have ability to get current IE settings in .NET Core.

.NET Framework watches for registry changes that signal change in proxy settings (reg key HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections - see AutoWebProxyScriptEngine.ListenForRegistry), then query WinHttp in WinHttpWebProxyBuilder

On .NET Core we read proxy settings only once at first use, by querying WinHttp in WinInetProxyHelper - it is the same function that .NET Framework uses.

I assume, we could implement similar registry change monitoring and refresh proxy info from WinHttp in such case (assuming it was not overridden by app/env variables).
We should check that it is officially supported way from WinHttp. And probably think about impersonation impact (something .NET Framework does).
We would also need to figure out how to update proxy in HttpConnectionPoolManager - currently it is designed to take proxy snapshot at the creation time.

The key remaining question is: How much does it matter? Is it only developer scenario (Fiddler turned on and off during application execution), and how commonly is it needed? Or are there other scenarios that would benefit from this?

[ahdung]

This is really regrettable. Ofcourse not only developer scenario, many programs with network access capabilities usually provide an option called Use IE proxy or Use System proxy, and .net framework has also supported this since 2.0, now you tell me the great .NET can't do this.
Even if we do not discuss whether it can be implemented, a Property(DefaultProxy) that can be arbitrarily sets cannot be called system proxy, you really did wrong on this thing, why don't you just obselete this API?
Apart from being alive, many things are not much matter, but about experience.

[karelz]

@ahdung I am sorry that you ran into this the hard way. Yes, it is some incompatibility with .NET Framework, which hit you by surprise. We wish we could be 100% compatible, but you can probably imagine that it is impossible task and we have to pick our battles and choose where to invest.

To answer your concerns:

The obsoletion of WebRequest didn't make it into 5.0 (which is sad), but is at least in 6.0 (master) branch, with some follow ups needed: #41759

Regarding DefaultProxy -- it is up to the application, or the environment to override system proxy - aka lie to itself, or bring even better system proxy detection or other proxy detection than .NET Core does.
If there are real-world cases, where the overriding causes problems, we'd love to hear about them. Let's not dismiss the API usefulness just on principle.

To make further progress, there are several separate "issues" raised here:

1. WebRequest.GetSystemWebProxy can be technically overridden by application/environment, which may be unexpected.
   • Open question: Does it cause real-world problems to applications?
2. Inability to get true/unmodified system (IE) proxy settings in .NET Core.
   • Question: How common is this scenario? Is it something that could be solved in external nuget package potentially with better detection on each OS? (at least as temporary workaround)
3. DefaultProxy does not react to changes to OS settings after it was initialized.
   • Question: How important is this scenario? Does it impact only dev scenarios, or are dynamic proxy changes also real-world scenarios for end users (non-developers)?
   • Note: This is IMO the most important question as the answer fundamentally impacts our design and assumptions we made in our new HTTP stack.

@ahdung did I miss any angle? From the list, which of the problems do you think is worth solving in .NET Core?

[ahdung]

@karelz My English is not good, I try to understand what you mean, please point out if there is any misunderstanding.

Regarding WebRequest, i hope don't obsolete it, although it has some issues now. i believe that many old apps still dependent it, this is absolutely a major breaking change, please be cautious.

Regarding DefaultProxy, it can be changed, that is not a problem in my opinion, like WebRequest.DefaultWebProxy, it not called system something. but it is precisely because it can be changed so i think should not be let GetSystemWebProxy use it. DefaultProxy can use GetSystemWebProxy on init, but GetSystemWebProxy shouldn't use DefaultProxy never.

Regarding "issues":

1. The IE proxy is inherently can be modified, it is a dynamic thing, everyone knows that, it's not unexpectedly.
2. I hope you sir support this need, even if you don't want use the old GetSystemWebProxy, maybe you can provide a new API called GetIEProxy something, otherwise the developers can only access registry for help self. use nuget is ok for me.
3. I hope DefaultProxy like WebRequest.DefaultWebProxy, if no set, it always follow the system proxy changing. Allowing end users to select proxy mode and set up his custom proxy has always been a feature pursued by network software providers, this is a about user's freedom thing. Think, a user sets his global proxy(IE proxy), and all his network software are work follow this, all thing are perfect he feel, but, there is a software which is developed by .net[core] that does not follow it, and can only set it separately, then he will hate this software. As we know, .net[core] not only about web, the Client-end softwares should also be taken care of.

[karelz]

My English is not good, I try to understand what you mean, please point out if there is any misunderstanding.

Absolutely. Happy to help. Let me know if some of my explanations are not clear.

Regarding WebRequest, i hope don't obsolete it, although it has some issues now. i believe that many old apps still dependent it, this is absolutely a major breaking change, please be cautious.

It is already obsoleted. Old apps can use it. They will not get full functionality or high performance though.

i think should not be let GetSystemWebProxy use it. ... GetSystemWebProxy shouldn't use DefaultProxy never.

I understand what you're saying. Given that GetSystemWebProxy is obsoleted API, we will keep it as it is.

Regarding your answers to "issues" 1-3: I am looking for real world scenarios. Cases you have hit where it matters. How important are those scenarios / cases and how common are they?
I agree with you that problems can happen. However, rarely - IE proxy changes are rare. Therefore, I suggest to keep it as is and focus on more important bugs, features. Based on the information so far, it does not seem like common scenario that would hit too many users.
Does that make sense?

[karelz]

Triage: We are ok to take a change with monitoring registry for IE proxy change and try to apply it. It is low pri for us though.
It may impact binary size for ILLinker scenario ...