Improved proxy support in the NuGet.Clients (ntlm proxy, etc)

[tebeco]

Details about Problem

NuGet product used :

• NuGet.exe
• VS UI
• dotnet.exe

NuGet version : NuGet Version: 4.6.0.4971

dotnet.exe --version : 2.1.300-rc1-008673 [C:\Program Files\dotnet\sdk]

VS version : Vs2017 15.7.2

OS version :

OS Name:                   Microsoft Windows 10 Enterprise
OS Version:                10.0.15063 N/A Build 15063

Worked before? Never probably ?

Detailed repro steps so we can see the same problem

1. Like many corporation, there's a "PAC" script define in the OS (IE/Edge ?) Proxy settings
2. Set HTTP_PROXY to the pac url (like : SET HTTP_PROXY=http://foo/bar/corp_proxyXYW.pac)
3. Try to use Nuget install with something like :

  dotnet tool install fake-cli ^
    --tool-path ./%BUILD_PACKAGES% ^
    --source-feed https://www.myget.org/F/fake-vsts/api/v3/index.json ^
    --version 5.0.0-rc*

WITHOUT anything in the HTTP_PROXY :
C:\Program Files\dotnet\sdk\2.1.300-rc1-008673\NuGet.targets(114,5): error : Unable to load the service index for source https://www.myget.org/F/fake-vsts/api/v3/index.json.

With it :
C:\Program Files\dotnet\sdk\2.1.300-rc1-008673\NuGet.targets(114,5): error : Response status code does not indicate success: 400 (Bad Request)

With the "real" proxy URL ... since i use both internal feed and external feed ... i have to set NO_PROXY ?
If there's no simple pattern for no_proxy ... how do i deal with it
The guys that are paid in corp to design the PAC Script took time to handle all that nicely so it would be fine using it right ?

SET HTTP_PROXY=http://realCorpProxy:1234
SET HTTPS_PROXY=http://realCorpProxy:1234
SET NO_PROXY=*.what;*.the;*.hell;*.i.should;*.not.have;*.todo;*.this

>  dotnet tool install fake-cli --tool-path ./fake-cli --source-feed https://www.myget.org/F/fake-vsts/api/v3/index.json --version 5.0.0-rc*

Output :

>  dotnet tool install fake-cli --tool-path ./fake-cli --source-feed https://www.myget.org/F/fake-vsts/api/v3/index.json --version 5.0.0-rc*
C:\Program Files\dotnet\sdk\2.1.300-rc1-008673\NuGet.targets(114,5): error : Unable to load the service index for source https://api.nuget.org/v3/index.json. [C:\Users\xxxxx\AppData\Local\Temp\idtbo33l.zuc\restore.csproj]
C:\Program Files\dotnet\sdk\2.1.300-rc1-008673\NuGet.targets(114,5): error :   Response status code does not indicate success: 407 (authenticationrequired). [C:\Users\xxxxx\AppData\Local\Temp\idtbo33l.zuc\restore.csproj]



The tool package could not be restored.
Tool 'fake-cli' failed to install. This failure may have been caused by:

* You are attempting to install a preview release and did not use the --version option to specify the version.
* A package by this name was found, but it was not a .NET Core tool.
* The required NuGet feed cannot be accessed, perhaps because of an Internet connection problem.
* You mistyped the name of the tool.

D:\Sources\SGithub\Sgme.NetStandard\sgmarkets-common>cat C:\Users\xxxxx\AppData\Local\Temp\idtbo33l.zuc\restore.csproj
cat: 'C:\Users\xxxxx\AppData\Local\Temp\idtbo33l.zuc\restore.csproj': No such file or directory

I will not set any user/password in an Env variable either because i would have to do so :

• on every developper computer ?
• On the every build agent ?
• If i use badly NO_PROXY it will destroy tool like npm yarn deploy stuff
• Tools are trying to use the PROXY to acces internal url ...

How are we supposed to use dotnet behind an NTLM proxy ?

[zacateras]

@tebeco did you come up with any alternative solution so far ?

[tebeco]

not really
i use a dumb nuget.config because ou cannot </clear> from the CLI (only --add-source)
and created artifactory in the LAN that can pass the proxy

though JFrog need to fix tons of bug
protocol v3 is 30% implemented and they try to guess the route when restoring instead of writhing the response

for example i cannot proxyfy feeds.io because JFrog think they are smart and use registration-semver2 and response indicate response-gz-semver2

[zacateras]

seems like a hack that may do the work, I will check it out

[tebeco]

its very problematic when i clone repository like
dotnet/aspnetcore
azure/azure-signalr
dotnet/corefx

and i attempt to restore/build il order to navigate in the code

for exemple, dotnet/arcade proxy support was not that fun ~1year ago when i had to debug http proxy around 2.1.4)

another issue is that sources are split between nuget.config and several RestoreSources in sources.props

finally there's like 4 to 8 differents feed so i have to create them at least once so at the beginning you have to create them one by one
and latter ... you have to check what is the name of the source to replace with what

[delixfe]

Any updates on that issue? Just run into the 407 due to our NTLM authenticated proxy and since we migrated to Always On VPN, I cannot even circumvent the proxy...

[michal-ciechan]

Getting a 403 because

dotnet restore doesn't respect NO_PROXY
nuget restore does....

[mungojam]

some of you might be running into a different issue that is actually easy to workaround. If you have a fixed proxy address that you can point at (which you can sometimes work out from the pac script), then you can tell Nuget about it and it will happily deal with NTLM I believe. It needs to be configured in your central nuget.config under %AppData%\NuGet as it won't work in any local nuget.config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
<config>
    <add key="http_proxy" value="http://your-proxy-server:proxy-port" />
</config>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
  </packageSources>
</configuration>

[michal-ciechan]

My issue is that we need to use proxy for a single source. and not for others.

We have HTTP_PROXY set so that everything picks it up correctly.

We also have NO_PROXY set for various domains/IP ranges that cannot go through the proxy.

Global NuGet.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <!-- This needs to go through Proxy -->
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />

    <!-- This cannot go through Proxy. Has exclusion in NO_PRXY -->
    <add key="TeamCityInternal" value="http://nuget.internal/" />
  </packageSources>
</configuration>

And then we have

HTTP_PROXY=http://somproxy
NO_PROXY=.internal

This works perfectly fine when running nuget restore using nuget.exe versions v5.9.1 or v5.10.0-preview2

It doesn't work when running dotnet restore using .NET SDK 5.0.300 or .NET SDK 6.0.100-preview.4.

Using dotnet restore get a 403 forbidden error. (Proxy uses standard NTLM)

[Martinn2]

In my case I had to set http_proxy in global nuget not local one as @mungojam said. But why is it not possible to set it in local nuget.config ? It is very confusing because nuget config http_proxy returns correct value even if it is in local file.

[tebeco]

the current nuget config is flawed the moment you have a source on public internet like nuget.org that needs a proxy and an on premise solution like sonarype/artifactory and you should skip proxy

or an in cloud private feed which require another proxy route

nuget team should move the "proxy" settings INSIDE the feed idea

1 feed => 1 or 0 proxy
and default to local/global proxy after

until that's done i strongly suggest you don't rely on proxy nuget client side and you ask you IT to setup a proxy feed to do that heavy lifting to reach outside feed with the configuration properly doing the proxy stuff per feed

you could even run a Baget instance locally but the moment you have teammate ... having a nuget feed acting as a proxy is also supposed to help bandwith / latency assuming caching policy is done properly