Sharing NuGet.Client libraries with Visual Studio components

[nkolev92]

NuGet.Client today

The NuGet tooling is multi-faceted.
The NuGet functionality lives in many 1st party products such as:

• nuget.exe
• dotnet.exe
• VS

While is the NuGet.Client primarily tooling we do have a bunch of library packages that represent things like frameworks, versioning and concepts and models related to PackageReference restore and package reading/installation. , https://github.com/NuGet/NuGet.Client/blob/dev/docs/nuget-sdk.md

NuGet extensibility in VS

Today NuGet defines a set of extensibility APIs in VS.
In particular they should all be documented here: https://docs.microsoft.com/en-us/nuget/visual-studio-extensibility/nuget-api-in-visual-studio.
For the definition in code see https://github.com/NuGet/NuGet.Client/tree/dev/src/NuGet.Clients/NuGet.VisualStudio/Extensibility

This allows components to primarily manipulate and understand packages installed to a certain project.

Limitations

The NuGet IVs APIs are scoped to the most common customer scenarios. The focus is package manipulation. If one needs to deal with an actual compile reference, they'd be expected to do so through after tasks such as RAR run.

Challenging scenarios in the past

Walking the line between being a tool and an SDK is a very challenging endeavor.
One requires requires a lot forward movement and sometimes breaking the models in order to improve them while the other requires full backward compatibility.

There are a few scenarios where VS components have wanted to make use of NUGet functionality, but that's been difficult to do for various reasons.

Examples are:

• NuGet SDK resolver (final solution ended up being moving and shipping the functionality from the NuGet code).
• Solution explorer dependency tree (similarly the final solution was moving all functionality to the NuGet.Client code)

Alternatives and challenges summarized

When looking into the solution explorer changes which were happening on the project-system side, here was our process.

NuGet has 3 product scenarios and 2 dependency flows.

• MSBuild.exe (Insertions into VS)

• VS (Insertions into VS)

• Dotnet.exe (Insertions into SDK, which are later followed by insertions of that SDK into VS, this is a 2 hop, so it’s kind of annoying)

• There 2 components that depend on NuGet implementations beyond the interop assemblies and those are:

  • MSBuilld for GetReferenceNearestTargetFrameworkTask
  • .NET Core SDK (which has compile time dependencies).

Now we want to add a dependency to NuGet.ProjectModel for a similar reason as the SDK, read the assets file.

The reason why we can do a 2 hop into the SDK and why it works is because the SDK carries the NuGet assemblies with them. Given how infrequent the lock file changes are, they rarely run into functional breaks. The worst thing that can happen is that we end up with multiple versions of the NuGet assemblies loaded in VS. Not ideal, but not the worst issue.
The NuGet.*.dll have a version that usually goes from 5.4.0.0 to 5.4.0.6 by the time we ship, where 6 is really the number of insertions.

Back to the problem at hand & our possible solutions:

• Binding redirects
  • I don’t think this is a viable option at this time. Given that you can use different SDK versions with one VS instance & the problems people with build tools & extensions would run into.
  • NuGet does a dual insertion like with the SDK.
  • This can become challenging to maintain and it puts strain on the project system. You’d need to insert every time NuGet inserts and there’ll be that window where things would not work.
• NuGet does a dual insertion like with the SDK, the project-system ships NuGet.* assemblies in its payload.
  • This would make things work the exact same way as they do in the SDK from a correctness perspective, but this something we’ve avoided doing for a reason. I imagine this immediately gets flagged as a perf regression
• Use an msbuild task instead.
  • Similar to the GetReferenceNearestTargetFrameworkTask approach. So the contract between PS & NuGet is the task instead of a compile time dependency. Not sure how feasible this is from your perspective.
• Use your own parser
  • Easy to get out of date (we don’t really change the assets file that often, not in a way that requires consumers to react at least).
• Alternatively you can use a submodule and compile it into your assembly, but you’d have to be careful with the public types.

[zivkan]

Roughly related to this issue, it would be good if we considered a way to ship experimental, opt-in NuGet features, that we'd like to ship out-of-band.

For VS UI features, we should ship a custom vsix with the experimental UI experience. This issue is specially about VS, so command line is out of scope, but for some background information, experimental command line tooling can be shipped as a dotnet global tool. Restore in VS is the only thing I can think of that we have no theoretical out-of-band possibility for.

For restoring a single project on the command line, using an MSBuild SDK that overwrites the restore task (or overwrites the MSBuild property that the SDK uses to find NuGet.targets) to load restore assemblies from the package allows customers to opt-in to experimental features.

For restoring a solution, I've created this msbuild issue asking to extend the Sdk= syntax to work for solutions: dotnet/msbuild#5832

This will allows experimental restore features to be shipped out of band from the command line, but will continue to use VS's official restore logic if opened in Visual Studio.

The "possible solutions" above start off sounding like options for how NuGet could be changed to support the customer scenario, although the "use your own parser" and "use submodule" doesn't make sense in the context of changes NuGet could change, but can only be a solution for customers who want to use NuGet's APIs.

Anyway, I just wanted to point out that if restore was changed so that it ran as a MSBuild task in VS, rather than as a pre-build VS event, this would enable the MSBuild SDK style out-of-band experimental feature shipping to work on both command line and VS.

If running restore in VS as an msbuild task is not going to be implemented, I still think it would be good to consider other options for how we might be able to enable experimental, out-of-band restore. Other brainstorming ideas that might not be at all feasible:

• NuGet just ships our experimental versions of our 'normal' vsix, which customers can install.
• Split NuGet vsix into separate restore and "other" vsixes, so we can ship experimental restore functionality independently of PM UI, PMC, etc.
• MEF or VS service extensibility point, so a customer can install a custom VSIX and restore logic comes from that, rather than what ships in VS.
• Take restore out of proc with a ServiceHub, and on solution load check for the MSBuildSDK package in Directory.Solution.props, and if it exists, load the out-of-proc servicehub from the MSBuildSDK package rather than the one that ships with VS.

We'll also need to consider telemetry and how we can ensure that customers using experimental features (restore or otherwise) do not contribute to release telemetry analysis.

[zivkan]

related (duplicate?) #6058

[nkolev92]

There's a good discussion in #12001.

cc @kfertitta