NuGet Restore on mixed solution with packages directory override produces inconsistent packages folder

[logiclrd]

Details about Problem

NuGet product used: NuGet.exe
NuGet version (x.x.x.xxx): 5.5.1.6542

If NuGet Restore is performed on a solution where some of the projects are based on packages.config and some are using <PackageReference>, then it effectively performs two separate restore steps: One for the packages.config file into the packages directory in your project, and one for the <PackageReference> elements into the centralized package cache for your user profile. But, if you use -PackageDirectory on the command-line, it redirects both of these restore operations, when, I think, it should only set the path for the first one. The result is that the packages restored for <PackageReference> that were supposed to go into the central folder for the user profile instead get placed into the packages folder in the project. This in turn causes problems because the first phase writes packages to the folder with V2 layout, and the second phase writes them with V3 layout. The result is a packages folder that is essentially damaged because of the mixed formats. I observed NuGet Pack to subsequently fail, because it assumes the packages directory it is looking at will be either V2 or V3 but never both, and because of this assumption, the first folder it sees determines what it expects for all other references -- but my NuGet Pack operation required it to read a reference restored with V2 format, and the first folder it saw was a V3 folder.

Detailed repro steps so we can see the same problem

1. Create a solution with two projects, one of which uses packages.config and the other one <PackageReference>. Ensure that each project references at least one NuGet package.

2. Run NuGet Restore on the solution with -PackageDirectory overriding the location of the packages folder.

After this operation, what I expect you will see is a packages folder where some packages are installed into folders like log4net.2.0.8 while others are installed into folders like fluentassertions/5.0.3.

[donnie-msft]

The experience here could be improved and will take some thought across products (eg, Dotnet.exe restore).

The folder structure is intended to be different between packages.config packages folder, and PackageReference global packages folder, so writing to the same folder may not make sense.

[logiclrd]

It makes sense for there to be different folder structures, but what doesn't make sense to me is for the restore code to start writing into an existing folder with one structure using a different structure.

A thought that occurs to me is, is there ever a situation where it would make sense for someone to redirect a V3-style "global" restore into a directory of their choosing? The -PackageDirectory option exists because of the V2 packages folder being present within the user's workspace, right? E.g. for putting the folder a level higher than the solution file, or what have you. If this is correct, and there isn't really any logical reason to want to put the global packages folder in a nonstandard location on a per-command basis, then perhaps the simplest solution is simply for NuGet Restore to ignore -PackageDirectory within the V3 flow?? Granted, this is speculation from the outside of the project. :-)

[r0bfr]

We have a related issue. To ensure that all nuget packages get downloaded to the same cache for every project in our repository, we redirect downloads using our nuget.config file:

<config>
    <add key="repositoryPath"       value="Output\Packages" />
    <add key="globalPackagesFolder" value="Output\Packages" />
</config>
  

It seems to be the case that packages downloaded using the repositoryPath key (VCXPROJ files) get downloaded to folders that use the V2 layout:
Output\Packages\<packageName>.<packageVersion>\

While packages downloaded using the globalPackagesFolder key (CSPROJ files) get downloaded to folders that use the V3 layout:
Output\Packages\<packageName>\<packageVersion>\

That is, globalPackagesFolder seems to group all versions of the same package under a parent folder named after the package, while repositoryPath does not. As a result, two separate copies of the same package are being downloaded when VCXPROJ and CSPROJ files both make use of them.

I understand that these two mechanisms for downloading NuGet packages each have their own format that gets used, and the separate formats are preserved by default to ensure compatibility, but when parity is desired (i.e. one wants to ensure that all packages.config and all <PackageReference/> downloading share the exact same cache), is there any reason why a new key couldn't be added to the nuget.config schema that tells the V2 codepath to use the same folder structure as the V3 codepath?

Since <PackageReference/> continues to be unsupported for C++, it would be nice to create a way for VCXPROJ and CSPROJ projects to live in harmony in the same repository. The same package used by each of these Project types should only have to be downloaded a single time.

As an additional use case, consider the scenario where (for development purposes), a developer would like to "patch" the contents of a NuGet package that's being used by the projects in their repository (their team also owns development of that package). Rather than having to rebuild new NuGet packages each time a patch is necessary, they would like to simply use a script to overwrite the contents of a folder in the downloaded cache copy of the package. If the V2 and V3 codepaths were sharing the same cache, this would ensure that a single folder needs to be overwritten each time a patch is needed. With things working the way they do now, however, one patch is needed for the V2 folder (created by VCXPROJ projects using the package), and another patch is needed for the V3 folder (created by CSPROJ projects using the package).