RestorePackagesWithLockFile: Nuget.exe generates different lock file than Visual Studio

[codereader]

NuGet Product Used

NuGet.exe, Visual Studio

Product Version

NuGet.exe 6.6.1 + Visual Studio 2022 17.6.3 (NuGet 6.6.0 built-in)

Worked before?

Not that I know of

Impact

I'm unable to use this version

Repro Steps & Context

I have a solution containing mixed C#/C++ projects. The attached repro solution contains one C#/.NET 6 Project referencing a C++/CLR project, which in turn references a plain C++ project. The C# project references a single NuGet Package System.Memory, and is using the Central Package Management feature (not sure if this is relevant). I'm aiming to achieve reproducible nuget restores, therefore I enabled the RestorePackagesWithLockFile setting.

Problem is, the packages.lock.json file is different depending on whether it's generated by nuget.exe or Visual Studio. When enabling the RestoreLockedMode setting, either one, VS or nuget.exe, will run into trouble when hitting a lock file generated by the other.

The attached demo solution contains the packages.lock.json files as generated by VS and has <RestoreLockedMode>true</RestoreLockedMode> activated in the Directory.Packages.props file. When trying to run a nuget restore using the CLI (v6.6.1) it will report that the dependencies of the C++ CLR project have changed:

NU1004: The project reference cppprojecta has changed. Current dependencies: None lock file's dependencies: Common

It boils down to Studio adding the "project" type reference to common to the lock file, whereas the CLI doesn't recognize the ProjectReference:

"net6.0": {
  "System.Memory": {
    "type": "Direct",
    "requested": "[4.5.5, )",
    "resolved": "4.5.5",
    "contentHash": "XIWiDvKPXaTveaB7HVganDlOCRoj03l+jrwNvcge/t8vhGYKvqV+dMv6G4SAX2NoNmN0wZfVPTAlFwZcZvVOUw=="
  },
  "common": {
    "type": "Project"
  },
  "cppprojecta": {
    "type": "Project",
    "dependencies": {
      "Common": "1.0.0"
    }
  }
}

I can twist it around and have nuget.exe generate the lock file, but then Visual Studio will complain about the lock file (sometimes with a more cryptic message about missing frameworks). Either way, one cannot deal with the lock file generated by the other toolchain.

Is there anything I can do to resolve this issue? I need the nuget client to produce the same result as the one in studio, since a nuget.exe restore is one of the first steps happening on the build agent, and if it can't deal with the lock files produced by the developer's VS, this prevents us from using the locked restore mode.

Of course, the actual productive solution is of course far more complex, this is just a boiled-down example.
LockedRestore.zip

Verbose Logs

nuget.exe restore LockedRestore.sln -UseLockFile -LockedMode -ForceEnglishOutput

MSBuild auto-detection: using msbuild version '17.6.3.22601' from 'C:\Program Files\Microsoft Visual Studio\2022\Professional\MSBuild\Current\Bin\amd64'.
WARNING: NU1507: There are 2 package sources defined in your configuration. When using central package management, please map your package sources with package source mapping (https://aka.ms/nuget-package-source-mapping) or specify a single package source. The following sources are defined: nuget.org, Private
NU1004: The project reference cppprojecta has changed. Current dependencies: None lock file's dependencies: Common.The packages lock file is inconsistent with the project dependencies so restore can't be run in locked mode. Disable the RestoreLockedMode MSBuild property or pass an explicit --force-evaluate option to run restore to update the lock file.
Writing assets file to disk. Path: C:\Path\To\ProjectA\obj\project.assets.json
Failed to restore C:\Path\To\ProjectA\ProjectA.csproj (in 262 ms).


Errors in C:\Path\To\ProjectA\ProjectA.csproj
    NU1004: The project reference cppprojecta has changed. Current dependencies: None lock file's dependencies: Common.The packages lock file is inconsistent with the project dependencies so restore can't be run in locked mode. Disable the RestoreLockedMode MSBuild property or pass an explicit --force-evaluate option to run restore to update the lock file.

NuGet Config files used:
    C:\Users\name\AppData\Roaming\NuGet\NuGet.Config
    C:\Program Files (x86)\NuGet\Config\Microsoft.VisualStudio.Offline.config

[nkolev92]

Bug is somewhere here: https://github.com/NuGet/NuGet.Client/blob/69ef354945cf5ec2029b35be024e7b34ec9becbf/src/NuGet.Core/NuGet.Commands/RestoreCommand/Utility/MSBuildRestoreUtility.cs#L520-L568.

Good news is static graph restore works.

Hey @codereader, can you try using msbuild /t:restore instead of nuget.exe?

Nowadays it supports packages.config, so as long as you use that if you have packages.config https://learn.microsoft.com/en-us/nuget/reference/msbuild-targets#restoring-packagereference-and-packagesconfig-projects-with-msbuild it'll be equivalent.

What does work is static graph evaluation restore.

https://learn.microsoft.com/en-us/nuget/reference/msbuild-targets#restoring-with-msbuild-static-graph-evaluation

Can you try that out and let us know how it goes.

[codereader]

Thanks for taking the time to look into this.

msbuild /t:restore results in the same error:

msbuild /t:restore LockedRestore.sln
Microsoft (R) Build Engine version 17.2.1+52cd2da31 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Building the projects in this solution one at a time. To enable parallel build, please add the "-m" switch.
Build started 22.06.2023 06:19:33.
Project "C:\path\to\LockedRestore.sln" on node 1 (Restore target(s)).
ValidateSolutionConfiguration:
  Building solution configuration "Debug|Any CPU".
_GetAllRestoreProjectPathItems:
  Determining projects to restore...
Restore:
  The project C:\path\to\ProjectA\ProjectA.csproj is using CentralPackageVersionManagement, a NuGet preview feature.
C:\path\to\ProjectA\ProjectA.csproj : warning NU1507: There are 4 package sources defined in your configuration. When using central package management, please map your package sources with package source mapping (https://aka.ms/nuget-package-source-mapping) or specify a single package source. [C:\path\to\LockedRestore.sln]
C:\path\to\ProjectA\ProjectA.csproj : error NU1004: The project reference cppprojecta has changed. Current dependencies: None lock file's dependencies: Common.The packages lock file is inconsistent with the project dependencies so restore can't be run in locked mode. Disable the RestoreLockedMode MSBuild property or pass an explicit --force-evaluate option to run restore to update the lock file. [C:\path\to\LockedRestore.sln]
  Writing assets file to disk. Path: C:\path\to\ProjectA\obj\project.assets.json
  Failed to restore C:\path\to\ProjectA\ProjectA.csproj (in 212 ms).

...

Done Building Project "C:\path\to\LockedRestore.sln" (Restore target(s)) -- FAILED.


Build FAILED.

"C:\path\to\LockedRestore.sln" (Restore target) (1) ->
(Restore target) ->
  C:\path\to\ProjectA\ProjectA.csproj : warning NU1507: There are 4 package sources defined in your configuration. When using central package management, please map your package sources with package source mapping (https://aka.ms/nuget-package-source-mapping) or specify a single package source. [C:\path\to\LockedRestore.sln]


"C:\path\to\LockedRestore.sln" (Restore target) (1) ->
(Restore target) ->
  C:\path\to\ProjectA\ProjectA.csproj : error NU1004: The project reference cppprojecta has changed. Current dependencies: None lock file's dependencies: Common.The packages lock file is inconsistent with the project dependencies so restore can't be run in locked mode. Disable the RestoreLockedMode MSBuild property or pass an explicit --force-evaluate option to run restore to update the lock file. [C:\path\to\LockedRestore.sln]

    1 Warning(s)
    1 Error(s)

Time Elapsed 00:00:01.10

But I can confirm that using msbuild -t:restore -p:RestoreUseStaticGraphEvaluation=true LockedRestore.sln is succeeding. First attempt setting the <RestoreUseStaticGraphEvaluation>true</RestoreUseStaticGraphEvaluation> property didn't work, but the command line argument did the trick, the restore finished successfully.

What's the deal with Static Graph Evaluation? I assume it's the only option I have, is it reliable to use at this point? The Microsoft article states it's "experimental" but the info appears to be from years ago. The issue page (#9803) advises me to check for differences by running restore with and without static graph evaluation, but obviously I can't compare these two when one of them is not finishing successfully.

Thanks so far for pointing me to a workaround, I'll try this in production.

[codereader]

Update: Static Graph Evaluation is failing on the bigger, productive solution file:

C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet.RestoreEx.targets(19,5): error : MSB0001: Internal MSBuild Error: Only transitive references
may reference inner builds that got generated by outer builds

[nkolev92]

What's the deal with Static Graph Evaluation? I assume it's the only option I have, is it reliable to use at this point? The Microsoft article states it's "experimental" but the info appears to be from years ago. The issue page (#9803) advises me to check for differences by running restore with and without static graph evaluation, but obviously I can't compare these two when one of them is not finishing successfully.

Restore with static graph evaluation is a feature that is really helpful for large repositories that have 50+ projects that would take a long time to parse an evaluate.
Despite it not being the default, it is heavily used by customers with large repos on their CIs, especially within Microsoft.

It's unfortunate that you're hitting that different issue. Not really sure what the problem is there. Do you mind creating a different issue for that?
cc @jeffkl