Building CORE_ROOT is broken

[BruceForstall]

src\tests\build.cmd x64 checked generatelayoutonly fails with

c:\gh\runtime\artifacts\bin\coreclr\windows.x64.Checked\build\Microsoft.NETCore.Native.Publish.targets(77,5): error : The PrivateSdkAssemblies ItemGroup is required for _ComputeAssembliesToCompileToNative [c:\gh\runtime\src\coreclr\tools\aot\crossgen2\crossgen2_publish.csproj]

full log:

c:\gh\runtime>src\tests\build.cmd x64 checked generatelayoutonly
BUILDTEST: Starting Build at 10:19:41.63
BUILDTEST: Commencing CoreCLR test build
BUILDTEST: Common MSBuild args: "/p:TargetOS=windows" "/p:Configuration=Checked" "/p:TargetArchitecture=x64" "/p:RuntimeFlavor=coreclr" /nologo /verbosity:minimal /clp:Summary /maxcpucount
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.12.0-pre.3.0
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
BUILDTEST: powershell -NoProfile -ExecutionPolicy ByPass -NoLogo -Command "c:\gh\runtime\eng\common\msbuild.ps1"   c:\gh\runtime\src\tests\build.proj -warnAsError:0 /t:TestBuild /nodeReuse:false  /p:RestoreDefaultOptimizationDataPackage=false /p:PortableBuild=true  /p:UsePartialNGENOptimization=false /maxcpucount '/flp:Verbosity=normal;LogFile="c:\gh\runtime\artifacts\log\TestBuild_windows__x64__Checked.log"' '/flp1:WarningsOnly;LogFile="c:\gh\runtime\artifacts\log\TestBuild_windows__x64__Checked.wrn"' '/flp2:ErrorsOnly;LogFile="c:\gh\runtime\artifacts\log\TestBuild_windows__x64__Checked.err"' '/bl:"c:\gh\runtime\artifacts\log\TestBuild_windows__x64__Checked.binlog"'  "/p:TargetOS=windows" "/p:Configuration=Checked" "/p:TargetArchitecture=x64" "/p:RuntimeFlavor=coreclr" /nologo /verbosity:minimal /clp:Summary /maxcpucount

  [10:19:48.56] Restoring all packages...

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\test_dependencies_fs\test_dependencies.fsproj (in 1.69 sec).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\test_dependencies\test_dependencies.csproj (in 178 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\CoreCLRTestLibrary\CoreCLRTestLibrary.csproj (in 298 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\GenerateHWIntrinsicTests\GenerateHWIntrinsicTests_Arm.csproj (in 170 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\GenerateHWIntrinsicTests\GenerateHWIntrinsicTests_General.csproj (in 165 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\GenerateHWIntrinsicTests\GenerateHWIntrinsicTests_X86.csproj (in 163 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\XUnitLogChecker\XUnitLogChecker.csproj (in 247 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\XUnitWrapperGenerator\XUnitWrapperGenerator.csproj (in 608 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\XUnitWrapperLibrary\XUnitWrapperLibrary.csproj (in 159 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\XHarnessRunnerLibrary\XHarnessRunnerLibrary.csproj (in 211 ms).
    1 of 2 projects are up-to-date for restore.

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\external\external.csproj (in 370 ms).

    Determining projects to restore...
    Restored c:\gh\runtime\src\tests\Common\ilasm\ilasm.ilproj (in 199 ms).
  [10:20:29.07] Restoring all packages...Done.
  Building Targeting Pack
  external ->
  Determining projects to restore...
  Restored c:\gh\runtime\src\tools\illink\src\linker\ref\Mono.Linker.csproj (in 220 ms).
  Restored c:\gh\runtime\src\coreclr\tools\aot\crossgen2\crossgen2_publish.csproj (in 220 ms).
  Restored c:\gh\runtime\src\coreclr\tools\aot\ILCompiler.ReadyToRun\ILCompiler.ReadyToRun.csproj (in 227 ms).
  Restored c:\gh\runtime\src\coreclr\tools\aot\ILCompiler.Diagnostics\ILCompiler.Diagnostics.csproj (in 220 ms).
  Restored c:\gh\runtime\src\installer\pkg\sfx\Microsoft.NETCore.App\Microsoft.NETCore.App.Crossgen2.sfxproj (in 220 ms).
  Restored c:\gh\runtime\src\coreclr\tools\aot\ILCompiler.TypeSystem\ILCompiler.TypeSystem.csproj (in 249 ms).
  Restored c:\gh\runtime\src\tools\illink\src\linker\Mono.Linker.csproj (in 249 ms).
  Restored c:\gh\runtime\src\coreclr\tools\aot\ILCompiler.DependencyAnalysisFramework\ILCompiler.DependencyAnalysisFramework.csproj (in 249 ms).
  Restored c:\gh\runtime\src\tools\illink\src\ILLink.RoslynAnalyzer\ILLink.RoslynAnalyzer.csproj (in 290 ms).
  Restored c:\gh\runtime\src\libraries\System.Runtime.InteropServices\gen\DownlevelLibraryImportGenerator\DownlevelLibraryImportGenerator.csproj (in 290 ms).
  Restored c:\gh\runtime\src\tools\illink\src\ILLink.CodeFix\ILLink.CodeFixProvider.csproj (in 295 ms).
  Restored c:\gh\runtime\src\libraries\System.Runtime.InteropServices\gen\Microsoft.Interop.SourceGeneration\Microsoft.Interop.SourceGeneration.csproj (in 297 ms).
  Restored c:\gh\runtime\src\tools\illink\src\ILLink.Tasks\ILLink.Tasks.csproj (in 428 ms).
  ILLink.RoslynAnalyzer -> c:\gh\runtime\artifacts\bin\ILLink.RoslynAnalyzer\x64\Release\netstandard2.0\ILLink.RoslynAnalyzer.dll
  ILLink.CodeFixProvider -> c:\gh\runtime\artifacts\bin\ILLink.CodeFixProvider\x64\Release\netstandard2.0\ILLink.CodeFixProvider.dll
  Mono.Linker -> c:\gh\runtime\artifacts\bin\Mono.Linker\ref\x64\Release\net9.0\illink.dll
  Mono.Linker -> c:\gh\runtime\artifacts\bin\Mono.Linker\x64\Release\net9.0\win-x64\illink.dll
  ILLink.Tasks -> c:\gh\runtime\artifacts\bin\ILLink.Tasks\x64\Release\net9.0\win-x64\ILLink.Tasks.dll
  Mono.Linker -> c:\gh\runtime\artifacts\bin\Mono.Linker\x64\Release\net9.0\illink.dll
  ILLink.Tasks -> c:\gh\runtime\artifacts\bin\ILLink.Tasks\x64\Release\net9.0\ILLink.Tasks.dll
  ILCompiler.DependencyAnalysisFramework -> c:\gh\runtime\artifacts\bin\ILCompiler.DependencyAnalysisFramework\x64\Checked\ILCompiler.DependencyAnalysisFramework.dll
  ILCompiler.TypeSystem -> c:\gh\runtime\artifacts\bin\ILCompiler.TypeSystem\x64\Checked\ILCompiler.TypeSystem.dll
  ILCompiler.Diagnostics -> c:\gh\runtime\artifacts\bin\ILCompiler.Diagnostics\x64\Checked\ILCompiler.Diagnostics.dll
  ILCompiler.ReadyToRun -> c:\gh\runtime\artifacts\bin\ILCompiler.ReadyToRun\x64\Checked\ILCompiler.ReadyToRun.dll
  crossgen2_publish -> c:\gh\runtime\artifacts\bin\crossgen2_publish\x64\Checked\crossgen2.dll
c:\gh\runtime\artifacts\bin\coreclr\windows.x64.Checked\build\Microsoft.NETCore.Native.Publish.targets(77,5): error : The PrivateSdkAssemblies ItemGroup is required for _ComputeAssembliesToCompileToNative [c:\gh\runtime\src\corec
lr\tools\aot\crossgen2\crossgen2_publish.csproj]
  Determining projects to restore...
  All projects are up-to-date for restore.

Build FAILED.

c:\gh\runtime\artifacts\bin\coreclr\windows.x64.Checked\build\Microsoft.NETCore.Native.Publish.targets(77,5): error : The PrivateSdkAssemblies ItemGroup is required for _ComputeAssembliesToCompileToNative [c:\gh\runtime\src\corec
lr\tools\aot\crossgen2\crossgen2_publish.csproj]
    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:59.19
Build failed with exit code 1. Check errors above.

Note that this is documented by https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/coreclr/testing.md#building-the-core_root

This was possibly broken by #106965?

@am11 @jkotas

[BruceForstall]

@agocke

[BruceForstall]

I see some CI jobs use /p:UsePublishedCrossgen2=false which causes the script to succeed (and maybe does less work, so might be faster, which is good).

[am11]

Those CI legs are separate SPMI related mostly, because they don't require crossgen2. This is passing in CI apparently.

[ivdiazsa]

Trying to reproduce it on the latest runtime. Just yesterday, I was able to generate the CORE_ROOT just fine with a few days old clone and it worked, at least on Linux.

[agocke]

Tried this locally using the exact instructions at https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/coreclr/testing.md#building-the-core_root, can't repro.

[jkotas]

@BruceForstall This error means that your build does not have the live native AOT compiler that is needed by src\tests\build.cmd now. What are the command that you use to build the runtime?