[release/8.0] Update SDK

[SteveSandersonMS]

We've got several weeks behind ingesting SDK changes to release/8.0. The main challenge has been several changes in the Razor compiler. This PR contains a fix for a regression that was due to one such change.

I'll hopefully later update this PR further to remove the temporary APIs we used for @formname and @rendermode since the native APIs are now present in the Razor compiler.

[wtgodbe]

LGTM, but can you look at the conflicts in the .js files?

[SteveSandersonMS]

@wtgodbe I've fixed the merge conflicts, but the builds are even less healthy than before. I've had many attempts to make this build but am just getting so many garbage errors from all over (which have nothing to do with our code as far as I can see).

Is this something that you, the current build-ops person, and First Responders could take over with? Either our release/8.0 branch is unhealthy, or RTM builds of the SDK are unhealthy.

[wtgodbe]

@marcpopMSFT @dsplaisted @nagilson @joeloff we're seeing a lot of restore errors on Linux after updating to the RC1 SDK - is this a known issue? I've never seen this exact failure case before:

The process cannot access the file '/tmp/505bdf97-0de5-4af8-8b26-2b27ae4ab4e5/project.assets.json' because it is being used by another process.

[dsplaisted]

@marcpopMSFT @dsplaisted @nagilson @joeloff we're seeing a lot of restore errors on Linux after updating to the RC1 SDK - is this a known issue? I've never seen this exact failure case before:

The process cannot access the file '/tmp/505bdf97-0de5-4af8-8b26-2b27ae4ab4e5/project.assets.json' because it is being used by another process.

This sounds like the known issue described here: dotnet/core#8799

We are working on a fix for GA.

FYI @JL03-Yue @marcpopMSFT

[wtgodbe]

@dsplaisted do you have a recommendation on how we can work around this in aspnetcore? The known issue mentions creating dotnet-tools.json files in subfolders & calling dotnet tool restore in each one, but the invocation of dotnet tool restore is done by Arcade - CC @dotnet/dnceng

[dsplaisted]

Is the list of tools to restore owned by Arcade or is that part of this repo? If it's entirely owned by Arcade, then the fix may need to be in Arcade :-(. If it's owned by this repo, then maybe you can disable the Arcade tools restore and add your own.

[wtgodbe]

We own the list of tools, and can move them into subfolders. @dotnet/dnceng can someone point me to where/how dotnet tool restore gets invoked? I couldn't find it with a simple search

[dougbu]

@dsplaisted Arcade handles the .config/dotnet-tools.json file by running dotnet tool restore in the repo root folder if the file exists. it does nothing if the file doesn't exist.

oh, and if the file exists, the command runs just before the restore target

[dougbu]

@wtgodbe look near the bottom of src\Microsoft.DotNet.Arcade.Sdk\tools\Tools.proj in dotnet/arcade

[wtgodbe]

Alright, we could probably get away with moving our dotnet-tools.json files into a different directory, with subdirs, and hook a custom target in https://github.com/dotnet/aspnetcore/blob/main/eng/Tools.props. I'll try that now

[dsplaisted]

Alright, we could probably get away with moving our dotnet-tools.json files into a different directory, with subdirs, and hook a custom target in https://github.com/dotnet/aspnetcore/blob/main/eng/Tools.props. I'll try that now

Sounds good. Let us know how it works. It's not a pretty workaround but was the only thing I could come up with.

[wtgodbe]

Seems to be working

[SteveSandersonMS]

@wtgodbe Still not quite working unfortunately:

.packages\microsoft.dotnet.helix.sdk\8.0.0-beta.23463.1\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Correlation Payload 'D:\a_work\1\s.packages\dotnet-dump\6.0.322601\dotnet-dump.6.0.322601.nupkg' not found.

[wtgodbe]

Going to try one more thing

[wtgodbe]

@SteveSandersonMS just pushed a new commit, can you resolve the conflict?

[SteveSandersonMS]

@SteveSandersonMS just pushed a new commit, can you resolve the conflict?

Done

[SteveSandersonMS]

Sadly the Correlation Payload ... not found error is still happening.

[wtgodbe]

@dotnet/dnceng @dsplaisted we're running dotnet tool restore on checked-in dotnet-tools.json files here, and the build logs claim that the tools are being restored:

Tool 'dotnet-ef' (version '8.0.0-rc.1.23419.6') was restored. Available commands: dotnet-ef

But they aren't winding up in the package cache:

##[error].packages\microsoft.dotnet.helix.sdk\8.0.0-beta.23463.1\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Correlation Payload 'D:\a_work\1\s.packages\dotnet-ef\8.0.0-rtm.23502.7\dotnet-ef.8.0.0-rc.1.23419.6.nupkg' not found.

In my local build I saw the same behavior. The package that was actually restored for me was 7.0.11. The binlog didn't have any extra info, it just showed the log message. Are we doing something wrong?

[wtgodbe]

Whoops, got versions crossed. Fixing

[wtgodbe]

@SteveSandersonMS I took the 8.0 versions of the .js files here just to get CI re-running, feel free to undo that if they need to be changed

[dsplaisted]

Whoops, got versions crossed. Fixing

Does this mean you figured out the issue where tools didn't seem to be restored correctly?

[wtgodbe]

Does this mean you figured out the issue where tools didn't seem to be restored correctly?

Latest build will confirm. I did see the wrong dotnet-ef package getting restored when I built locally, though, but hopefully that was user error.

[wtgodbe]

Ok @dsplaisted, error is still present:

Tool 'dotnet-ef' (version '8.0.0-rc.1.23419.6') was restored. Available commands: dotnet-ef

D:\a_work\1\s.packages\microsoft.dotnet.helix.sdk\8.0.0-beta.23463.1\tools\Microsoft.DotNet.Helix.Sdk.MonoQueue.targets(45,5): error : Correlation Payload 'D:\a_work\1\s.packages\dotnet-ef\8.0.0-rc.1.23419.6\dotnet-ef.8.0.0-rc.1.23419.6.nupkg' not found. [D:\a_work\1\s\eng\helix\helix.proj]

I'm gonna set the verbosity to diag in dotnet tool restore to try to get more info

[wtgodbe]

Yeah, something weird's going on with dotnet tool restore:

  [NuGet Manager] [Info]   GET https://pkgs.dev.azure.com/azure-public/3ccf6661-f8ce-4e8a-bb2e-eff943ddd3c7/_packaging/58ca65bb-e6c1-4210-88ac-fa55c1cd7877/nuget/v3/registrations2-semver2/dotnet-ef/index.json
  [NuGet Manager] [Info]   NotFound https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/d1622942-d16f-48e5-bc83-96f4539e7601/nuget/v3/registrations2-semver2/dotnet-ef/index.json 38ms
  [NuGet Manager] [Info]   NotFound https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/9c2ea29a-00e0-4bae-b470-161fdab1f360/nuget/v3/registrations2-semver2/dotnet-ef/index.json 38ms
  [NuGet Manager] [Info]   NotFound https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/a65e5cb4-26c0-410f-9457-06db3c5254be/nuget/v3/registrations2-semver2/dotnet-ef/index.json 39ms
  [NuGet Manager] [Info]   NotFound https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/c9f8ac11-6bd8-4926-8306-f075241547f7/nuget/v3/registrations2-semver2/dotnet-ef/index.json 38ms
  [NuGet Manager] [Info]   OK https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/e31c6eea-0277-49f3-8194-142be67a9f72/nuget/v3/registrations2-semver2/dotnet-ef/index.json 39ms
  [NuGet Manager] [Info]   NotFound https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/a8a526e9-91b3-4569-ba2d-ff08dbb7c110/nuget/v3/registrations2-semver2/dotnet-ef/index.json 41ms
  [NuGet Manager] [Info]   NotFound https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/1a5f89f6-d8da-4080-b15f-242650c914a8/nuget/v3/registrations2-semver2/dotnet-ef/index.json 50ms
  [NuGet Manager] [Info]   OK https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/registrations2-semver2/dotnet-ef/index.json 69ms
  [NuGet Manager] [Info]   GET https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/dotnet-ef/index.json
  [NuGet Manager] [Info]   OK https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/dotnet-ef/index.json 50ms
  [NuGet Manager] [Info]   GET https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/dotnet-ef/7.0.11/dotnet-ef.7.0.11.nupkg
  [NuGet Manager] [Info]   OK https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/dotnet-ef/7.0.11/dotnet-ef.7.0.11.nupkg 81ms
  Skipping NuGet package signature verification.
  Tool 'dotnet-ef' (version '8.0.0-rc.1.23419.6') was restored. Available commands: dotnet-ef

@dotnet/dnceng @dsplaisted the tool claims it's restoring the 8.0 version we want, but silently restores 7.0.11 (Note the bottom 3 lines). Is it a feed issue?

[wtgodbe]

Same thing for dotnet-serve

  [NuGet Manager] [Info]   OK https://pkgs.dev.azure.com/dnceng/9ee6d478-d288-47f7-aacc-f6e6d082ae6d/_packaging/45bacae2-5efb-47c8-91e5-8ec20c22b4f8/nuget/v3/flat2/dotnet-serve/1.10.172/dotnet-serve.1.10.172.nupkg 397ms
  Skipping NuGet package signature verification.
  Tool 'dotnet-serve' (version '1.10.93') was restored. Available commands: dotnet-serve

[wtgodbe]

A fix in the SDK is incoming to unblock this

[SteveSandersonMS]

@wtgodbe @mkArtakMSFT Do you think at this point we should consider merging anyway? It's not more correct in any sense to hold off from merging this. Even though we are getting "green" builds in release/8.0, they represent a completely fictional world where we're using a very incorrect version of the SDK - they don't do as much to prove the code is correct as if we got the latest SDK and allowed the Helix queue to be down.

If the SDK fix is going to land today for definite then holding off would be reasonable. If that's not known for sure, I suggest we need to find another way forwards in the short term.

[wtgodbe]

The SDK fix (dotnet/sdk#35884) should land today, I reached out to confirm it's ready to merge. I don't think we should merge this on red, as currently Helix tests are blocked from running entirely, so we'd have basically no test coverage on the repo. We could disable the tests that use these tools, but that's all Server tests and all EF tests, plus we wouldn't get dumps from failed tests.

The other option, if the SDK fix gets blocked for some reason, is to specify exactly the versions that the tools choose to download anyways (the bug is that dotnet tool restore always chooses the most recent non-preview version of the tool, regardless of what you specify), but that's pretty hacky and super fragile (the next time a non-preview version of any of the 4 tools gets pushed, we'd break). If we do get blocked, let me know if you want to go down that route. One potential wrinkle there is that we download a 7.0 version of dotnet-ef - I don't know if that would cause problems w/ our 8.0 bits