Fix some fast up to date check issues in VS

[Pilchie]

Fix several issues:

1. Places where it's explicitly disabled.
2. Places where we used CopyToOutputDirectory="Always"
3. One project wasn't actually set to be built in VS.
4. A few things marked as input or output files that don't exist and aren't generated.

There are still 3 projects that don't build properly, and 5 that fail the fast up to date check for me with this, but getting closer.

[Pilchie]

Tagging @KirillOsenkov, just because I always do when messing with FastUpToDateCheck, and @drewnoakes because there is one case I still don't understand.

Remaining issues:

172>FastUpToDate: Output 'C:\Code\aspnetcore\artifacts\obj\Microsoft.AspNetCore.App.Runtime\Debug\net6.0\win-x64\Microsoft.AspNetCore.App.Runtime.dll' does not exist, not up to date. (Microsoft.AspNetCore.App.Runtime)

410>FastUpToDate: No inputs are newer than earliest output 'C:\Code\aspnetcore\artifacts\bin\dotnet-microsoft.openapi.Tests\Debug\net6.0\Microsoft.DotNet.Open.Api.Tools.Tests.pdb' (4/24/2021 3:53:21 PM). (dotnet-microsoft.openapi.Tests)
410>FastUpToDate: Latest write timestamp on input marker is 4/24/2021 3:55:08 PM on 'C:\Code\aspnetcore\artifacts\obj\Microsoft.dotnet-openapi\Debug\net6.0\Microsoft.dotnet-openapi.csproj.CopyComplete'. (dotnet-microsoft.openapi.Tests)
410>FastUpToDate: Write timestamp on output marker is 4/24/2021 3:55:08 PM on 'C:\Code\aspnetcore\artifacts\obj\dotnet-microsoft.openapi.Tests\Debug\net6.0\dotnet-microsoft.openapi.Tests.csproj.CopyComplete'. (dotnet-microsoft.openapi.Tests)
410>FastUpToDate: Input marker is newer than output marker, not up to date. (dotnet-microsoft.openapi.Tests)

413>FastUpToDate: Output 'C:\Code\aspnetcore\artifacts\bin\Microsoft.Extensions.ApiDescription.Server\Debug\netcoreapp2.1\Microsoft.Extensions.ApiDescription.Server.dll' does not exist, not up to date. (Microsoft.Extensions.ApiDescription.Server)

458>FastUpToDate: Output 'C:\Code\aspnetcore\artifacts\obj\Microsoft.AspNetCore.ANCMSymbols\Debug\net6.0\win-x64\Microsoft.AspNetCore.ANCMSymbols.dll' does not exist, not up to date. (Microsoft.AspNetCore.ANCMSymbols)

494>FastUpToDate: Output 'C:\Code\aspnetcore\artifacts\obj\Microsoft.AspNetCore.App.Ref\Debug\net6.0\Microsoft.AspNetCore.App.Ref.dll' does not exist, not up to date. (Microsoft.AspNetCore.App.Ref)

172, 413, 458, and 494 are projects that don't actually generate an output assembly. Any idea how to make them play nicer with the fast up to date check @dotnet/aspnet-build or @drewnoakes.

410 is the one that is puzzling me. I don't actually know what that log means. @drewnoakes can you help me out?

[KirillOsenkov]

If you're looking at the build, I randomly noticed there are a couple of pending changes after a clean build (file BOM changes?) to blazor.server.js and a couple launchSettings.json files in tests rejiggered.

[KirillOsenkov]

Regarding 410, if you have a binlog, look at the timing of file writes for the CopyComplete markers - why is the CopyComplete marker for the Tests project written earlier than the marker for the non-test project? Could be insufficient build ordering?? (just random guesses here)

[KirillOsenkov]

You can append $time to the binlog viewer search query and it will show timestamps for results. So e.g. $task Csc $time. Also you can hover over targets, tasks etc. to view timestamps. There's also a crude Timeline view which I never have time to finish, but it can still be useful to understand the timing and ordering (time flows down, columns are build nodes).

[KirillOsenkov]

Hmm I've just realized it's the VS build, not command-line, it's harder to get a decent binlog that involves both projects. Not sure how to investigate this. If you keep building, does it still rebuild every time?

[drewnoakes]

172, 413, 458, and 494 are projects that don't actually generate an output assembly

You can remove the output item from the build, however if that leaves no outputs then the fast up-to-date check will always build it.

410 is the one that is puzzling me. I don't actually know what that log means.

410>FastUpToDate: No inputs are newer than earliest output 'C:\Code\aspnetcore\artifacts\bin\dotnet-microsoft.openapi.Tests\Debug\net6.0\Microsoft.DotNet.Open.Api.Tools.Tests.pdb' (4/24/2021 3:53:21 PM). (dotnet-microsoft.openapi.Tests)
410>FastUpToDate: Latest write timestamp on input marker is 4/24/2021 3:55:08 PM on 'C:\Code\aspnetcore\artifacts\obj\Microsoft.dotnet-openapi\Debug\net6.0\Microsoft.dotnet-openapi.csproj.CopyComplete'. (dotnet-microsoft.openapi.Tests)
410>FastUpToDate: Write timestamp on output marker is 4/24/2021 3:55:08 PM on 'C:\Code\aspnetcore\artifacts\obj\dotnet-microsoft.openapi.Tests\Debug\net6.0\dotnet-microsoft.openapi.Tests.csproj.CopyComplete'. (dotnet-microsoft.openapi.Tests)
410>FastUpToDate: Input marker is newer than output marker, not up to date. (dotnet-microsoft.openapi.Tests)

It looks like the dependency from Microsoft.dotnet-openapi.Tests.csproj on Microsoft.dotnet-openapi.csproj isn't tracking properly. I'm not familiar with the CopyComplete files. Could it possibly be related to using a variable in the ProjectReference in Microsoft.dotnet-openapi.Tests.csproj? Seems unlikely. I tried building your fork but don't have all the right bits on my machine and don't have time today to dig into that. Is this a repeatable failure (is there an observable race)?

[drewnoakes]

There seems to be a bunch of special handling of CopyComplete for Razor projects. Does that apply here? For example:

https://github.com/dotnet/sdk/blob/a30e465a2e2ea4e2550f319a2dc088daaafe5649/src/RazorSdk/Targets/Microsoft.NET.Sdk.Razor.Compilation.targets#L272-L274

[Pilchie]

@drewnoakes it did seem to be consistent.

@pranavkm / @captainsafia for the potential Razor SDK impact.

I feel like we struggled with projects that produced no output assembly other places too. @jasonmalinowski do you have any ideas there? Does Roslyn do anything interesting?

[jasonmalinowski]

Roslyn has a few projects we are really using for packaging; right now those build time. I haven't looked at fighting with it, maybe @tmat might know details.

[captainsafia]

There seems to be a bunch of special handling of CopyComplete for Razor projects. Does that apply here? For example:

That particular change only affects .NET 5 and below projects where we still produced a Views assembly. I don't suspect this will come into play with the changes here since we are targetting net6.0 and above.

[Pilchie]

@BrennanConroy / @HaoK if this passes, any objection to merging? There are still a few projects that don't produce outputs where the fast up to date check fails, but @drewnoakes is looking at those.

There are also two Blazor identity projects with issues, but I'll ask @javiercn to look at them when he's back.

Regardless, this set of changes should reduce needless rebuilds in VS significantly.

[BrennanConroy]

Looks good to me
Would be nice if NativeProjectReference handled the UpToDateCheckInput for us though 😃

[Pilchie]

Would be nice if NativeProjectReference handled the UpToDateCheckInput for us though

Agreed, but @rainersigwald said there isn't a reliable way to go from vcxproj -> output. Otherwise, we might be able to do it in our ResolveNativeProjectReference target. (Though I'm not sure if the fast up to date check runs targets to collect UpToDateCheckInput, or if they have to be static).

[rainersigwald]

Otherwise, we might be able to do it in our ResolveNativeProjectReference target. (Though I'm not sure if the fast up to date check runs targets to collect UpToDateCheckInput, or if they have to be static).

I though they had to be evaluation time constant but that might be wrong: https://github.com/dotnet/project-system/blob/c954a632f74afdac9118a1f505257107789ff00e/docs/up-to-date-check.md#customization talks about a target. So it might be doable to add the NativeContent item to UpToDateCheckInput after RAR/design-time RAR. @drewnoakes, that might be worth clarifying in that doc.

[dougbu]

Would it help to set $(GeneratePackageOnBuild) to true and add an @(UpToDateCheckBuilt) item for the produced package❔

[Pilchie]

Are you proposing that to solve the fact that this project doesn't generate an output assembly, or for some other reason?

[dougbu]

Yes, to solve the lack of an output assembly -- if it works 😃

[dougbu]

I'm confused by "Note that UpToDateCheckOutput exists but is deprecated and only maintained for backwards compatability." in the docs. The recommendation is to use UpToDateCheckBuilt instead but that controls project outputs, not inputs. @drewnoakes @rainersigwald❔ (Ignore this. I read UpToDateCheckInput for some reason.)

(Someone should fix the "compatibility" spelling in the doc too 😃)

[drewnoakes]

I though they had to be evaluation time constant but that might be wrong: https://github.com/dotnet/project-system/blob/c954a632f74afdac9118a1f505257107789ff00e/docs/up-to-date-check.md#customization talks about a target. So it might be doable to add the NativeContent item to UpToDateCheckInput after RAR/design-time RAR. @drewnoakes, that might be worth clarifying in that doc.

@rainersigwald yes, you can modify the set of UpToDateCheckInput/Output/Built items via the targets specified in those docs. We use design-time build data for that.

I'm unclear on the clarification you're after though.

I'm confused by "Note that UpToDateCheckOutput exists but is deprecated and only maintained for backwards compatability." in the docs.

@dougbu I know you edited this, but regardless: you're right, this is confusing. I've removed mention of the deprecation in dotnet/project-system#7840. Technically UpToDateCheckOutput is redundant, but in practice it's less confusing to allow its use, and we can't remove it anyway.

Hmm I've just realized it's the VS build, not command-line, it's harder to get a decent binlog that involves both projects. Not sure how to investigate this. If you keep building, does it still rebuild every time?

@KirillOsenkov since then you pointed me to the use of MSBuildDebugEngine environment variable for full-fidelity binlogs, as described in https://github.com/dotnet/project-system-tools#getting-higher-fidelity-logs-from-vs

@Pilchie can you suggest a solution filter and project to test the no-output case with? I've burnt my budget today on trying to get the solution building locally. Narrowing to a subset of projects might be helpful.

[Pilchie]

Can you suggest a solution filter and project to test the no-output case with?

src/Framework/Framework.slnf is probably the best example.

[Pilchie]

So it might be doable to add the NativeContent item to UpToDateCheckInput

Is the idea here to do something like:

<Target Name="CollectUpToDateCheckInputDesignTime" AfterTargets="ResolveAssemblyReference">
  <UpToDateCheckInput Include="@(NativeContent)" />
</Target>

And that would solve our native references?

@drewnoakes / @rainersigwald - does that look right?

[drewnoakes]

That's the idea. Check the implementation of that target in the .NET project system and make sure it's returning the same thing. From memory (I'm on a phone) you need Returns="@(UpToDateCheckInput)" on the target too. You may find BeforeTargets is easier.

I guess our documentation can be improved here with an example. I'll do that when I'm back.

[Pilchie]

Unable to repro the two npm/yarn projects writing to their own inputs today 🤷🏻‍♂️ .

[Pilchie]

Still looking if I can do something better with native content and no output projects, but I'm going to merge this in the meantime.