Build fails to touch out-of-date CopyUpToDateMarker

[sharwell]

During investigation of a Roslyn incremental build failure, I observed the following output:

10>FastUpToDate: No inputs are newer than earliest output 'C:\dev\roslyn\artifacts\obj\Microsoft.CodeAnalysis.Scripting.TestUtilities\Debug\net472\Microsoft.CodeAnalysis.Scripting.TestUtilities.xml' (12/20/2021 9:29:45 AM). Newest input is 'C:\dev\roslyn\artifacts\obj\Microsoft.CodeAnalysis.Test.Utilities\Debug\net472\ref\Microsoft.CodeAnalysis.Test.Utilities.dll' (12/20/2021 9:25:08 AM). (Microsoft.CodeAnalysis.Scripting.TestUtilities)
10>FastUpToDate: Latest write timestamp on input marker is 12/20/2021 9:29:40 AM on 'C:\dev\roslyn\artifacts\bin\Microsoft.CodeAnalysis.Test.Utilities\Debug\net472\Microsoft.CodeAnalysis.Test.Utilities.dll'. (Microsoft.CodeAnalysis.Scripting.TestUtilities)
10>FastUpToDate: Write timestamp on output marker is 12/15/2021 7:29:31 PM on 'C:\dev\roslyn\artifacts\obj\Microsoft.CodeAnalysis.Scripting.TestUtilities\Debug\net472\Microsoft.CodeAnalysis.Scripting.TestUtilities.csproj.CopyComplete'. (Microsoft.CodeAnalysis.Scripting.TestUtilities)
10>FastUpToDate: Input marker is newer than output marker, not up to date. (Microsoft.CodeAnalysis.Scripting.TestUtilities)
10>------ Build started: Project: Microsoft.CodeAnalysis.Scripting.TestUtilities, Configuration: Debug Any CPU ------
10>Microsoft.CodeAnalysis.Scripting.TestUtilities -> C:\dev\roslyn\artifacts\bin\Microsoft.CodeAnalysis.Scripting.TestUtilities\Debug\net6.0\Microsoft.CodeAnalysis.Scripting.TestUtilities.dll
10>Microsoft.CodeAnalysis.Scripting.TestUtilities -> C:\dev\roslyn\artifacts\bin\Microsoft.CodeAnalysis.Scripting.TestUtilities\Debug\net472\Microsoft.CodeAnalysis.Scripting.TestUtilities.dll
========== Build: 1 succeeded, 0 failed, 9 up-to-date, 0 skipped ==========

In this situation, the build was up-to-date except for the CopyUpToDateMarker. Unfortunately, due to #6698 the file was not touched, and the state was perpetually incorrect. A resolution in this situation is to ensure the _CopyFilesMarkedCopyLocal target touches CopyUpToDateMarker any time the marker is currently older than the newest build input.

[Forgind]

Check for understanding:
Normal build or incremental build #0 --> files are copied, but since files were copied, the CopyUpToDateMarker is not touched.
Incremental build #1+ --> no files are copied, so even though the CopyUpToDateMarker is out-of-date, it isn't touched.

I'm trying to think of what the ideal solution here is. Perhaps we should have one extra CopyUpToDateMarker2 that is touched if either CopyUpToDateMarker is touched or '@(ReferencesCopiedInThisBuild)' != ''. Then, if '@(ReferencesCopiedInThisBuild)' != '' and CopyUpToDateMarker2 was touched (in a subsequent incremental build), we also touch CopyUpToDateMarker? That feels kinda circular, but I think it would work. What do you think?

[sharwell]

Normal build or incremental build #0 --> files are copied, but since files were copied, the CopyUpToDateMarker is not touched.

I don't have a good way to verify this part of the question. It's not clear how I originally ended up in the situation, but I was stuck there and trying to automate the recovery from it.

[Forgind]

Also CC'ing @benvillalobos as author of the last CopyUpToDate change. What do you think of my CopyUpToDateMarker2 idea?

[Forgind]

Team triage: @sharwell, How did you get into a state in which all the copied files are up-to-date, but the marker isn't? If we copy all the files, we should also update the marker (since at least one would have been properly copied); if it was something external to mess with timestamps, that sounds adversarial.

[deralbert]

For those who may not be that familiar with the MsBuild, but are facing the same problem:

I have a solution with 121 projects. Some projects reference the others. Each project has around four or five references (of course there are no cycles). If I have changed something in any project (for example just saved .csproj again), it was enough to trigger a chain reaction, so that around 25 projects had to be rebuilt. The build log said exactly the same thing: No inputs are newer than earliest output... Input marker is newer than output marker, not up to date.

Solution (work around)

1. Create the file Directory.Build.targets near the .sln file.
2. Add to this file (that is the work around from @sharwell):

<Project>
    <!-- Workaround for https://github.com/dotnet/msbuild/issues/7161 -->
    <Target Name="WorkaroundDotnetMsbuild7161"
            AfterTargets="_CopyFilesMarkedCopyLocal"
            Condition="'$(BuildingInsideVisualStudio)' == 'true' AND '@(ReferenceCopyLocalPaths)' != ''">
        <Touch Files="@(CopyUpToDateMarker)"
               AlwaysCreate="true"
               Condition="'@(ReferencesCopiedInThisBuild)' != ''"/>
    </Target>
</Project>

After that, the problem should be resolved.

[benvillalobos]

@deralbert This solution will guarantee that every project will build every time because the marker is being updated. Is that what you want?

Could you also explain exactly how you encountered the scenario that prompted this fix? Did it consistently happen?

[deralbert]

@benvillalobos apologies for the delayed response.

Yes, the behavior with the buggy incremental build has been happening in my solution for years (at least three years). The solution is quite big and there are several projects that reference other projects. If I understand @sharwell correctly, the problem was that after a build, CopyUpToDateMarker was not touched. As a result, even if you didn't make any changes to the code, this file remained untouched. In my case, this file was for some reason never touched and that forced a new build.

I can't rule out that there is something else in my solution that could be the reason for this erroneous behavior, because the solution itself is quite old and big and mostly legacy code. Anyway, the code in Directory.Build.targets from above helped me.

Another thing I can suggest is that I could provide diagnostic data so you can understand what's going wrong in my solution. But I doubt if it would make much sense.

[benvillalobos]

If I understand _ correctly, the problem was that after a build, CopyUpToDateMarker was not touched.

At least we think that's the case. Unfortunately we don't know exactly what happened there, so it's hard to come to a solid conclusion:

I don't have a good way to verify this part of the question. It's not clear how I originally ended up in the situation, but I was stuck there and trying to automate the recovery from it.

You can file feedback at Developer Community and include confidential-to-Microsoft attachments. Please link the issue here so I can expedite routing it to our team. Most useful would be a binlog (ideally generated from the command line)

[deralbert]

@benvillalobos thank you for your answer.
I will discuss your suggestion in my development team and if we decide positively, I will do it.