[Xamarin.Android.Build.Tasks] Reduce rebuild cascades #549

jonpryor · 2017-04-06T02:28:53Z

Context: 518e57c

The scenario: rebuild Xamarin.Android.Build.Tasks.csproj:

$ xbuild src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.csproj

If nothing has changed, the expectation is that this should be
reasonably quick, because nothing has changed.

Unfortunately, that's not the case; a rebuild could take upwards of
30sec on my local machine, becaues of rebuild cascades:

Target CoreCompile needs to be built as input file 'Xamarin.Android.Tools.BootstrapTasks/GenerateProfile.cs' is newer than output file 'obj/Debug/Xamarin.Android.Tools.BootstrapTasks.dll'
Target _BuildJNIEnv needs to be built as input file '../../bin/BuildDebug/jnienv-gen.exe' is newer than output file 'Android.Runtime/JNIEnv.g.cs'
Target CoreCompile needs to be built as input file 'Android.Runtime/JNIEnv.g.cs' is newer than output file 'obj/Debug/android-25/Mono.Android.dll'
Target _GenerateMonoAndroidDex18 needs to be built as input file '../../bin/Debug/lib/xbuild-frameworks/MonoAndroid/v7.1/mono.android.jar' is newer than output file '../../bin/Debug/lib/xbuild-frameworks/MonoAndroid/v7.1/mono.android.dex'
Target _CopyExtractedMultiDexJar needs to be built as input file '$HOME/android-toolchain/sdk/extras/android/m2repository/com/android/support/multidex/1.0.1/multidex-1.0.1.aar' is newer than output file '../../bin/Debug/lib/xbuild/Xamarin/Android/../../../mandroid/android-support-multidex.jar'

...and if I'm really unlucky:

Target _BuildUnlessCached needs to be built as input file '.../xamarin-android/external/mono/autogen.sh' is newer than output file '../../bin/Debug//lib/xbuild-frameworks/MonoAndroid/v1.0/FSharp.Core.dll'

...as that means an external/mono rebuild (!).

Most of these are due to missing <Touch/> task use, to ensure that a
created/generated file is newer than the Inputs of the associated
target.

The <GenerateProfile/> task is slightly more complicated: in that
case, we only want the file timestamp to change if the file contents
have changed. Alter the GenerateProfile.Execute() logic to perform a
content diff before writing to the file, resulting in a new timestamp.

Finally, fix GenerateJavaCallableWrappers target use from
Mono.Android.csproj by providing a "real" value for
$(JavaCallableWrapperAbsAssembly) so that the
GenerateJavaCallableWrappers target's Inputs and Outputs
reference valid (existing) files. The previous value used a
$(JavaCallableWrapperOutputPathAbs) property, which doesn't appear
to have been defined anywhere, and thus was "".

Performing these changes reduces my typical "no change"
Xamarin.Android.Build.Tasks.csproj rebuild time from ~30sec down to
a more reasonable ~10sec, which is much better (though more than I'd
personally like).

atsushieno · 2017-04-06T04:23:32Z

src/Mono.Android/Mono.Android.targets

@@ -128,6 +129,7 @@
        SourceFiles="$(IntermediateOutputPath)__dex\classes.dex"
        DestinationFiles="$(OutputPath)mono.android.dex"
    />
+    <Touch Files="$(OutputPath)mono.android.dex" />


I don't understand why you have to Touch this file, and I believe unnecessary touching is bad because it can cause possible later rebuild cascades, but anyhow changes in this code path will be simply lost because it will be killed because it is jack!
https://github.com/xamarin/xamarin-android/pull/514/files#diff-77c7d22cbe4c79b8f43dd2e97eb81688

It should have been actually already killed (I was totally unaware that jack could be used outside our MSBuild tasks).

And from what I observed, these _GenerateMonoAndroidDex targets are not in use.

Losing changes here is fine; i just want to speed up a specific local rebuild, and I encountered this target constantly re-running.

If you'd prefer, I can just kill this target and integrate the appropriate changes from PR #514?

Either is fine, but the more important question here is: what is triggering this target? Could you check the diagnostic build output and see which targets depend on these _GenerateMonoAndroidDex* targets?

what is triggering this target?

The e.g. _GenerateMonoAndroidDex18 target is AfterTargets="GenerateJavaCallableWrappers", and GenerateJavaCallableWrappers is -- for Mono.Android.csproj -- AfterTargets="CoreBuild".

Yes, there are circumstances where AfterTargets works! :-)

Not that the use of AfterTargets is immediately apparent from diagnostic build output:

Building target "_GenerateMonoAndroidDex16" in project ".../xamarin-android/src/Mono.Android/Mono.Android.csproj" (".../xamarin-android/src/Mono.Android/Mono.Android.targets"); "GenerateJavaCallableWrappers" depends on it.

I think the message has it backwards, but the actual ordering is correct (truncated output):

Building target "GenerateJavaCallableWrappers" Done building target "GenerateJavaCallableWrappers" Building target "_GenerateMonoAndroidDex16" Building target "_GenerateMonoAndroidDex18" Done building target "GenerateJavaCallableWrappers"

But why am I NOT seeing any references to those targets?

/sources/monodroid/xamarin-android$ grep -R GenerateMonoAndroidDex
/sources/monodroid/xamarin-android$ cd ../monodroid
/sources/monodroid/monodroid$ grep -R GenerateMonoAndroidDex
/sources/monodroid/monodroid$

@atsushieno: Something is very fishy with that grep, as it's not finding the target definitions within the repo:

$ git grep GenerateMonoAndroidDex src/Mono.Android/Mono.Android.targets: <Target Name="_GenerateMonoAndroidDex16" src/Mono.Android/Mono.Android.targets: <Target Name="_GenerateMonoAndroidDex18"

Well, I was running it from my o-preview tree where I removed them by myself. Your grep results also say that they are not REFERENCED.

They're not explicitly referenced. They're implicitly referenced, via AfterTargets.

Alright. We should completely eliminate those use of "AfterTargets", but that can be done in different commits.

radekdoulik · 2017-04-06T11:06:54Z

src/Mono.Android/Mono.Android.csproj

@@ -315,7 +315,7 @@
  </PropertyGroup>
  <Import Project="Mono.Android.targets" />
  <PropertyGroup>
-    <JavaCallableWrapperAbsAssembly>$(JavaCallableWrapperOutputPathAbs)$(AssemblyName).dll</JavaCallableWrapperAbsAssembly>
+    <JavaCallableWrapperAbsAssembly>$(OutputPath)$(AssemblyName).dll</JavaCallableWrapperAbsAssembly>
    <JavaCallableWrapperAfterTargets>CoreBuild</JavaCallableWrapperAfterTargets>


IIRC, this needs to be an absolute path.

It seemed to build for me without being an absolute path...

radekdoulik · 2017-04-06T11:08:52Z

There's also similar issue with opentk-jcw rule. I have an unfinished patch for that. Will complete it when I get back.

Context: dotnet@518e57c The scenario: rebuild `Xamarin.Android.Build.Tasks.csproj`: $ xbuild src/Xamarin.Android.Build.Tasks/Xamarin.Android.Build.Tasks.csproj *If nothing has changed*, the expectation is that this should be reasonably quick, because *nothing has changed*. Unfortunately, that's not the case; a rebuild could take upwards of 30sec on my local machine, becaues of rebuild cascades: Target CoreCompile needs to be built as input file 'Xamarin.Android.Tools.BootstrapTasks/GenerateProfile.cs' is newer than output file 'obj/Debug/Xamarin.Android.Tools.BootstrapTasks.dll' Target _BuildJNIEnv needs to be built as input file '../../bin/BuildDebug/jnienv-gen.exe' is newer than output file 'Android.Runtime/JNIEnv.g.cs' Target CoreCompile needs to be built as input file 'Android.Runtime/JNIEnv.g.cs' is newer than output file 'obj/Debug/android-25/Mono.Android.dll' Target _GenerateMonoAndroidDex18 needs to be built as input file '../../bin/Debug/lib/xbuild-frameworks/MonoAndroid/v7.1/mono.android.jar' is newer than output file '../../bin/Debug/lib/xbuild-frameworks/MonoAndroid/v7.1/mono.android.dex' Target _CopyExtractedMultiDexJar needs to be built as input file '$HOME/android-toolchain/sdk/extras/android/m2repository/com/android/support/multidex/1.0.1/multidex-1.0.1.aar' is newer than output file '../../bin/Debug/lib/xbuild/Xamarin/Android/../../../mandroid/android-support-multidex.jar' ...and if I'm *really* unlucky: Target _BuildUnlessCached needs to be built as input file '.../xamarin-android/external/mono/autogen.sh' is newer than output file '../../bin/Debug//lib/xbuild-frameworks/MonoAndroid/v1.0/FSharp.Core.dll' ...as that means an `external/mono` rebuild (!). Most of these are due to missing `<Touch/>` task use, to ensure that a created/generated file is newer than the `Inputs` of the associated target. The `<GenerateProfile/>` task is slightly more complicated: in that case, we only want the file timestamp to change if the file contents have changed. Alter the `GenerateProfile.Execute()` logic to perform a content diff before writing to the file, resulting in a new timestamp. Finally, fix `GenerateJavaCallableWrappers` target use from `Mono.Android.csproj` by providing a "real" value for `$(JavaCallableWrapperAbsAssembly)` so that the `GenerateJavaCallableWrappers` target's `Inputs` and `Outputs` reference valid (existing) files. The previous value used a `$(JavaCallableWrapperOutputPathAbs)` property, which doesn't appear to have been defined anywhere, and thus was `""`. Performing these changes reduces my typical "no change" `Xamarin.Android.Build.Tasks.csproj` rebuild time from ~30sec down to a more reasonable ~10sec, which is much better (though more than I'd personally like).

jonpryor · 2017-04-06T22:22:49Z

build

atsushieno · 2017-04-07T14:55:03Z

build

dnfclas added the cla-already-signed label Apr 6, 2017

jonpryor requested review from radekdoulik and atsushieno April 6, 2017 02:28

atsushieno reviewed Apr 6, 2017

View reviewed changes

radekdoulik reviewed Apr 6, 2017

View reviewed changes

jonpryor force-pushed the jonp-fix-cascading-rebuilds branch from 6181258 to 8aee88b Compare April 6, 2017 14:15

atsushieno merged commit 631c046 into dotnet:master Apr 7, 2017

github-actions bot locked and limited conversation to collaborators Feb 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Xamarin.Android.Build.Tasks] Reduce rebuild cascades #549

[Xamarin.Android.Build.Tasks] Reduce rebuild cascades #549

Uh oh!

jonpryor commented Apr 6, 2017

Uh oh!

atsushieno Apr 6, 2017

Uh oh!

atsushieno Apr 6, 2017

Uh oh!

jonpryor Apr 6, 2017

Uh oh!

atsushieno Apr 6, 2017

Uh oh!

jonpryor Apr 6, 2017

Uh oh!

atsushieno Apr 6, 2017

Uh oh!

jonpryor Apr 6, 2017

Uh oh!

atsushieno Apr 7, 2017

Uh oh!

jonpryor Apr 7, 2017

Uh oh!

atsushieno Apr 7, 2017

Uh oh!

radekdoulik Apr 6, 2017

Uh oh!

jonpryor Apr 6, 2017

Uh oh!

radekdoulik commented Apr 6, 2017

Uh oh!

jonpryor commented Apr 6, 2017

Uh oh!

atsushieno commented Apr 7, 2017

Uh oh!

Uh oh!

[Xamarin.Android.Build.Tasks] Reduce rebuild cascades #549

[Xamarin.Android.Build.Tasks] Reduce rebuild cascades #549

Uh oh!

Conversation

jonpryor commented Apr 6, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

radekdoulik commented Apr 6, 2017

Uh oh!

jonpryor commented Apr 6, 2017

Uh oh!

atsushieno commented Apr 7, 2017

Uh oh!

Uh oh!