[Xamarin.Android.Build.Tasks] jnimarshalmethod-gen.exe integration (#2153)

Context: https://github.com/xamarin/xamarin-android/projects/1
Context: #2138

A "JNI Marshal Method" is a method which the JVM eventually executes
when a Java `native` method is invoked.  JNI Marshal Methods are
currently contained within binding assemblies for all `virtual`
methods, e.g. `Java.Lang.Object` contains:

	partial class Object {
	  static int n_GetHashCode (IntPtr jnienv, IntPtr native__this)
	  {
	    var __this = Object.GetObject<Object> (jnienv, native__this, JniHandleOwnership.DoNotTransfer);
	    return __this.GetHashCode ();
	  }
	}

If a C# class overrides `Java.Lang.Object.GetHashCode()`, then
`Object.n_GetHashCode()` will be invoked when Java code calls
`value.hashCode()` on an instance of that C# class.

JNI Marshal Methods are responsible for marshaling parameters and
return types.

However, there is one problem with JNI Marshal Methods as currently
constructed: they *require* the use of `System.Reflection.Emit`,
via `Android.Runtime.JNINativeWrapper.CreateDelegate()`.
`Object.n_GetHashCode()` isn't *directly* registered with JNI.
Instead, a "wrapper" is generated at runtime, and it's the wrapper
which is registered with JNI.  The wrapper effectively does:

	int wrapper_n_GetHashCode (IntPtr jnienv, IntPtr native__this)
	{
	  try {
	    JNIEnv.WaitForBridgeProcessing();
	    return n_GetHashCode (jnienv, native__this);
	  }
	  catch (Exception e) when Debugger.IsAttached || !JNIEnv.PropagateExceptions {
	    AndroidEnvironment.UnhandledException (e);
	  }
	}

Previously, the use of `DynamicMethod` and `System.Reflection.Emit`
was unavoidable, as we needed to ensure that all exceptions were
handled appropriately, no matter which `generator` version was used
to generate the JNI Marshal Methods.

Enter `jnimarshalmethod-gen.exe`, which is a utility which will
generate "complete" JNI Marshal Methods that can be registered
directly with JNI, *without* using `JNINativeWrapper` or requiring
use of `DynamicMethod`.

`jnimarshalmethod-gen.exe` would process the hypothetical C#
`GetHashCode()` override and generate the JNI Marshal Method:

	partial class ExampleObjectSubclass : Java.Lang.Object {
	  public override int GetHashCode () {return 42;}

	  /* generated by `jnimarshalmethod-gen.exe` */
	  partial class '__<$>_jni_marshal_methods' {
	    public int GetHashCode (IntPtr __jnienv, IntPtr native__this)
	    {
	      var jniTransition  = new JniTransition (__jnienv);
	      JniRuntime runtime = default;
	      try {
	        runtime          = JniEnvironment.Runtime;
	        var valueManager = runtime.ValueManager;
	        valueManager.WaitForGCBridgeProcessing ();
	        var __this       = valueManager.GetValue<ExampleObjectSubclass>(native__this);
	        return __this.GetHashCode ();
	      }
	      catch (Exception e) when (runtime.ExceptionShouldTransitionToJni (ex)) {
	        jniTransition.SetPendingException (ex);
	      }
	      finally {
	        jniTransition.Dispose ();
	      }
	    }
	  }
	}

The eventual hope and intent is that this will improve process
startup times, as we'll need to do less work.

This is an initial effort, and not yet complete.

`jnimarshalmethod-gen.exe` is invoked from the new
`_GenerateJniMarshalMethods` target, which uses *xamarin-android*'s
mono, *not* a system mono or other managed runtime.  This is done so
that `$MONO_PATH` can be overridden, allowing "normal" use of
System.Reflection *by `JniValueMarshaler` instances* during *build*
time, *not* runtime.  The
`$(AndroidGenerateJniMarshalMethodsAdditionalArguments)` MSBuild
property can be used to add additional parameters to the
`jnimarshalmethod-gen.exe` invocation.  This is useful for debugging,
so that options such as `-v`, `-d`, or `--keeptemp` can be used.

Enable use of `jnimarshalmethod-gen.exe` for user assemblies and
`Mono.Android.dll` by setting `$(AndroidGenerateJniMarshalMethods)`
to True.

This is currently only supported on non-Windows platforms.

Additionally, remove the `NotImplementedException` throws from the
`Android.Runtime.AndroidValueManager`.  `AndroidValueManager` is now
used from the generated marshal methods, which handle primitive
arrays.

In order to make this whole process work, some additional custom
value marshalers are needed.  Add custom JNI value marshalers for
`Android.Graphics.Color` and `Android.Runtime.IJavaObject`.

Additional examples `jnimarshalmethod-gen.exe` output:

	using System;
	using Java.Interop;
	using Android.Runtime;

	public static void n_setAdapter_Landroid_widget_ListAdapter_ (IntPtr __jnienv, IntPtr __this, IntPtr value)
	{
	  JniTransition jniTransition = new JniTransition (__jnienv);
	  JniRuntime runtime = default(JniRuntime);
	  try {
	    runtime = JniEnvironment.Runtime;
	    JniRuntime.JniValueManager valueManager = runtime.ValueManager;
	    valueManager.WaitForGCBridgeProcessing ();
	    AbsListView value2 = valueManager.GetValue<AbsListView> (__this);
	    IListAdapter listAdapter2 = value2.Adapter = Java.Interop.JavaConvert.FromJniHandle<IListAdapter> (value, JniHandleOwnership.DoNotTransfer);
	  } catch (Exception ex) when (runtime.ExceptionShouldTransitionToJni (ex)) {
	    jniTransition.SetPendingException (ex);
	  } finally {
	    jniTransition.Dispose ();
	  }
	}
	
	public static IntPtr n_getAdapter (IntPtr __jnienv, IntPtr __this)
	{
	  JniTransition jniTransition = new JniTransition (__jnienv);
	  JniRuntime runtime = default(JniRuntime);
	  try {
	    runtime = JniEnvironment.Runtime;
	    JniRuntime.JniValueManager valueManager = runtime.ValueManager;
	    valueManager.WaitForGCBridgeProcessing ();
	    AbsListView value = valueManager.GetValue<AbsListView> (__this);
	    IListAdapter adapter = value.Adapter;
	    return JNIEnv.ToLocalJniHandle (adapter);
	  } catch (Exception ex) when (runtime.ExceptionShouldTransitionToJni (ex)) {
	    jniTransition.SetPendingException (ex);
	    return default(IntPtr);
	  } finally {
	    jniTransition.Dispose ();
	  }
	  IntPtr intPtr = default(IntPtr);
	  return intPtr;
	}

Profiling results of the Xamarin.Forms Integration Test running on
Pixel 2 XL phone:

Old marshaling:

	133        1         15 Android.Runtime.JNIEnv:RegisterJniNatives (intptr,int,intptr,intptr,int)
	288        2          1 Android.Runtime.JNIEnv:Initialize (Android.Runtime.JnienvInitializeArgs*)

New marshaling:

	 68        1         15 Android.Runtime.JNIEnv:RegisterJniNatives (intptr,int,intptr,intptr,int)
	264        2          1 Android.Runtime.JNIEnv:Initialize (Android.Runtime.JnienvInitializeArgs*)

Native member registration for a type is ~2x faster and
`JNIEnv.Initialize()` is ~20ms faster. 

Finally, `RunJavaInteropTests` was moved from
`@(_RunParallelTestTarget)` item group to the `@(_RunTestTarget)`
group, because it overwrites `Java.Runtime.Environment.dll.config`,
which is required by `jnimarshalmethod-gen.exe` to run.  We need to
run these tests after the `.apk` tests.  This should be fixed in the
future, hopefully by adding and/or fixing the Inputs/Outputs of the
relevant targets.

Author: radekdoulik

build-tools/scripts/JavaInteropDllConfigs.targets

@@ -0,0 +1,26 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <UsingTask AssemblyFile="$(MSBuildThisFileDirectory)..\..\bin\Build$(Configuration)\xa-prep-tasks.dll" TaskName="Xamarin.Android.BuildTools.PrepTasks.ReplaceFileContents" />
+  <Target Name="_CreateJavaInteropDllConfigs"
+      Inputs="$(XAInstallPrefix)xbuild\Xamarin\Android\Java.Interop.dll;$(JavaInteropSourceDirectory)\src\Java.Runtime.Environment\Java.Runtime.Environment.dll.config"
+      Outputs="$(XAInstallPrefix)xbuild\Xamarin\Android\Java.Interop.dll.config;$(XAInstallPrefix)xbuild\Xamarin\Android\Java.Runtime.Environment.dll.config">
+    <ReadLinesFromFile
+        File="$(MSBuildThisFileDirectory)java-interop.dllmap">
+      <Output TaskParameter="Lines" ItemName="_JavaInteropDllMapContent" />
+    </ReadLinesFromFile>
+    <WriteLinesToFile
+        File="$(XAInstallPrefix)xbuild\Xamarin\Android\Java.Interop.dll.config"
+        Lines="&lt;configuration&gt;;@(_JavaInteropDllMapContent);&lt;/configuration&gt;"
+        Overwrite="True"
+    />
+    <PropertyGroup>
+      <_DllMaps>@(_JavaInteropDllMapContent->'%(Identity)', '%0a  ')</_DllMaps>
+    </PropertyGroup>
+    <ReplaceFileContents
+        Condition="Exists('$(JavaInteropSourceDirectory)\src\Java.Runtime.Environment\Java.Runtime.Environment.dll.config')"
+        SourceFile="$(JavaInteropSourceDirectory)\src\Java.Runtime.Environment\Java.Runtime.Environment.dll.config"
+        DestinationFile="$(XAInstallPrefix)xbuild\Xamarin\Android\Java.Runtime.Environment.dll.config"
+        Replacements="&lt;configuration&gt;=&lt;configuration&gt;%0a  $(_DllMaps)"
+    />
+  </Target>
+</Project>

build-tools/scripts/RunTests.targets

@@ -147,10 +147,10 @@
   </Target>
   <ItemGroup>
     <_RunParallelTestTarget Include="RunNUnitTests" />
-    <_RunParallelTestTarget Include="RunJavaInteropTests" />
     <_RunParallelTestTarget Include="RunApkTests" />
   </ItemGroup>
   <ItemGroup>
+    <_RunTestTarget Include="RunJavaInteropTests" />
     <_RunTestTarget Include="RunPerformanceTests" />
   </ItemGroup>
   <Target Name="RunAllTests">

external/Java.Interop

@@ -1,12 +1,18 @@
 using System;
 using System.Collections.Generic;
 using System.Linq;
+using System.Linq.Expressions;
+using System.Reflection;
 using System.Text;
 
 using Android.Runtime;
 
+using Java.Interop;
+using Java.Interop.Expressions;
+
 namespace Android.Graphics
 {
+	[JniValueMarshaler (typeof (ColorValueMarshaler))]
 	public struct Color
 	{
 		private int color;
@@ -386,4 +392,50 @@ public static void RGBToHSV (int red, int green, int blue, float[] hsv)
 		public static Color YellowGreen { get { return new Color (0xFF9ACD32); } }
 		#endregion
 	}
+
+	public class ColorValueMarshaler : JniValueMarshaler<Color>
+	{
+		public override Type MarshalType {
+			get { return typeof (int); }
+		}
+
+		public override Color CreateGenericValue (ref JniObjectReference reference, JniObjectReferenceOptions options, Type targetType)
+		{
+			throw new NotImplementedException ();
+		}
+
+		public override JniValueMarshalerState CreateGenericObjectReferenceArgumentState (Color value, ParameterAttributes synchronize)
+		{
+			throw new NotImplementedException ();
+		}
+
+		public override void DestroyGenericArgumentState (Color value, ref JniValueMarshalerState state, ParameterAttributes synchronize)
+		{
+			throw new NotImplementedException ();
+		}
+
+		public override Expression CreateParameterToManagedExpression (JniValueMarshalerContext context, ParameterExpression sourceValue, ParameterAttributes synchronize, Type targetType)
+		{
+			var c = typeof (Color).GetConstructor (new[]{typeof (int)});
+			var v = Expression.Variable (typeof (Color), sourceValue.Name + "_val");
+			context.LocalVariables.Add (v);
+			context.CreationStatements.Add (Expression.Assign (v, Expression.New (c, sourceValue)));
+
+			return v;
+		}
+
+		public override Expression CreateParameterFromManagedExpression (JniValueMarshalerContext context, ParameterExpression sourceValue, ParameterAttributes synchronize)
+		{
+			var r = Expression.Variable (MarshalType, sourceValue.Name + "_p");
+			context.LocalVariables.Add (r);
+			context.CreationStatements.Add (Expression.Assign (r, Expression.Field (sourceValue, "color")));
+
+			return r;
+		}
+
+		public override Expression CreateReturnValueFromManagedExpression (JniValueMarshalerContext context, ParameterExpression sourceValue)
+		{
+			return CreateParameterFromManagedExpression (context, sourceValue, 0);
+		}
+	}
 }

src/Mono.Android/Android.Graphics/Color.cs

@@ -363,12 +363,10 @@ public override void WaitForGCBridgeProcessing ()
 
 		public override void AddPeer (IJavaPeerable value)
 		{
-			throw new NotImplementedException ();
 		}
 
 		public override void RemovePeer (IJavaPeerable value)
 		{
-			throw new NotImplementedException ();
 		}
 
 		public override IJavaPeerable PeekPeer (JniObjectReference reference)
@@ -378,17 +376,15 @@ public override IJavaPeerable PeekPeer (JniObjectReference reference)
 
 		public override void CollectPeers ()
 		{
-			throw new NotImplementedException ();
 		}
 
 		public override void FinalizePeer (IJavaPeerable value)
 		{
-			throw new NotImplementedException ();
 		}
 
 		public override List<JniSurfacedPeerInfo> GetSurfacedPeers ()
 		{
-			throw new NotImplementedException ();
+			return null;
 		}
 	}
 }

src/Mono.Android/Android.Runtime/AndroidRuntime.cs

@@ -2,6 +2,7 @@
 
 namespace Android.Runtime {
 
+	[Java.Interop.JniValueMarshaler (typeof (IJavaObjectValueMarshaler))]
 	public interface IJavaObject : IDisposable {
 		IntPtr Handle { get; }
 	}

src/Mono.Android/Android.Runtime/IJavaObject.cs

@@ -0,0 +1,54 @@
+using System;
+using System.Linq;
+using System.Linq.Expressions;
+using System.Reflection;
+
+using Java.Interop;
+using Java.Interop.Expressions;
+
+namespace Android.Runtime
+{
+	sealed class IJavaObjectValueMarshaler : JniValueMarshaler<IJavaObject> {
+
+		internal    static  IJavaObjectValueMarshaler              Instance    = new IJavaObjectValueMarshaler ();
+
+		public override IJavaObject CreateGenericValue (ref JniObjectReference reference, JniObjectReferenceOptions options, Type targetType)
+		{
+			throw new NotImplementedException ();
+		}
+
+		public override JniValueMarshalerState CreateGenericObjectReferenceArgumentState (IJavaObject value, ParameterAttributes synchronize)
+		{
+			throw new NotImplementedException ();
+		}
+
+		public override void DestroyGenericArgumentState (IJavaObject value, ref JniValueMarshalerState state, ParameterAttributes synchronize)
+		{
+			throw new NotImplementedException ();
+		}
+
+		public override Expression CreateReturnValueFromManagedExpression (JniValueMarshalerContext context, ParameterExpression sourceValue)
+		{
+			return Expression.Call (
+				typeof (JNIEnv),
+				"ToLocalJniHandle",
+				null,
+				sourceValue);
+		}
+
+		public override Expression CreateParameterToManagedExpression (JniValueMarshalerContext context, ParameterExpression sourceValue, ParameterAttributes synchronize, Type targetType)
+		{
+			var r   = Expression.Variable (targetType, sourceValue.Name + "_val");
+			context.LocalVariables.Add (r);
+			context.CreationStatements.Add (
+					Expression.Assign (r,
+						Expression.Call (
+					                typeof (JavaConvert),
+					                "FromJniHandle",
+							new[]{targetType},
+					                sourceValue,
+					                Expression.Field (null, typeof (JniHandleOwnership), "DoNotTransfer"))));
+			return r;
+		}
+	}
+}

src/Mono.Android/Android.Runtime/IJavaObjectValueMarshaler.cs

@@ -83,6 +83,7 @@
       <Link>JavaNativeTypeManager.cs</Link>
     </Compile>
     <Compile Include="Android.Runtime\DynamicMethodNameCounter.cs" />
+    <Compile Include="Android.Runtime\IJavaObjectValueMarshaler.cs" />
   </ItemGroup>
   <Import Project="..\Xamarin.Android.NamingCustomAttributes\Xamarin.Android.NamingCustomAttributes.projitems" Label="Shared" Condition="Exists('..\Xamarin.Android.NamingCustomAttributes\Xamarin.Android.NamingCustomAttributes.projitems')" />
   <Import Project="$(MSBuildBinPath)\Microsoft.CSharp.targets" />

src/Mono.Android/Mono.Android.csproj

@@ -43,6 +43,7 @@
     <ConsolePause>false</ConsolePause>
     <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
     <AndroidLinkTool Condition=" '$(AndroidLinkTool)' == '' ">r8</AndroidLinkTool>
+    <AndroidGenerateJniMarshalMethods>True</AndroidGenerateJniMarshalMethods>
   </PropertyGroup>
   <ItemGroup>
     <Reference Include="System" />

src/Mono.Android/Test/Mono.Android-Tests.csproj

@@ -8,6 +8,7 @@
   </ItemGroup>
   <Import Project="Mono.Android-Tests.projitems" />
   <Import Project="..\..\..\build-tools\scripts\TestApks.targets" />
+  <Import Project="..\..\..\build-tools\scripts\JavaInteropDllConfigs.targets" />
   <Target Name="BuildNativeLibs"
       BeforeTargets="Build"
       DependsOnTargets="AndroidPrepareForBuild"
@@ -16,4 +17,8 @@
     <Error Text="Could not locate Android NDK." Condition="!Exists ('$(NdkBuildPath)')" />
     <Exec Command="&quot;$(NdkBuildPath)&quot;" />
   </Target>
+  <Target Name="EnsureJavaInteropDllConfigs"
+      BeforeTargets="_GenerateJniMarshalMethods"
+      DependsOnTargets="_CreateJavaInteropDllConfigs">
+  </Target>
 </Project>