[mono] new Sdk that selects mono runtime components (#54432)

* Work in progress: new Sdk that selects mono runtime components

* Add props and targets description to the components doc

* condition the _MonoRuntimeAvailableComponents by RuntimeIdentifier

* [cmake] Write a component-manifest.props file during build

If we're not building a mono aot cross compiler, generate a
component-manifest.props file in artifacts/obj/mono/<host>/ that indicates if
the host will use static or dynamic components, and a list of the available
components, and the properties for constructing their names.

* Build Microsoft.NETCore.App.Runtime.Mono.<RID>.Sdk shared framework nuget

It seems to also generate a symbols nuget.  And in the nuget there's a
tools/mono-sdk-what-is-this.deps.json file from the
SharedFrameworkHostFileNameOverride property.  It would be nice to exclude that
stuff.

* put the compoonent-manifest.targets into the Sdk

* delete WIP in mono/nuget/

* fixup static component names in component-manifest.targets

* delete fixed fixme

* add missing $

* fix whitespace

* [cmake] switch to configure_file instead of file(CONFIGURE)

* add missing trailing slashes in .props.in file

* Add new Sdk packs to the workload manifest

* rework component-manifest.targets to use ItemGroups; move to new SDK

* Rename shared framework to Microsoft.NETCore.App.Runtime.Mono.<RID>.Props.Sdk

And only include component-manifest.props, not the targets

* Update manifest to include the new Props.Sdk and MonoTargets.Sdk

* Move RuntimeConfigParserTask into Microsoft.NET.Runtime.MonoTargets.Sdk

Consolidate all platform-independent tasks and targets into a single Sdk

* Add iossimulator-x86 props

* update components design doc

* Fix typo

* improve docs

* Add _MonoRuntimeComponentDontLink target output

* Drop component-manifest.props into runtime pack build/ directory

Remove from the Microsoft.NETCore.App.Mono.Props.Sdk workload nuget

* remove Microsoft.NETCore.App.Mono.Props.Sdk

* Import component-manifest.props from the runtime pack

Move the MonoTargets.Sdk import to each target platform that we support

* Fix typos

Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>

* Apply suggestions from code review

Co-authored-by: Ankit Jain <radical@gmail.com>

* Add JsonToItemsTaskFactory

* fix whitespace

* Do some validation earlier in _MonoComputeAvailableComponentDefinitions

* Read component-manifest.json using the JsonToItemsTaskFactory

and bundle it in Microsoft.NET.Runtime.MonoTargets.Sdk

* remove ResolvedRuntimePack import from WorkloadManifest.targets

it's too early, and we have the JsonToItemsTaskFactory now to read the manifest

* Generate component-manifest.json in CMakeLists.txt

* Fix some copy-paste nits

* Use RuntimeFlavor==mono for runtime pack build directory

instead of TargetsMobile.  We want the build files (mono-components.json)
in every mono runtime pack, not just on mobile targets

* Apply suggestions from code review

Co-authored-by: Ankit Jain <radical@gmail.com>

* rename component-manifest to RuntimeComponentManifest

* fixup nullability annotations

* fix whitespace

* fix formatting

* Misc fixes to JsonToItemsTaskFactory

* Rename MonoRuntimeComponentManifestReadTask

from MonoRuntimeComponentsReadManifestTask

* undo nullability annotation

Build doesn't like it for some reason (probably net472)

* fix incorrect task parameter name

* Remove Identity metadata from dictionary at json parsing time

Also improve nullability a bit by making the properties immutable

* Throw correct json deserializer exceptions

* Catch JsonException in an async function

We get nice error messages now like

```
src/mono/nuget/Microsoft.NET.Runtime.MonoTargets.Sdk/Sdk/RuntimeComponentManifest.targets(8,5): error : Failed to deserialize json from file 'artifacts/bin/mono/iOSSimulator.x64.Release/build/RuntimeComponentManifest.json', JSON Path: $.items._MonoRuntimeAvailableComponents[2], Line: 14, Position: 1 [component-manifest.sample.proj]
src/mono/nuget/Microsoft.NET.Runtime.MonoTargets.Sdk/Sdk/RuntimeComponentManifest.targets(8,5): error : JsonException: The JSON value could not be converted to System.Collections.Generic.Dictionary`2[System.String,System.String]. Path: $.identity | LineNumber: 0 | BytePositionInLine: 16. Path: $.items._MonoRuntimeAvailableComponents[2] | LineNumber: 14 | BytePositionInLine: 1. [component-manifest.sample.proj]
src/mono/nuget/Microsoft.NET.Runtime.MonoTargets.Sdk/Sdk/RuntimeComponentManifest.targets(8,5): error : InvalidOperationException: Cannot get the value of a token type 'Number' as a string. [component-manifest.sample.proj]
src/mono/nuget/Microsoft.NET.Runtime.MonoTargets.Sdk/Sdk/RuntimeComponentManifest.targets(8,5):
error :
[component-manifest.sample.proj]
```

* fixup comments

Co-authored-by: Rolf Bjarne Kvinge <rolf@xamarin.com>
Co-authored-by: Ankit Jain <radical@gmail.com>

Author: 3 people

docs/design/mono/components.md

@@ -164,10 +164,10 @@ To implement `feature_X` as a component.  Carry out the following steps:
 	  { feature_X_cleanup },
 	  feature_X_hello,
     };
-	
+
 	MonoComponentFeatureX *
 	mono_component_feature_X_init (void) { return &fn_table; }
-	
+
 	void feature_X_cleanup (MonoComponent *self)
 	{
 	  static int cleaned = 0;
@@ -207,10 +207,10 @@ To implement `feature_X` as a component.  Carry out the following steps:
 	  { feature_X_cleanup },
 	  feature_X_hello,
     };
-	
+
 	MonoComponentFeatureX *
 	mono_component_feature_X_init (void) { return &fn_table; }
-	
+
 	void feature_X_cleanup (MonoComponent *self)
 	{
 	  static int cleaned = 0;
@@ -229,7 +229,7 @@ To implement `feature_X` as a component.  Carry out the following steps:
   ```c
     MonoComponentFeatureX*
 	mono_component_feature_X (void);
-	
+
     ...
     MonoComponentFeatureX*
 	mono_component_feature_X_stub_init (void);
@@ -238,7 +238,7 @@ To implement `feature_X` as a component.  Carry out the following steps:
 * Add an entry to the `components` list to load the component to `mono/metadata/components.c`, and also implement the getter for the component:
   ```c
     static MonoComponentFeatureX *feature_X = NULL;
-	
+
     MonoComponentEntry components[] = {
 	   ...
 	   {"feature_X", "feature_X", COMPONENT_INIT_FUNC (feature_X), (MonoComponent**)&feature_X, NULL },
@@ -265,16 +265,66 @@ To implement `feature_X` as a component.  Carry out the following steps:
 ## Detailed design - Packaging and runtime packs
 
 The components are building blocks to put together a functional runtime.  The
-runtime pack includes the base runtime and the components and additional
-properties and targets that enable the workload to construct a runtime for
-various scenarios.
-
-In each runtime pack we include:
-
-- The compiled compnents for the apropriate host architectures in a well-known subdirectory
-- An MSBuild props file that defines an item group that list each component name and has metadata that indicates:
-   - the path to the component in the runtime pack
-   - the path to the stub component in the runtime pack (if components are static)
-- An MSBuild targets file that defines targets to copy a specified set of components to the app publish folder (if components are dynamic); or to link the runtime together with stubs and a set of enabled components (if components are static)
-
-** TODO ** Write this up in more detail
+runtime pack includes the base runtime and the components.  The mono workload
+includes the runtime pack and additional tasks, properties and targets that
+enable the workload to construct a runtime for various scenarios.
+
+For the target RID, we expose:
+
+- `@(_MonoRuntimeComponentLinking)` set to either `'static'` or `'dynamic'` depending on whether the
+  current runtime pack for the current target includes runtime components as static archives or as
+  shared libraries, respectively.
+- `@(_MonoRuntimeComponentSharedLibExt)` and `@(_MonoRuntimeComponentStaticLibExt)` set to the file
+  extension of the runtime components for the current target (ie, `'.a', '.so', '.dylib'` etc).
+- `@(_MonoRuntimeAvailableComponents)` a list of component names without the `lib` prefix (if any)
+  or file extensions.  For example: `'hot_reload; diagnostics_tracing'`.
+
+Each of the above item lists has `RuntimeIdentifier` metadata.  For technical reasons the mono
+workload will provide a single `@(_MonoRuntimeAvailableComponent)` item list for all platforms.  We
+use the `RuntimeIdentifier` metadata to filter out the details applicable for the current platform.
+
+- The target `_MonoSelectRuntimeComponents` that has the following inputs and outputs:
+  - input `@(_MonoComponent)` (to be set by the workload) : a list of components that a workload wants to use for the current
+    app.  It is an error if this specifies any unknown component name.
+  - output `@(_MonoRuntimeSelectedComponents)` and `@(_MonoRuntimeSelectedStubComponents)` The names
+    of the components that were (resp, were not) selected.  For example `'hot_reload;
+    diagnostics_tracing'`.  Each item has two metadata properties `ComponentLib` and
+    `ComponentStubLib` (which may be empty) that specify the name of the static or dynamic library
+    of the component.  This is not the main output of the target, it's primarily for debugging.
+  - output `@(_MonoRuntimeComponentLink)` a list of library names (relative to the `native/`
+    subdirectory of the runtime pack) that (for dynamic components) must be placed next to the
+    runtime in the application bundle, or (for static components) that must be linked with the
+    runtime to enable the components' functionality.  Each item in the list has metadata
+    `ComponentName` (e.g. `'hot_reload'`), `IsStub` (`true` or `false`), `Linking` (`'static'` or
+    `'dynamic'`).  This output should be used by the workloads when linking the app and runtime if
+    the workload uses an allow list of native libraries to link or bundle.
+  - output `@(_MonoRuntimeComponentDontLink)` a list of library names (relative to the `native/`
+    subdirectory of the runtime pack) that should be excluded from the application bundle (for
+    dynamic linking) or that should not be passed to the native linker (for static linking).  This
+    output should be used by workloads that just link or bundle every native library from `native/`
+    in order to filter the contents of the subdirectory to exclude the disabled components (and to
+    exclude the static library stubs for the enabled components when static linking).
+
+Generally workloads should only use one of `@(_MonoRuntimeComponentLink)` or
+`@(_MonoRuntimeComponentDontLink)`, depending on whether they use an allow or block list for the
+contents of the `native/` subdirectory.
+
+Example fragment (assuming the mono workload has been imported):
+
+```xml
+  <Project>
+    <ItemGroup Condition="'$(Configuration)' == 'Debug'">
+      <_MonoComponent Include="hot_reload;diagnostics_tracing" />
+    </ItemGroup>
+
+    <Target Name="PrintComponents" DependsOnTargets="_MonoSelectRuntimeComponents">
+      <Message Importance="High" Text="Runtime identifier: $(RuntimeIdentifier)" />
+      <Message Importance="High" Text="Selected : @(_MonoRuntimeSelectedComponents) %(ComponentLib)" />
+      <Message Importance="High" Text="Stubbed out : @(_MonoRuntimeSelectedStubComponents) %(ComponentStubLib)" />
+      <Message Importance="High" Text="Linking with lib @(_MonoRuntimeComponentLink) Stub: %(IsStub) Linking: %(Linking) Component: %(ComponentName)"/>
+
+      <Message Importance="High" Text="UnSelected : @(_MonoRuntimeUnSelectedComponents) %(ComponentLib)" />
+      <Message Importance="High" Text="Exclude these from linking: @(_MonoRuntimeComponentDontLink) Stub: %(IsStub) Linking: %(Linking) Component: %(ComponentName)" />
+    </Target>
+  </Project>
+```

src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.props

@@ -70,6 +70,12 @@
         <TargetPath>runtimes/$(RuntimeIdentifier)/native/include/%(RecursiveDir)</TargetPath>
       </RuntimeFiles>
 
+      <RuntimeFiles Condition="'$(RuntimeFlavor)' == 'mono'"
+                    Include="$(MonoArtifactsPath)\build\**\*.*"
+                    ExcludeFromDataFiles="true">
+        <TargetPath>runtimes/$(RuntimeIdentifier)/build/%(RecursiveDir)</TargetPath>
+      </RuntimeFiles>
+
       <CoreCLRCrossTargetFiles PackOnly="true" />
       <CoreCLRCrossTargetFiles Condition="'%(FileName)' == 'clrjit' or '%(FileName)' == 'libclrjit'">
         <TargetPath>runtimes/$(CoreCLRCrossTargetComponentDirName)_$(TargetArchitecture)/native</TargetPath>

src/mono/mono.proj

@@ -431,6 +431,9 @@
     <ItemGroup Condition="'$(MonoComponentsStatic)' == 'true'">
       <_MonoCMakeArgs Include="-DSTATIC_COMPONENTS=1" />
     </ItemGroup>
+    <ItemGroup>
+      <_MonoCMakeArgs Include="-DMONO_COMPONENTS_RID=$(TargetOS)-$(TargetArchitecture)" />
+    </ItemGroup>
 
     <PropertyGroup>
       <_MonoCFLAGSOption>-DCMAKE_C_FLAGS="@(_MonoCPPFLAGS, ' ') @(_MonoCFLAGS, ' ')"</_MonoCFLAGSOption>
@@ -770,6 +773,7 @@
         <Destination>$(RuntimeBinDir)cross\$(PackageRID)\opt$(ExeExt)</Destination>
       </_MonoRuntimeArtifacts>
       <_MonoIncludeArtifacts Include="$(MonoObjDir)out\include\**" />
+      <_MonoRuntimeBuildArtifacts Include="$(MonoObjDir)\build\**" />
       <_MonoRuntimeArtifacts Condition="'$(_MonoIncludeInterpStaticFiles)' == 'true'" Include="$(MonoObjDir)out\lib\libmono-ee-interp.a">
         <Destination>$(RuntimeBinDir)libmono-ee-interp.a</Destination>
       </_MonoRuntimeArtifacts>
@@ -816,6 +820,11 @@
           SkipUnchangedFiles="true"
           Condition="'$(MonoGenerateOffsetsOSGroups)' == '' and ('$(TargetsMacCatalyst)' == 'true' or '$(TargetsiOS)' == 'true' or '$(TargetstvOS)' == 'true' or '$(TargetsAndroid)' == 'true' or '$(TargetsBrowser)' == 'true')"/>
 
+    <Copy SourceFiles="@(_MonoRuntimeBuildArtifacts)"
+          DestinationFiles="@(_MonoRuntimeBuildArtifacts->'$(RuntimeBinDir)build\%(RecursiveDir)%(Filename)%(Extension)')"
+          SkipUnchangedFiles="true"
+          Condition="'$(BuildMonoAOTCrossCompilerOnly)' != 'true'" />
+
     <Exec Condition="'$(BuildMonoAOTCrossCompilerOnly)' != 'true' and '$(MonoGenerateOffsetsOSGroups)' == '' and ('$(TargetsOSX)' == 'true' or '$(TargetsMacCatalyst)' == 'true' or '$(TargetsiOS)' == 'true' or '$(TargetstvOS)' == 'true')" Command="install_name_tool -id @rpath/$(MonoFileName) $(RuntimeBinDir)$(MonoFileName)" />
   </Target>
 

src/mono/mono/component/CMakeLists.txt

@@ -82,6 +82,7 @@ set(${MONO_DIAGNOSTICS_TRACING_COMPONENT_NAME}-dependencies
 
 #define a library for each component and component stub
 function(define_component_libs)
+  # NOTE: keep library naming pattern in sync with RuntimeComponentManifest.targets
   if (NOT DISABLE_LIBS)
     foreach(component IN LISTS components)
       add_library("mono-component-${component}-static" STATIC $<TARGET_OBJECTS:${component}-objects>)
@@ -121,6 +122,7 @@ endforeach()
 if(NOT DISABLE_COMPONENTS AND NOT STATIC_COMPONENTS)
   # define a shared library for each component
   foreach(component IN LISTS components)
+    # NOTE: keep library naming pattern in sync with RuntimeComponentManifest.targets
     if(HOST_WIN32)
       add_library("mono-component-${component}" SHARED "${${component}-sources}")
       target_compile_definitions("mono-component-${component}" PRIVATE -DCOMPILING_COMPONENT_DYNAMIC;-DMONO_DLL_IMPORT)
@@ -172,6 +174,26 @@ foreach(component IN LISTS components)
   list(APPEND mono-components-stub-objects $<TARGET_OBJECTS:${component}-stub-objects>)
 endforeach()
 
+if(NOT MONO_CROSS_COMPILE)
+  set(TemplateMonoRuntimeComponentSharedLibExt "${CMAKE_SHARED_LIBRARY_SUFFIX}")
+  set(TemplateMonoRuntimeComponentStaticLibExt "${CMAKE_STATIC_LIBRARY_SUFFIX}")
+  set(TemplateRuntimeIdentifier "${MONO_COMPONENTS_RID}")
+  if(DISABLE_COMPONENTS)
+    set(TemplateMonoRuntimeComponentLinking "static")
+    set(TemplateMonoRuntimeAvailableComponents "")
+  else()
+    list(TRANSFORM components REPLACE "^(.+)$" "{ \"identity\": \"\\1\", \"RuntimeIdentifier\": \"${TemplateRuntimeIdentifier}\" }," OUTPUT_VARIABLE TemplateMonoRuntimeAvailableComponentsList)
+    list(JOIN TemplateMonoRuntimeAvailableComponentsList "\n" TemplateMonoRuntimeAvailableComponents)
+    if(STATIC_COMPONENTS)
+      set(TemplateMonoRuntimeComponentLinking "static")
+    else()
+      set(TemplateMonoRuntimeComponentLinking "dynamic")
+    endif()
+  endif()
+  # Write a RuntimeComponentManifest.json file in the artifacts/obj/mono/<host>/build/ directory
+  # without the ../.. the file would go in artifacts/obj/mono/<host>/mono/mini
+  configure_file( "${MONO_COMPONENT_PATH}/RuntimeComponentManifest.json.in" "../../build/RuntimeComponentManifest.json")
+endif()
 
 # component tests
 set(MONO_EVENTPIPE_TEST_SOURCE_PATH "${MONO_EVENTPIPE_SHIM_SOURCE_PATH}/test")

src/mono/mono/component/RuntimeComponentManifest.json.in

@@ -0,0 +1,16 @@
+{
+    "items": {
+        "_MonoRuntimeComponentLinking": [
+            { "identity": "${TemplateMonoRuntimeComponentLinking}", "RuntimeIdentifier": "${TemplateRuntimeIdentifier}" },
+        ],
+        "_MonoRuntimeComponentSharedLibExt": [
+            { "identity": "${TemplateMonoRuntimeComponentSharedLibExt}", "RuntimeIdentifier": "${TemplateRuntimeIdentifier}" },
+        ],
+        "_MonoRuntimeComponentStaticLibExt": [
+            { "identity": "${TemplateMonoRuntimeComponentStaticLibExt}", "RuntimeIdentifier": "${TemplateRuntimeIdentifier}" },
+        ],
+        "_MonoRuntimeAvailableComponents": [
+            ${TemplateMonoRuntimeAvailableComponents}
+        ],
+    }
+}

src/mono/nuget/Microsoft.NET.Runtime.MonoTargets.Sdk/Microsoft.NET.Runtime.MonoTargets.Sdk.pkgproj

@@ -0,0 +1,23 @@
+<Project DefaultTargets="Build">
+  <Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.props))" />
+
+  <PropertyGroup>
+    <PackageDescription>Provides the tasks+targets, for consumption by mono-based workloads</PackageDescription>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <ProjectReference Include="$(RepoTasksDir)RuntimeConfigParser\RuntimeConfigParser.csproj" />
+    <ProjectReference Include="$(RepoTasksDir)JsonToItemsTaskFactory\JsonToItemsTaskFactory.csproj" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <PackageFile Include="Sdk\Sdk.props" TargetPath="Sdk" />
+    <PackageFile Include="Sdk\Sdk.targets" TargetPath="Sdk" />
+    <PackageFile Include="build\$(MSBuildProjectName).props" TargetPath="build" />
+    <PackageFile Include="Sdk\RuntimeConfigParserTask.props" TargetPath="Sdk" />
+    <PackageFile Include="Sdk\RuntimeComponentManifest.props" TargetPath="Sdk" />
+    <PackageFile Include="Sdk\RuntimeComponentManifest.targets" TargetPath="Sdk" />
+  </ItemGroup>
+
+  <Import Project="$([MSBuild]::GetPathOfFileAbove(Directory.Build.targets))" />
+</Project>

src/mono/nuget/Microsoft.NET.Runtime.MonoTargets.Sdk/README.md

@@ -0,0 +1,38 @@
+# Mono Runtime Host support targets
+
+This Sdk provides additional tasks and targets for workloads hosting the MonoVM .NET runtime.
+
+## component-manifest.targets
+
+See https://github.com/dotnet/runtime/blob/main/docs/design/mono/components.md
+
+## RuntimeConfigParserTask
+The `RuntimeConfigParserTask` task converts a json `runtimeconfig.json` to a binary blob for MonoVM's `monovm_runtimeconfig_initialize` API.
+To use the task in a project, reference the NuGet package, with the appropriate nuget source.
+
+### NuGet.config
+```xml
+<?xml version="1.0" encoding="utf-8"?>
+<configuration>
+  <packageSources>
+    <add key="dotnet6" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet6/nuget/v3/index.json" />
+  </packageSources>
+</configuration>
+```
+
+### In the project file
+```xml
+<!-- Import the NuGet package into the project -->
+<ItemGroup>
+    <PackageReference Include="Microsoft.NET.Runtime.MonoTargets.Sdk" Version="<desired-dotnet-6-sdk-version>" />
+</ItemGroup>
+
+<!-- Use the RuntimeConfigParser task in a target -->
+<Target>
+    <RuntimeConfigParserTask
+        RuntimeConfigFile="$(Path_to_runtimeconfig.json_file)"
+        OutputFile="$(Path_to_generated_binary_file)"
+        RuntimeConfigReservedProperties="@(runtime_properties_reserved_by_host)">
+    </RuntimeConfigParserTask>
+</Target>
+```

src/mono/nuget/Microsoft.NET.Runtime.MonoTargets.Sdk/Sdk/RuntimeComponentManifest.props

@@ -0,0 +1,14 @@
+<Project>
+  <PropertyGroup>
+    <JsonToItemsTaskFactoryTasksAssemblyPath Condition="'$(MSBuildRuntimeType)' == 'Core'">$(MSBuildThisFileDirectory)..\tasks\net6.0\JsonToItemsTaskFactory.dll</JsonToItemsTaskFactoryTasksAssemblyPath>
+    <JsonToItemsTaskFactoryTasksAssemblyPath Condition="'$(MSBuildRuntimeType)' != 'Core'">$(MSBuildThisFileDirectory)..\tasks\net472\JsonToItemsTaskFactory.dll</JsonToItemsTaskFactoryTasksAssemblyPath>
+  </PropertyGroup>
+  <UsingTask TaskName="MonoRuntimeComponentManifestReadTask" TaskFactory="JsonToItemsTaskFactory.JsonToItemsTaskFactory" AssemblyFile="$(JsonToItemsTaskFactoryTasksAssemblyPath)">
+    <ParameterGroup>
+      <_MonoRuntimeComponentSharedLibExt ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="false" Output="true" />
+      <_MonoRuntimeComponentStaticLibExt ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="false" Output="true" />
+      <_MonoRuntimeComponentLinking ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="false" Output="true" />
+      <_MonoRuntimeAvailableComponents ParameterType="Microsoft.Build.Framework.ITaskItem[]" Required="false" Output="true" />
+    </ParameterGroup>
+  </UsingTask>
+</Project>