Add support for emitting Mach-O R2R images

docs/design/coreclr/botr/readytorun-format.md

@@ -135,6 +135,7 @@ struct READYTORUN_CORE_HEADER
 | READYTORUN_FLAG_COMPONENT                  | 0x00000020 | This is a component assembly of a composite R2R image
 | READYTORUN_FLAG_MULTIMODULE_VERSION_BUBBLE | 0x00000040 | This R2R module has multiple modules within its version bubble (For versions before version 6.3, all modules are assumed to possibly have this characteristic)
 | READYTORUN_FLAG_UNRELATED_R2R_CODE         | 0x00000080 | This R2R module has code in it that would not be naturally encoded into this module
+| READYTORUN_FLAG_PLATFORM_NATIVE_IMAGE      | 0x00000100 | The owning composite executable is in the platform native format
 
 ## READYTORUN_SECTION
 

docs/design/coreclr/botr/readytorun-platform-native-envelope.md

@@ -22,6 +22,44 @@ Mach‑O support will only be supported for composite ReadyToRun when the target
 
 `crossgen2` does not produce the final shared library. A separate SDK / build linking step must preserve the `RTR_HEADER` export in the final `dylib`.
 
+### Mach-O Emitter Decisions
+
+There's a few cases in the R2R format that are not natively represented in the Mach-O format that have to be emulated. This section will describe some of the design decisions for the Mach-O R2R format.
+
+#### Sections
+
+Sections folded into `__TEXT,__text` that is in other sections in the PE envelope:
+
+- CLR metadata: In the PE format, put in .cormeta, corresponds to the PE Header's "COR Header directory"
+- Win32 Resources: In the PE format, put in .rsrc, corresponds to the PE Header's "Win32 Resources Header directory"
+- Managed Unwind Info: In the Mach-O format, this section is expected to be in the Mach-O unwind format. The unwind info used by the runtime must be in the Windows unwind format.
+- GC Info: Entries correspond to the unwind info.
+
+Data moved out of `__TEXT,__text`:
+
+- Precompiled managed code has been moved into `__TEXT,__managedcode`. `__TEXT,__text` gets special treatment by the linker and `__TEXT,__managedcode` matches NativeAOT.
+
+Data that stays in the corresponding locations as the PE envelope:
+
+- Read-only data such as jump tables and the R2R headers: `__TEXT,__const`
+- Read-write data, such as fixup tables: `__DATA,__data`
+- Import thunks: `__TEXT,__text`
+
+#### Relocations
+
+Symbol ranges are represented differently in Mach-O than other platforms. Apple linkers have issues when multiple symbols are defined at the same location. Additionally, the Mach format natively supports a "subtractor" reloc to represent the distance between two symbols. As a result, we can represent the start of the symbol range as the start symbol of the range. We can represent the size of the range we can represent as "end symbol location - start symbol location + end symbol size".
+
+#### Base Symbol and RVAs
+
+The R2R format, like the PE format, is heavily based around having RVAs emitted into the image that can be added to the base symbol of the image. The COFF object file format natively supports such a concept, and the PE format uses such a concept in the PE header. However, other formats do not natively support such a concept.
+
+The Apple linker does provide a base symbol for the Mach format, but the base symbol depends on the output type, generally in the form `__mh_<output>_header`. For dylibs, the symbol is `__mh_dylib_header`. This symbol is located at the address returned by `dlinfo` and `dladdr` for the base address. It also points to the Mach header, which can be used to find the size of the image to bound reads of the R2R data.
+
+As a result, we can emulate this support in the Mach format with ease:
+
+1. The base symbol that we use in the object writer will be `__mh_dylib_header`.
+2. To emit the distance from the base symbol, we will use a subtractor relocation to represent "symbol location - `__mh_dylib_header` location".
+
 ## Runtime: consuming a platform-native R2R image
 
 The runtime will be updated to handle platform-native R2R images during assembly load.

docs/project/library-servicing.md

@@ -7,7 +7,7 @@ Servicing branches represent shipped versions of .NET. The branch naming convent
 - **For .NET 8 and .NET 9**: Branch names use the format `release/X.0-staging`. Examples:
   - `release/9.0-staging`
   - `release/8.0-staging`
-  
+
 - **For .NET 10+**: Branch names use the format `release/X.0` (no `-staging` suffix). Examples:
   - `release/10.0`
   - `release/11.0`
@@ -36,7 +36,7 @@ All the servicing change must go through an approval process. You have two ways
   For .NET 8 and .NET 9 (use `-staging` suffix):
   - `/backport to release/9.0-staging`
   - `/backport to release/8.0-staging`
-  
+
   For .NET 10+ (no `-staging` suffix):
   - `/backport to release/10.0`
   - `/backport to release/11.0`
@@ -55,4 +55,4 @@ For all cases, you must:
 
 The area owner can then merge the PR once the CI looks good (it's either green or the failures are investigated and determined to be unrelated to the PR).
 
-**Note**: Applying the `Servicing-approved` label ensures the `check-service-labels` CI job passes, which is a mandatory requirement for merging a PR in a servicing branch.
+**Note**: Applying the `Servicing-approved` label ensures the `check-service-labels` CI job passes, which is a mandatory requirement for merging a PR in a servicing branch.

src/coreclr/crossgen-corelib.proj

@@ -17,32 +17,61 @@
   </PropertyGroup>
 
   <PropertyGroup>
-    <BuildDll>true</BuildDll>
-    <BuildDll Condition="'$(TargetOS)' == 'netbsd' or '$(TargetOS)' == 'illumos' or '$(TargetOS)' == 'solaris' or '$(TargetOS)' == 'haiku'">false</BuildDll>
+    <CrossgenTargetName>InvokeCrossgen2</CrossgenTargetName>
+
+    <PublishReadyToRun>true</PublishReadyToRun>
+    <!-- Disable crossgen on NetBSD, illumos, Solaris and Haiku for now. This can be revisited when we have full support. -->
+    <PublishReadyToRun Condition="'$(TargetOS)' == 'netbsd' or '$(TargetOS)' == 'illumos' or '$(TargetOS)' == 'solaris' or '$(TargetOS)' == 'haiku'">false</PublishReadyToRun>
+    <!-- TODO-WASM: we will have WASM R2R after https://github.com/dotnet/runtime/issues/121257 -->
+    <PublishReadyToRun Condition="'$(TargetOS)' == 'browser' or '$(TargetOS)' == 'wasi'">false</PublishReadyToRun>
+
+    <CrossgenTargetName Condition="!$(PublishReadyToRun)">CopyILCoreLib</CrossgenTargetName>
 
     <BuildPdb>false</BuildPdb>
-    <BuildPdb Condition="$(BuildDll) and '$(OS)' == 'Windows_NT' and '$(TargetOS)' == 'windows'">true</BuildPdb>
+    <BuildPdb Condition="'$(OS)' == 'Windows_NT' and '$(TargetOS)' == 'windows'">true</BuildPdb>
 
     <BuildPerfMap>false</BuildPerfMap>
-    <BuildPerfMap Condition="$(BuildDll) and '$(TargetOS)' == 'linux'">true</BuildPerfMap>
+    <BuildPerfMap Condition="'$(TargetOS)' == 'linux'">true</BuildPerfMap>
 
     <_MergeMibcFilesCacheFile>$(MibcOptimizationDataDir)/$(TargetOS)/$(TargetArchitecture)/merge_mibc_files.cache</_MergeMibcFilesCacheFile>
   </PropertyGroup>
 
+  <PropertyGroup Condition="'$(TargetsAppleMobile)' == 'true' and $(PublishReadyToRun)">
+    <PublishReadyToRunContainerFormat>macho</PublishReadyToRunContainerFormat>
+    <UseComposite>true</UseComposite>
+    <CrossgenTargetName>$(CrossgenTargetName);LinkCoreLibMachO</CrossgenTargetName>
+  </PropertyGroup>
+
+  <PropertyGroup Condition="'$(UseComposite)' == 'true'">
+    <CrossgenTargetName>$(CrossgenTargetName);CopyR2RComponentCoreLib</CrossgenTargetName>
+  </PropertyGroup>
+
   <ItemGroup>
     <OptimizationMibcFiles Include="$(MibcOptimizationDataDir)/$(TargetOS)/$(TargetArchitecture)/**/*.mibc" />
   </ItemGroup>
 
   <PropertyGroup>
     <CoreLibAssemblyName>System.Private.CoreLib</CoreLibAssemblyName>
     <CoreLibOutputPath>$([MSBuild]::NormalizePath('$(BinDir)', '$(CoreLibAssemblyName).dll'))</CoreLibOutputPath>
+    <CrossgenCoreLibOutputPath>$(CoreLibOutputPath)</CrossgenCoreLibOutputPath>
+    <CrossgenCoreLibComponentOutputPath></CrossgenCoreLibComponentOutputPath>
     <CoreLibNiPdbPath></CoreLibNiPdbPath>
     <CoreLibPerfMapPath></CoreLibPerfMapPath>
     <CoreLibNiPdbPath Condition="$(BuildPdb)">$([MSBuild]::NormalizePath('$(BinDir)', 'PDB', '$(CoreLibAssemblyName).ni.pdb'))</CoreLibNiPdbPath>
     <CoreLibPerfMapPath Condition="$(BuildPerfMap)">$([MSBuild]::NormalizePath('$(BinDir)', '$(CoreLibAssemblyName).ni.r2rmap'))</CoreLibPerfMapPath>
     <MergedMibcPath>$([MSBuild]::NormalizePath('$(BinDir)', 'StandardOptimizationData.mibc'))</MergedMibcPath>
   </PropertyGroup>
 
+  <PropertyGroup Condition="'$(TargetsAppleMobile)' == 'true'">
+    <CoreLibObjOutputPath>$([MSBuild]::NormalizePath('$(IntermediateOutputPath)', '$(CoreLibAssemblyName).o'))</CoreLibObjOutputPath>
+    <CrossgenCoreLibOutputPath>$(CoreLibObjOutputPath)</CrossgenCoreLibOutputPath>
+    <CoreLibDylibOutputPath>$([MSBuild]::NormalizePath('$(BinDir)', '$(CoreLibAssemblyName).dylib'))</CoreLibDylibOutputPath>
+  </PropertyGroup>
+
+  <PropertyGroup Condition="'$(UseComposite)' == 'true'">
+    <CrossgenCoreLibComponentOutputPath>$([MSBuild]::NormalizePath('$(IntermediateOutputPath)', '$(CoreLibAssemblyName).dll'))</CrossgenCoreLibComponentOutputPath>
+  </PropertyGroup>
+
   <!-- Creates a hash file that changes whenever the input mibc file modification times change.
        This will trigger a re-merge when updating to a new version of the optimization data,
        even if the new timestamps are still days in the past. -->
@@ -88,30 +117,25 @@
   <Target Name="CreateMergedMibcFile"
           DependsOnTargets="ResolveProjectReferences;MergeMibcFiles" />
 
-  <Target Name="PrepareInvokeCrossgen" DependsOnTargets="ResolveProjectReferences;CreateMergedMibcFile">
+  <Target Name="PrepareInvokeCrossgen2" DependsOnTargets="ResolveProjectReferences;CreateMergedMibcFile">
     <ItemGroup>
       <Crossgen2Inputs Include="@(CoreLib)" />
       <Crossgen2Inputs Include="$(MergedMibcPath)" />
       <Crossgen2Inputs Include="@(Crossgen2Files->Metadata('OutputPath'))" />
     </ItemGroup>
   </Target>
 
-  <Target Name="InvokeCrossgenBrowser"
-          DependsOnTargets="PrepareInvokeCrossgen"
-          Inputs="@(Crossgen2Inputs)"
-          Outputs="$(CoreLibOutputPath)"
-          Condition="'$(TargetOS)' == 'browser'"
-          AfterTargets="Build">
-    <!-- TODO-WASM: we will have WASM R2R later, for now we just need to copy the file into expected location -->
+  <Target Name="CopyILCoreLib"
+          DependsOnTargets="ResolveProjectReferences"
+          Inputs="@(CoreLib)"
+          Outputs="$(CoreLibOutputPath)">
     <Copy SourceFiles="@(CoreLib)" DestinationFiles="$(CoreLibOutputPath)" UseHardlinksIfPossible="true" />
   </Target>
 
-  <Target Name="InvokeCrossgen"
-          DependsOnTargets="PrepareInvokeCrossgen;CreateMergedMibcFile"
+  <Target Name="InvokeCrossgen2"
+          DependsOnTargets="PrepareInvokeCrossgen2;CreateMergedMibcFile"
           Inputs="@(Crossgen2Inputs)"
-          Outputs="$(CoreLibOutputPath);$(CoreLibNiPdbPath);$(CoreLibPerfMapPath)"
-          Condition="'$(TargetOS)' != 'browser'"
-          AfterTargets="Build">
+          Outputs="$(CoreLibOutputPath);$(CoreLibNiPdbPath);$(CoreLibPerfMapPath);$(CrossgenCoreLibOutputPath);$(CrossgenCoreLibComponentOutputPath)">
 
     <MakeDir
       Directories="$(BinDir);$(IntermediatesDir);$(LogsDir)" />
@@ -121,9 +145,11 @@
 
     <PropertyGroup>
       <CrossGenDllCmd>@(Crossgen2Files->Metadata('OutputPath')->WithMetadataValue('Filename','crossgen2')->WithMetadataValue('Extension','$(ExeSuffix)'))</CrossGenDllCmd>
-      <CrossGenDllCmd>$(CrossGenDllCmd) -o:$(CoreLibOutputPath)</CrossGenDllCmd>
+      <CrossGenDllCmd>$(CrossGenDllCmd) -o:$(CrossgenCoreLibOutputPath)</CrossGenDllCmd>
       <CrossGenDllCmd>$(CrossGenDllCmd) -r:$([MSBuild]::NormalizePath('$(BinDir)', 'IL', '*.dll'))</CrossGenDllCmd>
       <CrossGenDllCmd>$(CrossGenDllCmd) --targetarch:$(TargetArchitecture)</CrossGenDllCmd>
+      <CrossGenDllCmd  Condition="'$(PublishReadyToRunContainerFormat)' != ''">$(CrossGenDllCmd) --obj-format:$(PublishReadyToRunContainerFormat)</CrossGenDllCmd>
+      <CrossGenDllCmd  Condition="'$(UseComposite)' == 'true'">$(CrossGenDllCmd) --composite</CrossGenDllCmd>
       <CrossGenDllCmd Condition="'$(TargetsAndroid)' != 'true'">$(CrossGenDllCmd) --targetos:$(TargetOS)</CrossGenDllCmd>
       <!-- Unless and until Android requires R2R specific customizations, we're just dealing with another linux -->
       <CrossGenDllCmd Condition="'$(TargetsAndroid)' == 'true'">$(CrossGenDllCmd) --targetos:linux</CrossGenDllCmd>
@@ -145,21 +171,113 @@
       <CrossGenDllCmd>$(CrossGenDllCmd) --perfmap --perfmap-path:$(BinDir)</CrossGenDllCmd>
     </PropertyGroup>
 
-    <Message Condition="$(BuildDll)" Importance="High" Text="$(CrossGenDllCmd)" />
+    <Message Importance="High" Text="$(CrossGenDllCmd)" />
+
+    <Exec Command="$(CrossGenDllCmd)" />
+
+    <Message Importance="High" Text="Crossgenning of System.Private.CoreLib succeeded." />
+    <Message Importance="High" Text="Product binaries are available at $(BinDir)" />
+  </Target>
+
+  <Target Name="CopyR2RComponentCoreLib"
+          DependsOnTargets="InvokeCrossgen2"
+          Inputs="$(CrossgenCoreLibComponentOutputPath)"
+          Outputs="$(CoreLibOutputPath)">
+    <Copy SourceFiles="$(CrossgenCoreLibComponentOutputPath)" DestinationFiles="$(CoreLibOutputPath)" UseHardlinksIfPossible="true" />
+  </Target>
+
+  <Target
+    Name="LinkCoreLibMachO"
+    Inputs="$(CoreLibObjOutputPath)"
+    Outputs="$(CoreLibDylibOutputPath)">
+    <PropertyGroup>
+      <_AppleTargetArchitecture Condition="'$(TargetArchitecture)' == 'x64'">x86_64</_AppleTargetArchitecture>
+      <_AppleTargetArchitecture Condition="'$(TargetArchitecture)' == 'arm64'">arm64</_AppleTargetArchitecture>
+    </PropertyGroup>
+
+    <PropertyGroup>
+      <AppleMinOSVersion Condition="'$(AppleMinOSVersion)' == '' and '$(TargetOS)' == 'osx'">12.0</AppleMinOSVersion>
+      <AppleMinOSVersion Condition="'$(AppleMinOSVersion)' == '' and '$(TargetOS)' == 'maccatalyst'">15.0</AppleMinOSVersion>
+      <AppleMinOSVersion Condition="'$(AppleMinOSVersion)' == '' and ($(TargetOS.StartsWith('ios')) or $(TargetOS.StartsWith('tvos')))">12.2</AppleMinOSVersion>
+
+      <_AppleSdkName Condition="'$(TargetOS)' == 'ios'">iphoneos</_AppleSdkName>
+      <_AppleSdkName Condition="'$(TargetOS)' == 'iossimulator'">iphonesimulator</_AppleSdkName>
+      <_AppleSdkName Condition="'$(TargetOS)' == 'tvos'">appletvos</_AppleSdkName>
+      <_AppleSdkName Condition="'$(TargetOS)' == 'tvossimulator'">appletvsimulator</_AppleSdkName>
+      <_AppleSdkName Condition="'$(TargetOS)' == 'maccatalyst' or '$(TargetOS)' == 'osx'">macosx</_AppleSdkName>
+
+      <_AppleTripleOS Condition="'$(TargetOS)' == 'osx'">macos</_AppleTripleOS>
+      <_AppleTripleOS Condition="'$(TargetOS)' == 'maccatalyst' or $(TargetOS.StartsWith('ios'))">ios</_AppleTripleOS>
+      <_AppleTripleOS Condition="$(TargetOS.StartsWith('tvos'))">tvos</_AppleTripleOS>
+
+      <_AppleTripleAbi Condition="'$(TargetOS)' == 'ios' or '$(TargetOS)' == 'tvos'">macho</_AppleTripleAbi>
+      <_AppleTripleAbi Condition="'$(TargetOS)' == 'maccatalyst'">macabi</_AppleTripleAbi>
+      <_AppleTripleAbi Condition="$(TargetOS.EndsWith('simulator'))">simulator</_AppleTripleAbi>
+
+      <TargetTriple Condition="'$(_AppleTripleAbi)' == ''">$(_AppleTargetArchitecture)-apple-$(_AppleTripleOS)$(AppleMinOSVersion)</TargetTriple>
+      <TargetTriple Condition="'$(_AppleTripleAbi)' != ''">$(_AppleTargetArchitecture)-apple-$(_AppleTripleOS)$(AppleMinOSVersion)-$(_AppleTripleAbi)</TargetTriple>
+    </PropertyGroup>
 
-    <Exec Condition="$(BuildDll)" Command="$(CrossGenDllCmd)" />
+    <PropertyGroup>
+      <Xcrun Condition="'$(Xcrun)' == ''">xcrun</Xcrun>
+      <_WhereXcrun>0</_WhereXcrun>
+    </PropertyGroup>
 
-    <Message Condition="$(BuildPdb)" Importance="High" Text="$(CrossGenPdbCmd)" />
+    <Exec Command="command -v &quot;$(Xcrun)&quot;" IgnoreExitCode="true" StandardOutputImportance="Low">
+      <Output TaskParameter="ExitCode" PropertyName="_WhereXcrun" />
+    </Exec>
+    <Error Condition="'$(_WhereXcrun)' != '0'"
+      Text="'$(Xcrun)' not found in PATH. Make sure '$(Xcrun)' is available in PATH." />
 
-    <Exec Condition="$(BuildPdb) and '$(CrossGenPdbCmd)' != ''" Command="$(CrossGenPdbCmd)" />
+    <Exec Command="&quot;$(Xcrun)&quot; --sdk $(_AppleSdkName) --show-sdk-path" IgnoreExitCode="true" StandardOutputImportance="Low" ConsoleToMsBuild="true">
+      <Output TaskParameter="ConsoleOutput" PropertyName="SysRoot" />
+    </Exec>
 
-    <Message Condition="$(BuildPerfMap)" Importance="High" Text="$(CrossGenPerfMapCmd)" />
+    <Error Condition="!Exists('$(SysRoot)')"
+      Text="Apple SDK was not found in: '$(SysRoot)'" />
 
-    <Exec Condition="$(BuildPerfMap) and '$(CrossGenPerfMapCmd)' != ''" Command="$(CrossGenPerfMapCmd)" />
+    <Exec Command="&quot;$(Xcrun)&quot; --sdk $(_AppleSdkName) --find clang" IgnoreExitCode="true" StandardOutputImportance="Low" ConsoleToMsBuild="true">
+      <Output TaskParameter="ConsoleOutput" PropertyName="_AppleClang" />
+    </Exec>
 
-    <Copy Condition="!$(BuildDll)" SourceFiles="@(CoreLib)" DestinationFiles="$(CoreLibOutputPath)" UseHardlinksIfPossible="true" />
+    <Error Condition="!Exists('$(_AppleClang)')"
+      Text="Apple Clang was not found at: '$(_AppleClang)'" />
 
-    <Message Importance="High" Text="Crossgenning of System.Private.CoreLib succeeded." />
-    <Message Importance="High" Text="Product binaries are available at $(BinDir)" />
+    <ItemGroup>
+      <_MachLinkerArg Include="-gz=zlib" />
+      <_MachLinkerArg Include="-isysroot &quot;$(SysRoot)&quot;" />
+      <_MachLinkerArg Include="--target=$(TargetTriple)" />
+      <_MachLinkerArg Include="-g" />
+      <_MachLinkerArg Include="-dynamiclib" />
+      <_MachLinkerArg Include="-Wl,-dead_strip" />
+    </ItemGroup>
+
+    <Exec Command="&quot;$(_AppleClang)&quot; --version" IgnoreExitCode="true" StandardOutputImportance="Low" ConsoleToMSBuild="true">
+      <Output TaskParameter="ExitCode" PropertyName="_XcodeVersionStringExitCode" />
+      <Output TaskParameter="ConsoleOutput" PropertyName="_XcodeVersionString" />
+    </Exec>
+
+    <PropertyGroup Condition="('$(_XcodeVersionStringExitCode)' == '0' or '$(_XcodeVersionStringExitCode)' == '1') and '$(_XcodeVersionString)' != ''">
+      <_XcodeVersion>$([System.Text.RegularExpressions.Regex]::Match($(_XcodeVersionString), '[1-9]\d*'))</_XcodeVersion>
+    </PropertyGroup>
+
+    <ItemGroup Condition="'$(UseLdClassicXCodeLinker)' != 'false'">
+      <_MachLinkerArg Condition="'$(UseLdClassicXCodeLinker)' == 'true' or '$(_XcodeVersion)' == '15' or '$(_XcodeVersion)' == '16'" Include="-ld_classic" />
+    </ItemGroup>
+
+    <ItemGroup>
+      <_MachLinkerArg Include="-Wl,-install_name,&quot;@rpath/$(CoreLibAssemblyName).dylib&quot;" />
+      <_MachLinkerArg Include="$(CoreLibObjOutputPath)" />
+      <_MachLinkerArg Include="-o $(CoreLibDylibOutputPath)" />
+    </ItemGroup>
+
+    <MakeDir Directories="$([System.IO.Path]::GetDirectoryName($(CoreLibDylibOutputPath)))" />
+
+    <Exec Command="&quot;$(_AppleClang)&quot; @(_MachLinkerArg, ' ')" />
+
+    <!-- remove executable flag -->
+    <Exec Command="chmod 644 &quot;$(CoreLibDylibOutputPath)&quot;" />
   </Target>
+
+  <Target Name="CrossgenCoreLib" DependsOnTargets="$(CrossgenTargetName)" AfterTargets="CoreBuild" />
 </Project>