Add initial doc for future R2R platform-native image (Mach-O, composite only) (#120584)

Create a doc with a tentative plan for the platform-native R2R support
we are intending to add (Mach-O, composite only). We can update this as
we go.

Author: elinor-fung

docs/design/coreclr/botr/readytorun-format.md

@@ -30,8 +30,8 @@ The COR header and ECMA 335 metadata pointed to by the COM descriptor data direc
 in the COFF header represent a full copy of the input IL and MSIL metadata it was generated from.
 
 **Composite R2R files** currently conform to Windows PE executable file format as the
-native envelope. Moving forward we plan to gradually add support for platform-native
-executable formats (ELF on Linux, MachO on OSX) as the native envelopes. There is a
+native envelope. Moving forward we [plan to gradually add support for platform-native
+executable formats](./readytorun-platform-native-envelope.md) (ELF on Linux, MachO on OSX) as the native envelopes. There is a
 global CLI / COR header in the file, but it only exists to facilitate pdb generation, and does
 not participate in any usages by the CoreCLR runtime. The ReadyToRun header structure is pointed to
 by the well-known export symbol `RTR_HEADER` and has the `READYTORUN_FLAG_COMPOSITE` flag set.

docs/design/coreclr/botr/readytorun-overview.md

@@ -60,7 +60,7 @@ Some of this information can be omitted or stored in more efficient form, e.g.:
 - The garbage collection information can be omitted for environments with conservative garbage collection, such as IL2CPP.
 - The full metadata information is not strictly required for 'private' methods or types so it is possible to strip it from the CLI image.
 - The metadata can be stored in more efficient form, such as the .NET Native metadata format.
-- The platform native executable format (ELF, Mach-O) can be used as envelope instead of PE to take advantage of platform OS loader.
+- As of .NET 10, the PE format is used on all platforms. A [future improvement](#platform-native-envelope-support) could be support for the platform native executable format (ELF, Mach-O) to take advantage of platform OS loader.
 
 
 ## Definition of Version Compatibility for Native Code
@@ -332,4 +332,8 @@ Another important observation is that `MethodTable` contains other very frequent
 
 # Current State
 
-The design and implementation is a work in progress under code name ReadyToRun (`FEATURE_READYTORUN`). RyuJIT is used as the code generator to produce the ReadyToRun images currently.
+As of .NET 6, the .NET SDK can publish applications as ReadyToRun using the `crossgen2` tool. The tool lives under `src/coreclr/aot/crossgen2`. Support in the coreclr runtime is under `FEATURE_READYTORUN`. RyuJIT is used as the code generator to produce the ReadyToRun images currently.
+
+## Platform-Native Envelope Support
+
+Through .NET 10, ReadyToRun uses the PE format on all platforms. In .NET 11, we plan to start adding support for other formats, with Mach-O being the first target. For more details, see [ReadyToRun Platform-Native Envelope](./readytorun-platform-native-envelope.md).

docs/design/coreclr/botr/readytorun-platform-native-envelope.md

@@ -0,0 +1,65 @@
+# ReadyToRun Platform Native Envelope
+
+Up through .NET 10, ReadyToRun (R2R) uses the Windows PE format as the native envelope on every platform. Non‑Windows platforms therefore load a PE file with the .NET loader performing the required fixups and code activation.
+
+In .NET 11, we plan to start adding support beyond the PE format. We will target support for:
+- Composite R2R only
+- Mach-O object files emitted by `crossgen2`
+- Runtime using a composite R2R image that is a Mach-O shared library
+   - Linking the object files into a shared library is expected to be handled by the SDK and is not covered in this document.
+
+The tentative high-level design is outlined below. As we implement this support, this document should be updated with more details and the [ReadyToRun overview](./readytorun-overview.md) and [ReadyToRun format](./readytorun-format.md) should be updated to reflect the changes.
+
+## crossgen2: producing Mach-O object files
+
+Mach‑O support will only be supported for composite ReadyToRun when the target OS is macOS. It will be opt-in via a new `crossgen2` flag:
+- `--obj-format macho`
+
+`crossgen2` will:
+- Produce a Mach-O object file as the composite R2R image with the `RTR_HEADER` export for the `READYTORUN_HEADER`.
+- Mark each input IL assembly as a component R2R assembly: `READYTORUN_FLAG_COMPONENT`.
+- Mark each input IL assembly with a new flag indicating that the associated composite image is in the platform-native format: `READYTORUN_FLAG_PLATFORM_NATIVE_IMAGE`
+
+`crossgen2` does not produce the final shared library. A separate SDK / build linking step must preserve the `RTR_HEADER` export in the final `dylib`.
+
+## Runtime: consuming a platform-native R2R image
+
+The runtime will be updated to handle platform-native R2R images during assembly load.
+
+1. Load IL assembly and determine if it is a R2R assembly.
+2. If it is not a component R2R assembly, proceed with existing R2R load logic.
+  - We will not have platform-native support for this scenario
+3. If it is a component R2R assembly with the new `READYTORUN_FLAG_PLATFORM_NATIVE_IMAGE` flag set:
+   a. Read `OwnerCompositeExecutable` value.
+   b. Invoke host callback with component assembly path and owner composite name.
+   c. On success, obtain pointer to composite `READYTORUN_HEADER` and use it for native method lookup / fixups.
+   d. On failure, fall back to IL/JIT path.
+4. If the platform-native flag is not set, proceed with existing R2R load logic (PE assembly lookup and load).
+
+### Host callback
+
+The [`host_runtime_contract`](/src/native/corehost/host_runtime_contract.h) will be updated with a new callback for getting native code information.
+
+```c
+struct native_code_context
+{
+    size_t size;                       // size of this struct
+    const char* assembly_path;         // component assembly path
+    const char* owner_composite_name;  // name from component R2R header
+};
+
+struct native_code_data
+{
+   size_t size;           // size of this struct
+   void* r2r_header_ptr;  // ReadyToRun header
+   size_t image_size;     // size of the image
+   void* image_base;      // base address where the image was loaded
+};
+
+bool get_native_code_data(
+   const struct native_code_context* context,
+   /*out*/ struct native_code_data* data
+);
+```
+
+This leaves it to the host to do the actual load (for example, `dlopen` of a shared library, using something statically linked into the host itself) of the platform-native image. It is also responsible for any caching desired.