[API Proposal]: Update to the `LibraryImport` custom marshalling shape

[AaronRobinsonMSFT]

Background and motivation

There has been substantial feedback regarding the current LibraryImport custom marshaller shapes. The interop team has taken this feedback (internal and external), along with practical usage in the ASP.Net Core and WinForms repositories and decided to revise the current shapes. This revision is primarily designed to improve UX and future use cases with reverse P/Invoke (for example, COM source generation).

This API would also apply to all previously approved marshaller APIs.

#66623
#66121
#68248

This will also apply to/supplant outstanding source generator proposals.

#69403
#69281
#69031

API Proposal

Added APIs

namespace System.Runtime.InteropServices.Marshalling;

/// <summary>
/// An enumeration representing the different marshalling scenarios in our marshalling model
/// </summary>
public enum Scenario
{
    /// <summary>
    /// All scenarios. A marshaller specified with this scenario will be used if there is not a specific
    /// marshaller specified for a given usage scenario.
    /// </summary>
    Default,
    /// <summary>
    /// By-value and <c>in</c> parameters in managed-to-unmanaged scenarios, like P/Invoke.
    /// </summary>
    ManagedToUnmanagedIn,
    /// <summary>
    /// <c>ref</c> parameters in managed-to-unmanaged scenarios, like P/Invoke.
    /// </summary>
    ManagedToUnmanagedRef,
    /// <summary>
    /// <c>out</c> parameters in managed-to-unmanaged scenarios, like P/Invoke.
    /// </summary>
    ManagedToUnmanagedOut,
    /// <summary>
    /// By-value and <c>in</c> parameters in unmanaged-to-managed scenarios, like Reverse P/Invoke.
    /// </summary>
    UnmanagedToManagedIn,
    /// <summary>
    /// <c>ref</c> parameters in unmanaged-to-managed scenarios, like Reverse P/Invoke.
    /// </summary>
    UnmanagedToManagedRef,
    /// <summary>
    /// <c>out</c> parameters in unmanaged-to-managed scenarios, like Reverse P/Invoke.
    /// </summary>
    UnmanagedToManagedOut,
    /// <summary>
    /// Elements of arrays passed with <c>in</c> or by-value in interop scenarios.
    /// </summary>
    ElementIn,
    /// <summary>
    /// Elements of arrays passed with <c>ref</c> or passed by-value with both <see cref="InAttribute"/> and <see cref="OutAttribute" /> in interop scenarios.
    /// </summary>
    ElementRef,
    /// <summary>
    /// Elements of arrays passed with <c>out</c> or passed by-value with only <see cref="OutAttribute" /> in interop scenarios.
    /// </summary>
    ElementOut
}

/// <summary>
/// Specifies which marshaller to use for a given managed type and marshalling scenario.
/// </summary>
/// <remarks>
/// This attribute should be placed on a marshaller entry-point type.
/// </remarks>
[AttributeUsage(AttributeTargets.Class, AllowMultiple = true)]
public sealed class CustomMarshallerAttribute : Attribute
{
    public CustomMarshallerAttribute(Type managedType, Scenario scenario, Type marshallerType);

    /// <summary>
    /// Placeholder type for generic parameter
    /// </summary>
    public struct GenericPlaceholder
    {
    }
}

/// <summary>
/// Specifies that a particular generic parameter is the collection element's unmanaged type.
/// </summary>
/// <remarks>
/// If this attribute is provided on a generic parameter of a marshaller, then the generator will assume
/// that it is a linear collection marshaller.
/// </remarks>
[AttributeUsage(AttributeTargets.GenericParameter)]
public sealed class ElementUnmanagedTypeAttribute : Attribute
{
}

[CustomMarshaller(typeof(string), Scenario.Default, typeof(Utf8StringMarshaller))]
[CustomMarshaller(typeof(string), Scenario.ManagedToUnmanagedIn, typeof(ManagedToUnmanaged))]
public static class Utf8StringMarshaller
{
    public static byte* ConvertToUnmanaged(string managed);

    public static string ConvertToManaged(byte* unmanaged);

    public static void Free(byte* unmanaged);

    public ref struct ManagedToUnmanaged
    {
        public static int BufferSize { get; }

        public void FromManaged(string managed, Span<byte> buffer);

        public byte* ToUnmanaged();

        public void FromUnmanaged(byte* unmanaged);

        public string ToManaged();

        public void Free();
    }
}

[CustomMarshaller(typeof(string), Scenario.Default, typeof(AnsiStringMarshaller))]
[CustomMarshaller(typeof(string), Scenario.ManagedToUnmanagedIn, typeof(ManagedToUnmanaged))]
public static class AnsiStringMarshaller
{
    public static byte* ConvertToUnmanaged(string managed);

    public static string ConvertToManaged(byte* unmanaged);

    public static void Free(byte* unmanaged);

    public ref struct ManagedToUnmanaged
    {
        public static int BufferSize { get; }

        public void FromManaged(string managed, Span<byte> buffer);

        public byte* ToUnmanaged();

        public void FromUnmanaged(byte* unmanaged);

        public string ToManaged();

        public void Free();
    }
}

[CustomMarshaller(typeof(string), Scenario.Default, typeof(BStrStringMarshaller))]
public static class BStrStringMarshaller
{
    public static ushort* ConvertToUnmanaged(string managed);

    public static string ConvertToManaged(ushort* unmanaged);

    public static void Free(ushort* unmanaged);

    public ref struct ManagedToUnmanaged
    {
        public static int BufferSize { get; }

        public void FromManaged(string managed, Span<ushort> buffer);

        public ushort* ToUnmanaged();

        public void FromUnmanaged(ushort* unmanaged);

        public string ToManaged();

        public void Free();
    }
}

[CustomMarshaller(typeof(string), Scenario.Default, typeof(Utf16StringMarshaller))]
public static class Utf16StringMarshaller
{
    public static ushort* ConvertToUnmanaged(string managed);

    public static string ConvertToManaged(ushort* unmanaged);

    public static void Free(ushort* unmanaged);

    public static ref char GetPinnableReference(string str);
}

[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder[]), Scenario.Default, typeof(ArrayMarshaller<,>))]
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder[]), Scenario.ManagedToUnmanagedIn, typeof(ArrayMarshaller<,>.In))]
public unsafe static class ArrayMarshaller<T, [ElementUnmanagedType] TUnmanagedElement>
    where TUnmanagedElement : unmanaged
{
    public static byte* AllocateContainerForUnmanagedElements(T[]? managed, out int numElements);

    public static ReadOnlySpan<T> GetManagedValuesSource(T[] managed);

    public static Span<T> GetManagedValuesDestination(T[] managed);

    public static Span<TUnmanagedElement> GetUnmanagedValuesDestination(byte* nativeValue, int numElements);

    public static ReadOnlySpan<TUnmanagedElement> GetUnmanagedValuesSource(byte* nativeValue, int numElements);

    public static T[] AllocateContainerForManagedElements(byte* nativeValue, int length);
    public static void Free(byte* native);

    public unsafe ref struct In
    {
        public static int BufferSize { get; }

        public void FromManaged(T[]? array, Span<TUnmanagedElement> buffer);

        public ReadOnlySpan<T> GetManagedValuesSource();

        public Span<TUnmanagedElement> GetUnmanagedValuesDestination();

        public ref TUnmanagedElement GetPinnableReference();

        public byte* ToUnmanaged();

        public void Free();

        public static ref T GetPinnableReference(T[] managed);
    }
}

[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder*[]), Scenario.Default, typeof(PointerArrayMarshaller<,>))]
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder*[]), Scenario.ManagedToUnmanagedIn, typeof(PointerArrayMarshaller<,>.In))]
public unsafe static class PointerArrayMarshaller<T, [ElementUnmanagedType] TUnmanagedElement>
    where T : unmanaged
    where TUnmanagedElement : unmanaged
{
    public static byte* AllocateContainerForUnmanagedElements(T*[]? managed, out int numElements);

    public static ReadOnlySpan<IntPtr> GetManagedValuesSource(T*[] managed);

    public static Span<IntPtr> GetManagedValuesDestination(T*[] managed);

    public static Span<TUnmanagedElement> GetUnmanagedValuesDestination(byte* nativeValue, int numElements);

    public static ReadOnlySpan<TUnmanagedElement> GetUnmanagedValuesSource(byte* nativeValue, int numElements);

    public static T*[] AllocateContainerForManagedElements(byte* nativeValue, int length);
    public static void Free(byte* native);

    public unsafe ref struct In
    {
        public static int BufferSize { get; }

        public void FromManaged(T*[]? array, Span<TUnmanagedElement> buffer);

        public ReadOnlySpan<IntPtr> GetManagedValuesSource();

        public Span<TUnmanagedElement> GetUnmanagedValuesDestination();

        public ref TUnmanagedElement GetPinnableReference();

        public byte* ToUnmanaged();

        public void Free();

        public static ref T* GetPinnableReference(T*[] managed);
    }
}

[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder*[]), Scenario.Default, typeof(SpanMarshaller<,>))]
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder*[]), Scenario.ManagedToUnmanagedIn, typeof(SpanMarshaller<,>.In))]
public unsafe static class SpanMarshaller<T, [ElementUnmanagedType] TUnmanagedElement>
    where TUnmanagedElement : unmanaged
{
    public static byte* AllocateContainerForUnmanagedElements(Span<T> managed, out int numElements);

    public static ReadOnlySpan<T> GetManagedValuesSource(Span<T> managed);

    public static Span<T> GetManagedValuesDestination(Span<T> managed);

    public static Span<TUnmanagedElement> GetUnmanagedValuesDestination(byte* nativeValue, int numElements);

    public static ReadOnlySpan<TUnmanagedElement> GetUnmanagedValuesSource(byte* nativeValue, int numElements);

    public static Span<T> AllocateContainerForManagedElements(byte* nativeValue, int length);
    public static void Free(byte* native);

    public unsafe ref struct In
    {
        public static int BufferSize { get; }

        public void FromManaged(Span<T> array, Span<TUnmanagedElement> buffer);

        public ReadOnlySpan<T> GetManagedValuesSource();

        public Span<TUnmanagedElement> GetUnmanagedValuesDestination();

        public ref TUnmanagedElement GetPinnableReference();

        public byte* ToUnmanaged();

        public void Free();

        public static ref T GetPinnableReference(Span<T> managed);
    }
}


[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder*[]), Scenario.ManagedToUnmanagedIn, typeof(SpanMarshaller<,>.ManagedToUnmanaged))]
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder*[]), Scenario.UnmanagedToManagedOut, typeof(SpanMarshaller<,>.UnmanagedToManaged))]
public unsafe static class ReadOnlySpanMarshaller<T, [ElementUnmanagedType] TUnmanagedElement>
    where TUnmanagedElement : unmanaged
{
    public unsafe ref struct ManagedToUnmanaged
    {
        public static int BufferSize { get; }

        public void FromManaged(ReadOnlySpan<T> array, Span<TUnmanagedElement> buffer);

        public ReadOnlySpan<T> GetManagedValuesSource();

        public Span<TUnmanagedElement> GetUnmanagedValuesDestination();

        public ref TUnmanagedElement GetPinnableReference();

        public byte* ToUnmanaged();

        public void Free();

        public static ref T GetPinnableReference(ReadOnlySpan<T> managed);
    }

    public unsafe static class UnmanagedToManaged
    {
        public static byte* AllocateContainerForUnmanagedElements(ReadOnlySpan<T> managed, out int numElements);

        public static ReadOnlySpan<T> GetManagedValuesSource(ReadOnlySpan<T> managed);

        public static Span<TUnmanagedElement> GetUnmanagedValuesDestination(byte* nativeValue, int numElements);
    }
}

Removed APIs

namespace System.Runtime.InteropServices.Marshalling;

 [AttributeUsage(AttributeTargets.Struct)]
 public sealed class CustomTypeMarshallerAttribute : Attribute
 {
      public CustomTypeMarshallerAttribute(Type managedType, CustomTypeMarshallerKind marshallerKind = CustomTypeMarshallerKind.Value)
      {
           ManagedType = managedType;
           MarshallerKind = marshallerKind;
      }
 
      public Type ManagedType { get; }
      public CustomTypeMarshallerKind MarshallerKind { get; }
      public int BufferSize { get; set; }
      public CustomTypeMarshallerDirection Direction { get; set; } = CustomTypeMarshallerDirection.Ref;
      public CustomTypeMarshallerFeatures Features { get; set; }
      public struct GenericPlaceholder
      {
      }
 }
 
 public enum CustomTypeMarshallerKind
 {
      Value,
      LinearCollection
 }
 
 [Flags]
 public enum CustomTypeMarshallerFeatures
 {
      None = 0,
      /// <summary>
      /// The marshaller owns unmanaged resources that must be freed
      /// </summary>
      UnmanagedResources = 0x1,
      /// <summary>
      /// The marshaller can use a caller-allocated buffer instead of allocating in some scenarios
      /// </summary>
      CallerAllocatedBuffer = 0x2,
      /// <summary>
      /// The marshaller uses the two-stage marshalling design for its <see cref="CustomTypeMarshallerKind"/> instead of the one-stage design.
      /// </summary>
      TwoStageMarshalling = 0x4
 }
 [Flags]
 public enum CustomTypeMarshallerDirection
 {
      /// <summary>
      /// No marshalling direction
      /// </summary>
      [EditorBrowsable(EditorBrowsableState.Never)]
      None = 0,
      /// <summary>
      /// Marshalling from a managed environment to an unmanaged environment
      /// </summary>
      In = 0x1,
      /// <summary>
      /// Marshalling from an unmanaged environment to a managed environment
      /// </summary>
      Out = 0x2,
      /// <summary>
      /// Marshalling to and from managed and unmanaged environments
      /// </summary>
      Ref = In | Out,
 }

 public ref struct ArrayMarshaller<T>
 {}

 public ref struct PointerArrayMarshaller<T>
 {}

 public ref struct Utf8StringMarshaller
 {}
 public ref struct AnsiStringMarshaller
 {}
 public ref struct Utf16StringMarshaller
 {}
 public ref struct BStrStringMarshaller
 {}

API Usage

[CustomMarshaller(typeof(TManaged), Scenario.ManagedToUnmanagedIn, typeof(ManagedToUnmanaged))]
[CustomMarshaller(typeof(TManaged), Scenario.ManagedToUnmanagedRef, typeof(Bidirectional))]
[CustomMarshaller(typeof(TManaged), Scenario.ManagedToUnmanagedOut, typeof(UnmanagedToManaged))]
[CustomMarshaller(typeof(TManaged), Scenario.UnmanagedToManagedIn, typeof(UnmanagedToManaged))]
[CustomMarshaller(typeof(TManaged), Scenario.UnmanagedToManagedRef, typeof(Bidirectional))]
[CustomMarshaller(typeof(TManaged), Scenario.UnmanagedToManagedOut, typeof(ManagedToUnmanaged))]
[CustomMarshaller(typeof(TManaged), Scenario.ElementIn, typeof(Element))]
[CustomMarshaller(typeof(TManaged), Scenario.ElementRef, typeof(Element))]
[CustomMarshaller(typeof(TManaged), Scenario.ElementOut, typeof(Element))]
public unsafe static class Marshaller // Must be static class
{
    public unsafe ref struct ManagedToUnmanaged
    {
        public void FromManaged(TManaged managed) => throw null; // Optional caller allocation, Span<T>
        public ref byte GetPinnableReference() => throw null; // Optional, allowed on all "stateful" shapes
        public TUnmanaged ToUnmanaged() => throw null;
        public void Free() => throw null; // Should not throw exceptions. Use Free instead of Dispose to avoid issues with marshallers needing to follow the Dispose pattern guidance.

        // We will pattern-match this method and use it when available.
        // Canonical case is for GC.KeepAlive().
        public void NotifyInvokeSucceeded() => throw null;
    }

    public unsafe ref struct Bidirectional
    {
        public void FromManaged(TManaged managed) => throw null; // Optional caller allocation, Span<T>
        public ref byte GetPinnableReference() => throw null; // Optional, allowed on all "stateful" shapes.
        public TUnmanaged ToUnmanaged() => throw null; // Should not throw exceptions.
        public void FromUnmanaged(TUnmanaged native) => throw null; // Should not throw exceptions.
        public TManaged ToManaged() => throw null;
        // ToManagedGuaranteed requests the generator to move the unmarshalling method calls to be called as part of the GuaranteedUnmarshalling stage.
        // The generator will pattern-match this method as an alternative to ToManaged().
        public TManaged ToManagedGuaranteed() => throw null;
        public void Free() => throw null; // Should not throw exceptions.

        public void NotifyInvokeSucceeded() => throw null; // See notes in ManagedToUnmanaged on usage.
    }

    public unsafe struct UnmanagedToManaged
    {
        public void FromUnmanaged(TUnmanaged native) => throw null;
        public TManaged ToManaged() => throw null; // Should not throw exceptions.
        public void Free() => throw null; // Should not throw exceptions.
    }

    // Currently only support stateless. May support stateful in the future
    public static class Element
    {
        // Defined by public interface IMarshaller<TManaged, TUnmanaged> where TUnmanaged : unmanaged
        public static TUnmanaged ConvertToUnmanaged(TManaged managed) => throw null;
        public static TManaged ConvertToManaged(TUnmanaged native) => throw null;
        public static void Free(TUnmanaged native) => throw null; // Should not throw exceptions.
    }
}


[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder[]), Scenario.Default, typeof(ArrayMarshaller<,>))]
[CustomMarshaller(typeof(CustomMarshallerAttribute.GenericPlaceholder[]), Scenario.ManagedToUnmanagedIn, typeof(ArrayMarshaller<,>.In))]
public unsafe static class ArrayMarshaller<T, [ElementUnmanagedType] TUnmanagedElement> // Must be static class
    where TUnmanagedElement : unmanaged
{
    // Defined by public interface IMarshaller<TManaged, TUnmanaged> where TUnmanaged : unmanaged
    public static byte* AllocateContainerForUnmanagedElements(T[]? managed, out int numElements)
    {
        if (managed is null)
        {
            numElements = 0;
            return null;
        }
        numElements = managed.Length;
        return (byte*)Marshal.AllocCoTaskMem(checked(sizeof(TUnmanagedElement) * numElements));
    }

    public static ReadOnlySpan<T> GetManagedValuesSource(T[] managed) => managed;

    public static Span<T> GetManagedValuesDestination(T[] managed) => managed;

    public static Span<TUnmanagedElement> GetUnmanagedValuesDestination(byte* nativeValue, int numElements)
        => new Span<TUnmanagedElement>(nativeValue, numElements);

    public static ReadOnlySpan<TUnmanagedElement> GetUnmanagedValuesSource(byte* nativeValue, int numElements)
        => new Span<TUnmanagedElement>(nativeValue, numElements);

    public static T[] AllocateContainerForManagedElements(byte* nativeValue, int length) => new T[length];
    public static void Free(byte* native) => Marshal.FreeCoTaskMem((IntPtr)native);

    public unsafe ref struct In
    {
        private T[]? _managedArray;
        private IntPtr _allocatedMemory;
        private Span<TUnmanagedElement> _span;

        public static int BufferSize { get; } = 20;

        /// <summary>
        /// Initializes a new instance of the <see cref="ArrayMarshaller{T}"/>.
        /// </summary>
        /// <param name="array">Array to be marshalled.</param>
        /// <param name="buffer">Buffer that may be used for marshalling.</param>
        /// <param name="sizeOfUnmanagedElement">Size of the native element in bytes.</param>
        /// <remarks>
        /// The <paramref name="buffer"/> must not be movable - that is, it should not be
        /// on the managed heap or it should be pinned.
        /// <seealso cref="CustomTypeMarshallerFeatures.CallerAllocatedBuffer"/>
        /// </remarks>
        public void FromManaged(T[]? array, Span<TUnmanagedElement> buffer)
        {
            _allocatedMemory = default;
            if (array is null)
            {
                _managedArray = null;
                _span = default;
                return;
            }

            _managedArray = array;

            // Always allocate at least one byte when the array is zero-length.
            int bufferSize = checked(array.Length * sizeof(TUnmanagedElement));
            int spaceToAllocate = Math.Max(bufferSize, 1);
            if (spaceToAllocate <= buffer.Length)
            {
                _span = buffer[0..spaceToAllocate];
            }
            else
            {
                _allocatedMemory = Marshal.AllocCoTaskMem(spaceToAllocate);
                _span = new Span<TUnmanagedElement>((void*)_allocatedMemory, spaceToAllocate);
            }
        }

        /// <summary>
        /// Gets a span that points to the memory where the managed values of the array are stored.
        /// </summary>
        /// <returns>Span over managed values of the array.</returns>
        /// <remarks>
        /// <seealso cref="CustomTypeMarshallerDirection.In"/>
        /// </remarks>
        public ReadOnlySpan<T> GetManagedValuesSource() => _managedArray;

        /// <summary>
        /// Returns a span that points to the memory where the native values of the array should be stored.
        /// </summary>
        /// <returns>Span where native values of the array should be stored.</returns>
        public Span<TUnmanagedElement> GetUnmanagedValuesDestination() => _span;

        /// <summary>
        /// Returns a reference to the marshalled array.
        /// </summary>
        public ref TUnmanagedElement GetPinnableReference() => ref MemoryMarshal.GetReference(_span);

        /// <summary>
        /// Returns the native value representing the array.
        /// </summary>
        public byte* ToUnmanaged() => (byte*)Unsafe.AsPointer(ref GetPinnableReference());

        /// <summary>
        /// Frees native resources.
        /// </summary>
        public void Free()
        {
            Marshal.FreeCoTaskMem(_allocatedMemory);
        }

        public static ref T GetPinnableReference(T[] managed) // Optional, allowed on all shapes
        {
            if (managed is null)
            {
                return ref Unsafe.NullRef<T>();
            }
            return ref MemoryMarshal.GetArrayDataReference(managed);
        }
    }

    public unsafe ref struct Out
    {
        private T[]? _managedArray;
        private IntPtr _allocatedMemory;
        private Span<TUnmanagedElement> _span;

        /// <summary>
        /// Gets a span that points to the memory where the unmarshalled managed values of the array should be stored.
        /// </summary>
        /// <param name="length">Length of the array.</param>
        /// <returns>Span where managed values of the array should be stored.</returns>
        public Span<T> GetManagedValuesDestination(int length) => _allocatedMemory == IntPtr.Zero ? null : _managedArray = new T[length];

        /// <summary>
        /// Returns a span that points to the memory where the native values of the array are stored after the native call.
        /// </summary>
        /// <param name="length">Length of the array.</param>
        /// <returns>Span over the native values of the array.</returns>
        public ReadOnlySpan<TUnmanagedElement> GetUnmanagedValuesSource(int length)
        {
            if (_allocatedMemory == IntPtr.Zero)
                return default;

            return _span = new Span<TUnmanagedElement>((void*)_allocatedMemory, length);
        }

        /// <summary>
        /// Returns a reference to the marshalled array.
        /// </summary>
        public ref TUnmanagedElement GetPinnableReference() => ref MemoryMarshal.GetReference(_span);

        /// <summary>
        /// Sets the native value representing the array.
        /// </summary>
        /// <param name="value">The native value.</param>
        public void FromUnmanaged(byte* value)
        {
            _allocatedMemory = (IntPtr)value;
        }

        /// <summary>
        /// Returns the managed array.
        /// </summary>
        public T[]? ToManaged() => _managedArray;

        /// <summary>
        /// Frees native resources.
        /// </summary>
        public void Free()
        {
            Marshal.FreeCoTaskMem(_allocatedMemory);
        }
    }
}

Alternative Designs

Instead of specifying the BufferSize as a static property on the marshaller type, developers could specify it on an attribute. We determined we did not want to go down this route as we're planning on moving to an interface-based approach as the primary usage scenario in .NET 8 and a static get-only property can be specified on an interface with static abstract members. This also frees us to make the value not a constant at compile time, and part of the API contract, which can be useful if we decide that a particular value needs to be tweaked for performance purposes.

/// <summary>
/// Define features for a custom type marshaller.
/// </summary>
public sealed class CustomTypeMarshallerFeaturesAttribute : Attribute
{
    /// <summary>
    /// Desired caller buffer size for the marshaller.
    /// </summary>
    public int BufferSize { get; set; }
}

Instead of specifying the specific marshallers on the marshaller entry-point type with a single attribute and an enum, we also considered providing separate attributes that required explicitly specifying the marshaller type for each scenario. We decided that requiring the user to specify marshallers for every individual scenario was overly verbose and a bad user experience.

/// <summary>
/// Base class attribute for custom marshaller attributes.
/// </summary>
/// <remarks>
/// Use a base class here to allow doing ManagedToUnmanagedMarshallersAttribute.GenericPlaceholder, etc. without having 3 separate placeholder types.
/// For the following attribute types, any marshaller types that are provided will be validated by an analyzer to have the correct members to prevent
/// developers from accidentally typoing a member like Free() and causing memory leaks.
/// </remarks>
public abstract class CustomUnmanagedTypeMarshallersAttributeBase : Attribute
{
    /// <summary>
    /// Placeholder type for generic parameter
    /// </summary>
    public struct GenericPlaceholder { }
}

/// <summary>
/// Specify marshallers used in the managed to unmanaged direction (that is, P/Invoke)
/// </summary>
public sealed class ManagedToUnmanagedMarshallersAttribute : CustomUnmanagedTypeMarshallersAttributeBase
{
    /// <summary>
    /// Create instance of <see cref="ManagedToUnmanagedMarshallersAttribute"/>.
    /// </summary>
    /// <param name="managedType">Managed type to marshal</param>
    public ManagedToUnmanagedMarshallersAttribute(Type managedType) { }

    /// <summary>
    /// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>in</c> keyword.
    /// </summary>
    public Type? InMarshaller { get; set; }

    /// <summary>
    /// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>ref</c> keyword.
    /// </summary>
    public Type? RefMarshaller { get; set; }

    /// <summary>
    /// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>out</c> keyword.
    /// </summary>
    public Type? OutMarshaller { get; set; }
}

/// <summary>
/// Specify marshallers used in the unmanaged to managed direction (that is, Reverse P/Invoke)
/// </summary>
public sealed class UnmanagedToManagedMarshallersAttribute : CustomUnmanagedTypeMarshallersAttributeBase
{
    /// <summary>
    /// Create instance of <see cref="UnmanagedToManagedMarshallersAttribute"/>.
    /// </summary>
    /// <param name="managedType">Managed type to marshal</param>
    public UnmanagedToManagedMarshallersAttribute(Type managedType) { }

    /// <summary>
    /// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>in</c> keyword.
    /// </summary>
    public Type? InMarshaller { get; set; }

    /// <summary>
    /// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>ref</c> keyword.
    /// </summary>
    public Type? RefMarshaller { get; set; }

    /// <summary>
    /// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>out</c> keyword.
    /// </summary>
    public Type? OutMarshaller { get; set; }
}

/// <summary>
/// Specify marshaller for array-element marshalling and default struct field marshalling.
/// </summary>
public sealed class ElementMarshallerAttribute : CustomUnmanagedTypeMarshallersAttributeBase
{
    /// <summary>
    /// Create instance of <see cref="ElementMarshallerAttribute"/>.
    /// </summary>
    /// <param name="managedType">Managed type to marshal</param>
    /// <param name="elementMarshaller">Marshaller type to use for marshalling <paramref name="managedType"/>.</param>
    public ElementMarshallerAttribute(Type managedType, Type elementMarshaller) { }
}

Risks

This is close to an RC and is a non-trivial update to the Marshaller shape. The ASP.Net Core and WinForms repos will need to respond as well. This is introducing complexity in the development slow down following an RC release in several weeks. There is no doubt risk here. A potential escape is to go with the current shape and "make it work". Given the internal/external feedback we have received the risk is worth it from the perspective of the Interop team.

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Background and motivation

There has been substantial feedback regarding the current LibraryImport custom marshaller shapes. The interop team has taken this feedback (internal and external), along with practical usage in the ASP.Net Core and WinForms repositories and decided to revise the current shapes. This revision is primarily designed to improve UX and future use cases with reverse P/Invoke (for example, COM source generation).

This API would also apply to all previously approved marshaller APIs.

#66623
#66121
#68248

This will also apply to/supplant outstanding source generator proposals.

#69403
#69281
#69031

API Proposal

/// <summary>
/// Define features for a custom type marshaller.
/// </summary>
public sealed class CustomTypeMarshallerFeaturesAttribute : Attribute
{
    /// <summary>
    /// Desired caller buffer size for the marshaller.
    /// </summary>
    public int BufferSize { get; set; }
}


/// <summary>
/// Base class attribute for custom marshaller attributes.
/// </summary>
/// <remarks>
/// Use a base class here to allow doing ManagedToUnmanagedMarshallersAttribute.GenericPlaceholder, etc. without having 3 separate placeholder types.
/// For the following attribute types, any marshaller types that are provided will be validated by an analyzer to have the correct members to prevent
/// developers from accidentally typoing a member like Free() and causing memory leaks.
/// </remarks>
public abstract class CustomUnmanagedTypeMarshallersAttributeBase : Attribute
{
    /// <summary>
    /// Placeholder type for generic parameter
    /// </summary>
    public sealed class GenericPlaceholder { }
}

/// <summary>
/// Specify marshallers used in the managed to unmanaged direction (that is, P/Invoke)
/// </summary>
public sealed class ManagedToUnmanagedMarshallersAttribute : CustomUnmanagedTypeMarshallersAttributeBase
{
    /// <summary>
    /// Create instance of <see cref="ManagedToUnmanagedMarshallersAttribute"/>.
    /// </summary>
    /// <param name="managedType">Managed type to marshal</param>
    public ManagedToUnmanagedMarshallersAttribute(Type managedType) { }

    /// <summary>
    /// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>in</c> keyword.
    /// </summary>
    public Type? InMarshaller { get; set; }

    /// <summary>
    /// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>ref</c> keyword.
    /// </summary>
    public Type? RefMarshaller { get; set; }

    /// <summary>
    /// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>out</c> keyword.
    /// </summary>
    public Type? OutMarshaller { get; set; }
}

/// <summary>
/// Specify marshallers used in the unmanaged to managed direction (that is, Reverse P/Invoke)
/// </summary>
public sealed class UnmanagedToManagedMarshallersAttribute : CustomUnmanagedTypeMarshallersAttributeBase
{
    /// <summary>
    /// Create instance of <see cref="UnmanagedToManagedMarshallersAttribute"/>.
    /// </summary>
    /// <param name="managedType">Managed type to marshal</param>
    public UnmanagedToManagedMarshallersAttribute(Type managedType) { }

    /// <summary>
    /// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>in</c> keyword.
    /// </summary>
    public Type? InMarshaller { get; set; }

    /// <summary>
    /// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>ref</c> keyword.
    /// </summary>
    public Type? RefMarshaller { get; set; }

    /// <summary>
    /// Marshaller to use when a parameter of the managed type is passed by-value or with the <c>out</c> keyword.
    /// </summary>
    public Type? OutMarshaller { get; set; }
}

/// <summary>
/// Specify marshaller for array-element marshalling and default struct field marshalling.
/// </summary>
public sealed class ElementMarshallerAttribute : CustomUnmanagedTypeMarshallersAttributeBase
{
    /// <summary>
    /// Create instance of <see cref="ElementMarshallerAttribute"/>.
    /// </summary>
    /// <param name="managedType">Managed type to marshal</param>
    /// <param name="elementMarshaller">Marshaller type to use for marshalling <paramref name="managedType"/>.</param>
    public ElementMarshallerAttribute(Type managedType, Type elementMarshaller) { }
}

/// <summary>
/// Specifies that a particular generic parameter is the collection element's unmanaged type.
/// </summary>
/// <remarks>
/// If this attribute is provided on a generic parameter of a marshaller, then the generator will assume
/// that it is a linear collection marshaller.
/// </remarks>
[AttributeUsage(AttributeTargets.GenericParameter)]
public sealed class ElementUnmanagedTypeAttribute : Attribute
{
}

API Usage

[ManagedToUnmanagedMarshallers(
    typeof(TManaged),
    InMarshaller = typeof(ManagedToUnmanaged),
    RefMarshaller = typeof(Bidirectional),
    OutMarshaller = typeof(UnmanagedToManaged))]
[UnmanagedToManagedMarshallers(
    typeof(TManaged),
    InMarshaller = typeof(UnmanagedToManaged),
    RefMarshaller = typeof(Bidirectional),
    OutMarshaller = typeof(ManagedToUnmanaged))]
[ElementMarshaller(
    typeof(TManaged),
    typeof(Element))]
public unsafe static class Marshaller // Must be static class
{
    public unsafe ref struct ManagedToUnmanaged
    {
        public void FromManaged(TManaged managed) => throw null; // Optional caller allocation, Span<T>
        public ref byte GetPinnableReference() => throw null; // Optional, allowed on all "stateful" shapes
        public TUnmanaged ToUnmanaged() => throw null;
        public void Free() => throw null; // Should not throw exceptions. Use Free instead of Dispose to avoid issues with marshallers needing to follow the Dispose pattern guidance.
    }

    public unsafe ref struct Bidirectional
    {
        public void FromManaged(TManaged managed) => throw null; // Optional caller allocation, Span<T>
        public ref byte GetPinnableReference() => throw null; // Optional, allowed on all "stateful" shapes.
        public TUnmanaged ToUnmanaged() => throw null; // Should not throw exceptions.
        public void FromUnmanaged(TUnmanaged native) => throw null; // Should not throw exceptions.
        public TManaged ToManaged() => throw null;
        public void Free() => throw null; // Should not throw exceptions.
    }

    public unsafe struct UnmanagedToManaged
    {
        public void FromUnmanaged(TUnmanaged native) => throw null;
        public TManaged ToManaged() => throw null; // Should not throw exceptions.
        public void Free() => throw null; // Should not throw exceptions.
    }

    // Currently only support stateless. May support stateful in the future
    public static class Element
    {
        // Defined by public interface IMarshaller<TManaged, TUnmanaged> where TUnmanaged : unmanaged
        public static TUnmanaged ConvertToUnmanaged(TManaged managed) => throw null;
        public static TManaged ConvertToManaged(TUnmanaged native) => throw null;
        public static void Free(TUnmanaged native) => throw null; // Should not throw exceptions.
    }
}

[ManagedToUnmanagedMarshallers(typeof(ManagedToUnmanagedMarshallersAttribute.GenericPlaceholder[]),
    InMarshaller = typeof(ArrayMarshaller<,>.In),
    RefMarshaller = typeof(ArrayMarshaller<,>),
    OutMarshaller = typeof(ArrayMarshaller<,>.Out))]
[UnmanagedToManagedMarshallers(typeof(UnmanagedToManagedMarshallersAttribute.GenericPlaceholder[]),
    InMarshaller = typeof(ArrayMarshaller<,>),
    RefMarshaller = typeof(ArrayMarshaller<,>),
    OutMarshaller = typeof(ArrayMarshaller<,>))]
[ElementMarshaller(typeof(ElementMarshallerAttribute.GenericPlaceholder[]), typeof(ArrayMarshaller<,>))]
[CustomTypeMarshallerFeatures(BufferSize = 20)]
public unsafe static class ArrayMarshaller<T, [ElementUnmanagedType] TUnmanagedElement> // Must be static class
    where TUnmanagedElement : unmanaged
{
    // Defined by public interface IMarshaller<TManaged, TUnmanaged> where TUnmanaged : unmanaged
    public static byte* AllocateContainerForUnmanagedElements(T[]? managed, out int numElements)
    {
        if (managed is null)
        {
            numElements = 0;
            return null;
        }
        numElements = managed.Length;
        return (byte*)Marshal.AllocCoTaskMem(checked(sizeof(TUnmanagedElement) * numElements));
    }

    public static ReadOnlySpan<T> GetManagedValuesSource(T[] managed) => managed;

    public static Span<T> GetManagedValuesDestination(T[] managed) => managed;

    public static Span<TUnmanagedElement> GetUnmanagedValuesDestination(byte* nativeValue, int numElements)
        => new Span<TUnmanagedElement>(nativeValue, numElements);

    public static ReadOnlySpan<TUnmanagedElement> GetUnmanagedValuesSource(byte* nativeValue, int numElements)
        => new Span<TUnmanagedElement>(nativeValue, numElements);

    public static T[] AllocateContainerForManagedElements(int length) => new T[length];
    public static void Free(byte* native) => Marshal.FreeCoTaskMem((IntPtr)native);

    [CustomTypeMarshallerFeatures(BufferSize = 20)] // As our buffer is typed to our native element type, we limit the buffer size based on number of elements, not number of bytes.
    public unsafe ref struct In
    {
        private T[]? _managedArray;
        private IntPtr _allocatedMemory;
        private Span<TUnmanagedElement> _span;

        /// <summary>
        /// Initializes a new instance of the <see cref="ArrayMarshaller{T}"/>.
        /// </summary>
        /// <param name="array">Array to be marshalled.</param>
        /// <param name="buffer">Buffer that may be used for marshalling.</param>
        /// <param name="sizeOfUnmanagedElement">Size of the native element in bytes.</param>
        /// <remarks>
        /// The <paramref name="buffer"/> must not be movable - that is, it should not be
        /// on the managed heap or it should be pinned.
        /// <seealso cref="CustomTypeMarshallerFeatures.CallerAllocatedBuffer"/>
        /// </remarks>
        public void FromManaged(T[]? array, Span<TUnmanagedElement> buffer)
        {
            _allocatedMemory = default;
            if (array is null)
            {
                _managedArray = null;
                _span = default;
                return;
            }

            _managedArray = array;

            // Always allocate at least one byte when the array is zero-length.
            int bufferSize = checked(array.Length * sizeof(TUnmanagedElement));
            int spaceToAllocate = Math.Max(bufferSize, 1);
            if (spaceToAllocate <= buffer.Length)
            {
                _span = buffer[0..spaceToAllocate];
            }
            else
            {
                _allocatedMemory = Marshal.AllocCoTaskMem(spaceToAllocate);
                _span = new Span<TUnmanagedElement>((void*)_allocatedMemory, spaceToAllocate);
            }
        }

        /// <summary>
        /// Gets a span that points to the memory where the managed values of the array are stored.
        /// </summary>
        /// <returns>Span over managed values of the array.</returns>
        /// <remarks>
        /// <seealso cref="CustomTypeMarshallerDirection.In"/>
        /// </remarks>
        public ReadOnlySpan<T> GetManagedValuesSource() => _managedArray;

        /// <summary>
        /// Returns a span that points to the memory where the native values of the array should be stored.
        /// </summary>
        /// <returns>Span where native values of the array should be stored.</returns>
        public Span<TUnmanagedElement> GetUnmanagedValuesDestination() => _span;

        /// <summary>
        /// Returns a reference to the marshalled array.
        /// </summary>
        public ref TUnmanagedElement GetPinnableReference() => ref MemoryMarshal.GetReference(_span);

        /// <summary>
        /// Returns the native value representing the array.
        /// </summary>
        public byte* ToUnmanaged() => (byte*)Unsafe.AsPointer(ref GetPinnableReference());

        /// <summary>
        /// Sets the native value representing the array.
        /// </summary>
        /// <param name="value">The native value.</param>
        public void FromUnmanaged(byte* value)
        {
            _allocatedMemory = (IntPtr)value;
        }

        /// <summary>
        /// Returns the managed array.
        /// </summary>
        public T[]? ToManaged() => _managedArray;

        /// <summary>
        /// Frees native resources.
        /// </summary>
        public void Free()
        {
            Marshal.FreeCoTaskMem(_allocatedMemory);
        }

        public static ref T GetPinnableReference(T[] managed) // Optional, allowed on all shapes
        {
            if (managed is null)
            {
                return ref Unsafe.NullRef<T>();
            }
            return ref MemoryMarshal.GetArrayDataReference(managed);
        }
    }

    public unsafe ref struct Out
    {
        private T[]? _managedArray;
        private IntPtr _allocatedMemory;
        private Span<TUnmanagedElement> _span;

        /// <summary>
        /// Gets a span that points to the memory where the unmarshalled managed values of the array should be stored.
        /// </summary>
        /// <param name="length">Length of the array.</param>
        /// <returns>Span where managed values of the array should be stored.</returns>
        public Span<T> GetManagedValuesDestination(int length) => _allocatedMemory == IntPtr.Zero ? null : _managedArray = new T[length];

        /// <summary>
        /// Returns a span that points to the memory where the native values of the array are stored after the native call.
        /// </summary>
        /// <param name="length">Length of the array.</param>
        /// <returns>Span over the native values of the array.</returns>
        public ReadOnlySpan<TUnmanagedElement> GetUnmanagedValuesSource(int length)
        {
            if (_allocatedMemory == IntPtr.Zero)
                return default;

            return _span = new Span<TUnmanagedElement>((void*)_allocatedMemory, length);
        }

        /// <summary>
        /// Returns a reference to the marshalled array.
        /// </summary>
        public ref TUnmanagedElement GetPinnableReference() => ref MemoryMarshal.GetReference(_span);

        /// <summary>
        /// Sets the native value representing the array.
        /// </summary>
        /// <param name="value">The native value.</param>
        public void FromUnmanaged(byte* value)
        {
            _allocatedMemory = (IntPtr)value;
        }

        /// <summary>
        /// Returns the managed array.
        /// </summary>
        public T[]? ToManaged() => _managedArray;

        /// <summary>
        /// Frees native resources.
        /// </summary>
        public void Free()
        {
            Marshal.FreeCoTaskMem(_allocatedMemory);
        }
    }
}

Alternative Designs

No response

Risks

This is close to an RC and is a non-trivial update to the Marshaller shape. The ASP.Net Core and WinForms repos will need to respond as well. This is introducing complexity in the development slow down following an RC release in several weeks. There is no doubt risk here. A potential escape is to go with the current shape and "make it work". Given the internal/external feedback we have received the risk is worth it from the perspective of the Interop team.

Author:	AaronRobinsonMSFT
Assignees:	-
Labels:	api-suggestion, area-System.Runtime.InteropServices, blocking
Milestone:	7.0.0

[AaronRobinsonMSFT]

/cc @pavelsavara

[jkoritzinsky]

The existing GenericPlaceholder type is a struct. Since a struct can be substituted into more generic constraints, do we want to make the GenericPlaceholder type in this PR also a struct?

[jkotas]

CustomUnmanagedTypeMarshallersAttributeBase
ManagedToUnmanagedMarshallersAttribute
UnmanagedToManagedMarshallersAttribute
ElementMarshallerAttribute

I am wondering whether it would look better to replace all these attributes with just one attribute and enum. Something like:

public class CustomMarshallerAttribute : Attribute
{
    public CustomMarshallerAttribute(Type managedType) { }

    public CustomMarshallerAttribute(Type managedType, CustomMarshallersKind kind, Type marshallerType) { }

    /// <summary>
    /// Placeholder type for generic parameter
    /// </summary>
    public sealed class GenericPlaceholder { }
}

public enum CustomMarshallersKind
{
    Default,
    ManagedToUnmanaged, // Optional - allows you to specify marshaller for all managed->unmanaged cases without repeating the marshaller type 3x
    ManagedToUnmanagedIn,
    ManagedToUnmanagedRef,
    ManagedToUnmanagedOut,
    UnmanagedToManaged, // Optional - allows you to specify marshaller for all unmanaged->managed cases without repeating the marshaller type 3x
    UnmanagedToManagedIn,
    UnmanagedToManagedRef,
    UnmanagedToManagedOut,
    ElementMarshaller
    ...
}

[jkotas]

CustomTypeMarshallerFeaturesAttribute

Now that all marshaller features are based on pattern matching, the BufferSize that is the only thing left in CustomTypeMarshallerFeaturesAttribute can be switched to pattern matching too by defining as const field:

public unsafe ref struct In
{
    public const int StackBufferSize = 20;
}

Field may be more consistent with the overall design than an attribute. (I do not have a strong opinion about it.)

[elinor-fung]

We had the buffer size field vs attribute in the alternate designs to make sure to bring it up during API review. (I don't think any of us had a strong opinion either.)