Description
Background and Motivation
As part of the DllImport Source Generator proposal, we need a mechanism to allow users to enable marshalling their struct types from managed code to native code. This issue proposes a design along with an extension we have been using in the DllImportGenerator that is flexible enough to enable us to create a struct interop source generator if we so desire. This design uses the following attributes as well as a convention-based model for the emitted code as described in the linked design doc. The DllImportGenerator will consume the proposed attributes in combination with the DisableRuntimeMarshallingAttribute for its definition of 'does not require marshalling' to allow marshalling of user-defined types.
Proposed API
Related proposal: #46822
namespace System.Runtime.InteropServices
{
/// <summary>
/// Indicates the default marshalling when the attributed type is used in source-generated interop.
/// </summary>
[AttributeUsage(AttributeTargets.Struct | AttributeTargets.Class)]
public sealed class NativeMarshallingAttribute : Attribute
{
/// <summary>
/// Initializes a new instance of the <see cref="NativeMarshallingAttribute"/>
/// </summary>
/// <param name="nativeType">Type that should be used when marshalling the attributed type. The type must not require marshalling.</param>
public NativeMarshallingAttribute(Type nativeType) {}
}
/// <summary>
/// Indicates the marshalling to use in source-generated interop.
/// </summary>
[AttributeUsage(AttributeTargets.Parameter | AttributeTargets.ReturnValue | AttributeTargets.Field, AllowMultiple = true)]
public sealed class MarshalUsingAttribute : Attribute
{
/// <summary>
/// Initializes a new instance of the <see cref="MarshalUsingAttribute"/>
/// </summary>
public MarshalUsingAttribute() {}
/// <summary>
/// Initializes a new instance of the <see cref="MarshalUsingAttribute"/>
/// </summary>
/// <param name="nativeType">Type that should be used when marshalling the attributed parameter or field. The type must not require marshalling.</param>
public MarshalUsingAttribute(Type nativeType) {}
/// <summary>
/// Type that should be used when marshalling the attributed parameter, field or, when ElementIndirectionLevel is specified, collection element on the attributed item.
/// </summary>
public Type? NativeType { get; }
/// <summary>
/// Name of the parameter that contains the number of native collection elements. A value of <see cref="ReturnsCountValue"/> indicates the return value of the p/invoke should be used.
/// </summary>
public string CountElementName { get; set; }
/// <summary>
/// Number of elements in the native collection.
/// </summary>
public int ConstantElementCount { get; set; }
/// <summary>
/// The level of indirection in a collection to which this attribute should apply.
/// </summary>
public int ElementIndirectionLevel { get; set; }
/// <summary>
/// When returned by <see cref="CountElementName" />, indicates the return value of the p/invoke should be used as the count.
/// </summary>
public const string ReturnsCountValue = "return-value";
}
/// <summary>
/// Indicates this type is a marshaller type to be used in source-generated interop.
/// </summary>
[AttributeUsage(AttributeTargets.Struct)]
public sealed class CustomTypeMarshallerAttribute : Attribute
{
public CustomTypeMarshallerAttribute(Type managedType, CustomTypeMarshallerKind marshallerKind = CustomTypeMarshallerKind.Value)
{
ManagedType = managedType;
MarshallerKind = marshallerKind;
}
/// <summary>
/// The managed type that the attributed type is a marshaller for.
/// </summary>
public Type ManagedType { get; }
/// <summary>
/// The expected shape of this marshaller type
/// </summary>
public CustomTypeMarshallerKind MarshallerKind { get; }
/// <summary>
/// The size of the caller-allocated buffer in scenarios where using caller-allocated buffer is support.
/// </summary>
public int BufferSize { get; set; }
/// <summary>
/// Whether or not the caller-allocated buffer must be stack-allocated.
/// </summary>
public bool RequiresStackBuffer { get; set; }
/// <summary>
/// This type is used as a placeholder for the first generic parameter when generic parameters cannot be used
/// to identify the managed type (i.e. when the marshaller type is generic over T and the managed type is T[])
/// </summary>
public struct GenericPlaceholder
{
}
}
/// <summary>
/// The kind (or shape) of the custom marshaller type.
/// </summary>
public enum CustomTypeMarshallerKind
{
/// <summary>
/// This marshaller represents marshalling a single value.
/// </summary>
Value,
/// <summary>
/// This marshaller represents marshalling a Span-like collection of values
/// </summary>
SpanCollection
}
}The types used with MarshalUsing/NativeMarshalling or attributed with GenericContiguousCallectionMarshaller are expected to conform to a defined shape. Analyzers will be provided to validate their usage.
Naming:
As noted in #46838 (comment), these are three different attributes with three different forms of 'marshal'. We may want to rework the names to all use the same form of 'marshal'.
Usage Examples
NativeMarshallingAttribute
The [NativeMarshalling] attribute is applied on a struct and points to a struct that does not require marshalling or a struct with a Value property that does not required marshalling (as per the StructMarshalling design doc). This attribute indicates the default marshalling for the type when passed to a method that uses source-generated interop such as the DllImportGenerator source generator.
[NativeMarshalling(typeof(StringContainerNative))]
public struct StringContainer
{
public string str1;
public string str2;
}
[CustomTypeMarshaller(typeof(StringContainer))]
public struct StringContainerNative
{
public IntPtr str1;
public IntPtr str2;
public StringContainerNative(StringContainer managed)
{
str1 = Marshal.StringToCoTaskMemUTF8(managed.str1);
str2 = Marshal.StringToCoTaskMemUTF8(managed.str2);
}
public StringContainer ToManaged()
{
return new StringContainer
{
str1 = Marshal.PtrToStringUTF8(str1),
str2 = Marshal.PtrToStringUTF8(str2)
};
}
public void FreeNative()
{
Marshal.FreeCoTaskMem(str1);
Marshal.FreeCoTaskMem(str2);
}
}MarshalUsingAttribute
The [MarshalUsing] attribute on a parameter, return type, or field indicates the marshaller type to use for the parameter, return value, or field in this particular case. This choice overrides any default marshalling rules, including the [NativeMarshalling] attribute.
[GeneratedDllImport("NativeLib", EntryPoint = "get_long_bytes_as_double")]
public static partial double GetLongBytesAsDouble([MarshalUsing(typeof(DoubleToLongMarshaller))] double d);
[CustomTypeMarshaller(typeof(double))]
public struct DoubleToLongMarshaller
{
public long l;
public DoubleToLongMarshaller(double d)
{
l = MemoryMarshal.Cast<double, long>(MemoryMarshal.CreateSpan(ref d, 1))[0];
}
public double ToManaged() => MemoryMarshal.Cast<long, double>(MemoryMarshal.CreateSpan(ref l, 1))[0];
public long Value
{
get => l;
set => l = value;
}
}Example of code that could be generated:
[System.Runtime.CompilerServices.SkipLocalsInitAttribute]
public static partial double GetLongBytesAsDouble(double d)
{
long __d_gen_native;
double __retVal;
//
// Setup
//
global::SharedTypes.DoubleToLongMarshaller __d_gen_native__marshaller = default;
//
// Marshal
//
__d_gen_native__marshaller = new(d);
__d_gen_native = __d_gen_native__marshaller.Value;
//
// Invoke
//
__retVal = __PInvoke__(__d_gen_native);
return __retVal;
//
// Local P/Invoke
//
[System.Runtime.InteropServices.DllImportAttribute("NativeLib", EntryPoint = "get_long_bytes_as_double")]
extern static unsafe double __PInvoke__(long d);
}The CountElementName and ConstantElementCount properties can be used to simply specify collection count information without specifying a native type.
[GeneratedDllImport("NativeLib")]
[return: MarshalUsing(ConstantElementCount = 10)]
public static partial byte[] Method(
[MarshalUsing(CountElementName = "outSize")] out byte[] outBuffer,
out int outSize);SpanCollection shape
The [MarshalUsing] or [NativeMarshalling] attribute can be used to point to a type marked [CustomTypeMarshaller(typeof(...), CustomTypeMarshallerKind.SpanCollection)] to support custom marshalling of contiguous collections.
[GeneratedDllImport("NativeLib")]
public static partial void Method([MarshalUsing(typeof(ListMarshaller<int>))] List<int> values);
[CustomTypeMarshaller(typeof(List<>), CustomTypeMarshallerKind.SpanCollection)]
public unsafe ref struct ListMarshaller<T>
{
private List<T> managedList;
private readonly int sizeOfNativeElement;
private IntPtr allocatedMemory;
public ListMarshaller(int sizeOfNativeElement)
: this(null, sizeOfNativeElement)
{ }
public ListMarshaller(List<T> managed, int sizeOfNativeElement)
{
allocatedMemory = default;
this.sizeOfNativeElement = sizeOfNativeElement;
if (managed is null)
{
managedList = null;
NativeValueStorage = default;
return;
}
managedList = managed;
int spaceToAllocate = Math.Max(managed.Count * sizeOfNativeElement, 1);
allocatedMemory = Marshal.AllocCoTaskMem(spaceToAllocate);
NativeValueStorage = new Span<byte>((void*)allocatedMemory, spaceToAllocate);
}
public Span<T> ManagedValues => CollectionsMarshal.AsSpan(managedList);
public Span<byte> NativeValueStorage { get; private set; }
public void SetUnmarshalledCollectionLength(int length)
{
managedList = new List<T>(length);
for (int i = 0; i < length; i++)
{
managedList.Add(default);
}
}
public ref byte GetPinnableReference() => ref NativeValueStorage.GetPinnableReference();
public byte* Value
{
get => (byte*)Unsafe.AsPointer(ref GetPinnableReference());
set
{
if (value is null)
{
managedList = null;
NativeValueStorage = default;
}
else
{
allocatedMemory = (IntPtr)value;
NativeValueStorage = new Span<byte>(value, (managedList?.Count ?? 0) * sizeOfNativeElement);
}
}
}
public void FreeNative() => Marshal.FreeCoTaskMem(allocatedMemory);
}Analyzer Diagnostics
An analyzer will be provided to validate the usage of these APIs as per the design are defined in https://github.com/dotnet/runtime/blob/29cc2a2c23ce15469575a0b25c8b8b453a1d975f/src/libraries/System.Runtime.InteropServices/gen/DllImportGenerator/Analyzers/ManualTypeMarshallingAnalyzer.cs
These will focus on ensuring the types specified in MarshalUsingAttribute or NativeMarshallingAttribute and attributed with CustomTypeMarshallerAttribute conform to the expected shapes for use with p/invoke source generator. The generator will also error if the types do not conform to the expected shapes:
// Shape for Value kind
[CustomTypeMarshaller(typeof(ManagedType))]
public struct Marshaller
{
// Support for marshalling from managed to native.
// Required if ToManaged() is not defined.
// May be omitted if marshalling from managed to native is not required.
public Marshaller(ManagedType managed) {}
// Optional.
// Support for marshalling from managed to native using a caller-allocated buffer
// Requires the BufferSize field to be set on the CustomTypeMarshaller attribute.
public Marshaller(ManagedType managed, Span<byte> buffer) {}
// Support for marshalling from native to managed.
// Required if constructor taking TManaged is not defined.
// May be omitted if marshalling from native to managed is not required.
public ManagedType ToManaged() {}
// Optional.
// If defined, Value will be passed to native instead of the Marshaller itself
// If setter is defined, Value will be set when marshalling from native to managed.
// NativeType must not require marshalling
public NativeType Value { get; set; }
// Optional.
// If defined and Value is defined, will be called before Value property getter and its return value will be pinned
public ref NativeType GetPinnableReference() {}
// Optional.
// Release native resources.
public void FreeNative() {}
}
// Shape for SpanCollection kind
[CustomTypeMarshaller(typeof(GenericCollection<>), CustomTypeMarshallerKind.SpanCollection)]
public struct GenericContiguousCollectionMarshallerImpl<T>
{
// Support for marshalling from native to managed.
// May be omitted if marshalling from native to managed is not required.
public GenericContiguousCollectionMarshallerImpl(int nativeSizeOfElement);
// Support for marshalling from managed to native.
// May be omitted if marshalling from managed to native is not required.
public GenericContiguousCollectionMarshallerImpl(GenericCollection<T> collection, int nativeSizeOfElement);
// Optional.
// Support for marshalling from managed to native using a caller-allocated buffer
// Requires the BufferSize field to be set on the CustomTypeMarshaller attribute.
public GenericContiguousCollectionMarshallerImpl(GenericCollection<T> collection, Span<byte> buffer, int nativeSizeOfElement);
// Required.
// Points to the memory where the managed values of the collection are stored (in the marshalling case) or should be stored (in the unmarshalling case).
public Span<CollectionElement> ManagedValues { get; }
// Set the expected length of the managed collection based on the parameter/return value/field marshalling information.
// Required for unmarshalling (native to managed) support.
public void SetUnmarshalledCollectionLength(int length);
// Required.
// Points to the memory where the native values of the collection should be stored.
public unsafe Span<byte> NativeValueStorage { get; }
// Required.
// NativeType must not require marshalling
public NativeType Value { get; set; }
}Risks
This proposal relies on a definition of 'does not require marshalling'. The DllImportGenerator will require DisableRuntimeMarshallingAttribute (assembly-level) for any user-defined types in p/invokes and emit an error if the attribute is not present. This also means that any usage of the DllImportGenerator will require switching over the entire assembly (rather than individual p/invokes) to use the proposed MarshalUsing or NativeMarshalling attributes.
The successful usage of these attributes also rely on the user defining types that conform to a certain shape, which can be a bit complicated - especially in the SpanCollection case. The UX around writing those types might not be great, as walidation is only provided through analyzer diagnostics and generator errors.
Fixes
Fixes #8719