[vm/ffi] int, long, size_t, uintptr_t, wchar_t, etc

[dcharkes]

Update 2022-01-20: This will be available in dart:ffi from Dart 2.17 (and from now on tip of tree). In the mean time this is also available in package:ffi for Dart 2.16 (dev release).

Update 2021-01-07: We will implement this by implementing #42563 and probably adding the common types to package:ffi.

Currently dart:ffi only supports:

• int8_t
• int16_t
• int32_t
• int64_t
• uint8_t
• uint16_t
• uint32_t
• uint64_t
• intptr_t
• float
• double

However, many C APIs use C types such as int, long, size_t, uintptr_t, and wchar_t.

We could support these types. The question is if we support these types, then what other types should we support.

An alternative could be to not support these types but provide a way for users to specify types which differ in size per platform.

@jonasfj @sjindel-google

[dcharkes]

sizeof(int) on iOS arm64 returns 4. Thus, we cannot safely substitute int with intptr_t.

[artob]

From a user perspective, the lack of these standard variable-sized types is probably the single largest remaining deficiency in Dart's FFI support. It does rather complicate making FFI bindings for Dart.

As someone routinely working on a number of FFI bindings for a multitude of different languages (most recently, OpenXR bindings), the specific list of missing, absolutely essential variable-sized types in Dart is:

• int and unsigned int
• long and unsigned long
• size_t (stddef.h)

And some other, somewhat rarer variable-sized types to consider supporting would be:

• off_t (sys/types.h)
• ssize_t (sys/types.h)
• wchar_t (stddef.h)

All the aforementioned types listed in priority order, in terms of how badly and frequently I have needed them.

What's the plan for supporting at least int, long, and size_t?

[dcharkes]

These types are not part of the ABI, and the C standard has a very flexible specification (see this quick explanation). So, a compiler can choose what size/alignment these types have.

dart:ffi needs to be able to know what the size/alignment of types is of the compiled shared object loaded into Dart. The DartVM knows what OS (Linux,Windows,MacOS,iOS,Android) and hardware (x64,ia32,arm32,arm64) it is running on, so it can rely on the ABI and use the types specified in the ABI.

However, dart:ffi does not know with what compiler (and what compiler options) the shared object that is being loaded is compiled. So it can not know how to treat these C types. (Adding a possibly endless list of compilers/compiler options to DartVM is probably not a good approach, because it would require us to change the DartVM every time someone tries to use a new combination.)

It might be the case that for some of these types, all the compilers targeting the various ABIs (OS/Hardware combinations) agree on a size/alignment. Then we could incorporate these types into the ABI-logic. The downside is that if ever some new compiler comes up that does not respect the status quo, stuff breaks.

Does anyone have experience with whether the most common compilers agree on these types are handled in specific ABIs? Do we have any prior art in FFIs in other languages?

Another direction we could go to is to pass in the specs for these types into a bindings generator. The downside here is that one needs to re-run the bindings generator for every compiler/compiler options/hardware/OS combination when releasing an app.

@artob, what are you currently doing to work around this limitation?

[artob]

dart:ffi needs to be able to know what the size/alignment of types is of the compiled shared object loaded into Dart. The DartVM knows what OS (Linux,Windows,MacOS,iOS,Android) and hardware (x64,ia32,arm32,arm64) it is running on, so it can rely on the ABI and use the types specified in the ABI.

However, dart:ffi does not know with what compiler (and what compiler options) the shared object that is being loaded is compiled. So it can not know how to treat these C types. (Adding a possibly endless list of compilers/compiler options to DartVM is probably not a good approach, because it would require us to change the DartVM every time someone tries to use a new combination.)

@dcharkes Respectfully, you might be overthinking this. Just about any FFI for any programming language has the same theoretical problem, yet they all provide these FFI types.

It might be the case that for some of these types, all the compilers targeting the various ABIs (OS/Hardware combinations) agree on a size/alignment. Then we could incorporate these types into the ABI-logic. The downside is that if ever some new compiler comes up that does not respect the status quo, stuff breaks.

Does anyone have experience with whether the most common compilers agree on these types are handled in specific ABIs? Do we have any prior art in FFIs in other languages?

Every single FFI in every single programming language I've worked with--with the sole exception of Dart--supports these basic types: at the very least, int, long, and size_t are universal. Off the top of my head, here's a partial list of FFIs I have prior experience with, and how they map the types I mentioned:

• Common Lisp CFFI: :int, :long, :size-t
• Haskell FFI: CInt, CLong, CSize, CSsize, CWchar
• Java JNA: int, NativeLong, Structure.FFIType.size_t, platform.linux.XAttr.ssize_t, char
• Julia: Cint, Clong, Csize_t, Cssize_t, Cwchar_t
• Node.js FFI: int, long, size_t
• Python ctypes: c_int, c_long, c_size_t, c_ssize_t, c_wchar
• OCaml ctypes: int, long, size_t, ssize_t
• Racket FFI: _int, _long, _size, _ssize, _wchar
• Ruby FFI: :int, :long, :size_t, :ssize_t, :wchar_t
• Rust FFI: c_int, c_long, size_t, ssize_t

I omitted Go, LuaJIT, Nim, PHP, and Zig from the list, since their FFI implementations all parse C header syntax directly (and implicitly support these types).

@artob, what are you currently doing to work around this limitation?

I'm not. This, plus #35763, is blocking my efforts for nontrivial FFI bindings for Dart and Flutter. It would be a royal pain in the arse to work around both of these blockers, so I'm focusing my efforts elsewhere for now and hoping to come back to all this after a future Dart release announcement.

[artob]

@artob, what are you currently doing to work around this limitation?

Oh, and I might as well mention this here as an aside given that I haven't found an existing issue for it: the lack of general type aliases--for example, defining CLong as a typedef for either Int32 or Int64 at compile time--also hampers creating FFI bindings.

[dcharkes]

Oh, and I might as well mention this here as an aside given that I haven't found an existing issue for it: the lack of general type aliases--for example, defining CLong as a typedef for either Int32 or Int64 at compile time--also hampers creating FFI bindings.

See dart-lang/language#65 for that.