Proposal for changes in GDNative for Godot 4.0

[touilleMan]

While working on Godot-Python, I've encountered my share of quirks in the GDNative API, I guess it's time to fix them for Godot 4.0 ;-)

Issues in api.json:

• methods parameters of type StringName are exposed as String.
• BulletPhysicsServer not marked as singleton but inherits PhysicsServer
• Object.free is virtual (should use godot_object_destroy instead of calling it directly), but not info about it in api.json
• field instanciable renamed into instantiable ?
• missing Script.has_property, Script.has_method and Script.has_script_signal
• Some properties has / in there name to indicate they need an additional parameter (provided by the "index" field, which is set to -1 when the value is not needed...). Would be better to provide a additional_arguments parameter that take a list, or have a boolean has_index_argument field (in the style of the has_default_value boolean field)
• Object.set/get doesn't return a boolean to indicate if the operation couldn't succeed if the property doesn't exists
• default values are... strange. This is due to the fact they are formatted by there operator String() which often lack clarity (and is subject of innocent modifications breaking the api.json format !). It would be better to define those informations throught a custom string (this is already done in the doc xml files). Exemple of strange values:
  • [Object:null] and Null
  • [RID]
  • ((1, 0), (0, 1), (0, 0)) (should be Vector3((1, 0), (0, 1), (0, 0)))
  • PoolColorArray displayed as [PoolColorArray] but PoolVector2Array displayed as []
  • 1, 0, 0, 0, 1, 0, 0, 0, 1 - 0, 0, 0 for Transform here the - is especially misleading :/
• property PrimitiveMesh.material has type SpatialMaterial,ShaderMaterial. This is fine from a documentation point of view, but given the documentation already overwrite fields in it xml files, I guess it would be better to replace that by a simplerMaterial value
• missing Object.get_class_name (which returns a StringName, useful for fast retrieval of a wrapper class when getting a generic object from variant). This would bring a bit of confusion with Object.get_class though, I guess we could rename that into String Object.get_class_name() and StringName Object.get_class_name_as_string_name()

Issues in gdnative api:

• godot_quat_new_with_axis_angle inconsistent with godot_basis_new_with_axis_and_angle
• missing godot_basis_get_column (but oddly godot_basis_get_row is present)
• missing godot_array_operator_equal & godot_pool_x_array_operator_equal
• missing Transform2D origin/axis getter/setter
• missing godot_basis_axis (unlike godot_vector3_axis which is present)
• missing Transform2D(Transform)
• missing Transform(Basis)
• missing Transform(Transform2D)
• pluginscript api use godot_string everywhere where most of the time godot_string_name would be better suited (e.g. godot_bool godot_pluginscript_instance_desc_set_prop(godot_pluginscript_instance_data* p_data, godot_string* p_name, godot_variant* p_value)) (Use StringName in pluginscript's set/get_prop and add_global_constant #35812)
• ScriptLanguage::add_named_global_constant (exposed by pluginscript) api is weird: global constant can only be integer, but the function takes a variant as parameter
• missing pluginscript binding of ScriptLanguage::add_named_global_constant and ScriptLanguage::remove_named_global_constant. On top of that add_named_global and add_global_constant have misleading names (from what I understand, the difference between them is the later can have any type of value where the former is only an integer...)
• get_rpc_mode and get_rset_mode fields in godot_pluginscript_instance_desc are never used and should be removed (Remove useless pluginscript godot_pluginscript_script_desc.get_rpc/rset_mode fields #35811)
• godot_dictionary_operator_equal doesn't do key/value comparison but only test if the underlying hashmap is the same (Modify Array/Dictionary::operator== to do real key/value comparison #35816)
• Remove deprecated GDNative wrapper code (Is GDNATIVE_API_INIT & GDnative wrapper lib still used ? #27054)

Issues in ptrcall:

• return value need to be initialized by the caller even if it is always overwritten (see [GDnative] ptrcall returning godot_string need to initialized #34264)
• OS.get_static_memory_usage returns a uint64_t which is not compatible with godot_int (see [GDnative] ptrcall on OS.get_static_memory_usage cause segfault due to godot_int != uint64_t #34254, solved by GDNative: Make godot_int an int64_t #36163)

That's a big wishlist I guess ! I guess I've already found what I'll be working on at next week's GodotCon sprints ^^

[touilleMan]

related issues:

• Is GDNATIVE_API_INIT & GDnative wrapper lib still used ? #27054
• [GDnative] ptrcall returning godot_string need to initialized #34264
• [GDnative] ptrcall on OS.get_static_memory_usage cause segfault due to godot_int != uint64_t #34254
• [GDNative] Misleading godot_x_new_copy/operator_equal functions for array/dictionary/pool arrays #35542
• [GDNative] Crash when method returns reference objects #35609

[touilleMan]

another related PR #35795 (Fix return type for GDNative's godot_color_to_XXX64)

[karroffel]

• gdnative_init_options should include a Godot version field.

• The GDNative JSON API should include the Godot version as well.

• ALL generated struct names should include their version. Right now only "next-filed extensions" have the version in the name.

• Push more code about POD core types (such as Vector2/3, Color, Basis) into bindings. It seems like most bindings do not use many other functions other than the "construct" and "deconstruct" functions.
  For example the Rust bindings receive those values, then convert them to types from the euclid crate and then do all operations on that. So all "accessor" functions are unused. Similar for the C++ bindings.

  Those types would be defined as actual structs with proper members, for example

  struct godot_vector3 {
      float x;
      float y;
      float z;
  };

  Functions should never return those complex values directly but always by pointer. Languages like Haskell have troubles with functions returning unboxed data.

[raniejade]

Push more code about POD core types (such as Vector2/3, Color, Basis) into bindings. It seems like most bindings do not use many other functions other than the "construct" and "deconstruct" functions.
For example the Rust bindings receive those values, then convert them to types from the euclid crate and then do all operations on that. So all "accessor" functions are unused. Similar for the C++ bindings.

Those types would be defined as actual structs with proper members, for example

struct godot_vector3 {
    float x;
    float y;
    float z;
};

Functions should never return those complex values directly but always by pointer. Languages like Haskell have troubles with functions returning unboxed data.

Wouldn't this create API disparity between the native bindings and GDScript (core godot)? Going this route means will just create duplicate code across the bindings. Having a consistent API in terms of the interface and behaviour is pretty advantageous, specially for users not wanting to use GDScript. That's why my binding (https://github.com/raniejade/godot-kotlin) does not re-implement any of the core types, it just delegates everything to Godot. Users can always refer to Godot's official documentation regardless of what binding they are using.

[raniejade]

Pulling this in from the godot_headers repo. godotengine/godot-headers#65

[karroffel]

Yes, but generally there are idiomatic ways to handle linear algebra types in most languages. So it might be better to use common libraries that will work well with other libraries for example, which is what Rust does. When new functions are added to those types the GDNative API has to change. It takes some care to not break binary compatibility and also makes it harder for bindings to just correctly add new functionality. Also by going through the C API you have some unavoidable overhead, as no LTO or cross-language inlining works with runtime provided function pointers. For things that should be as easy as a simple `add` or `mov` you now need to convert arguments and return values from a C API. Of course, the behavior being the same as with Godot is useful, but this is not a technical problem but an ecosystem/idiomatic one. We could keep those functions, but I think in environments that already have more-or-less common ways to do those operations going through a C API with custom types is not very nice.

[karroffel]

• introduce the concept of constants to NativeScript so that scripts can create them

• make all object references be ObjectIDs and not raw pointers. This includes all calls to ptrcall or varcall

[raniejade]

I totally agree with the technical side of things, however, the ecosystem aspect is something that should not be dismissed as it could hurt the adoption of Godot. Learning resources available about Godot is quite sparse (relative to Unity and Unreal), transferability of those resources to other bindings is useful for people who prefers not to use GDScript (for various reasons).

[raniejade]

• Include documentation in the generated api.json. Various languages have different tools to show documentation (rustdoc, kdoc, ...), it will be nice to have the actual documentation of a class/method/property locally.

#39588 is another case of return value initialization inconsistency. In this case there is no way for API consumers to handle methods that return godot_array consistently without leaking memory (initializing pointers even when they're overwritten).

[Zylann]

In NativeScript:

• Rich registration: Correct me if I'm wrong, but godot_nativescript_register_method doesn't seem to allow for declaration of argument types, return type and names. Which leaves Godot and users completely in the dark when it comes to use registered methods. They are "trusted to use methods properly", with no reliable typing, auto-completion or static checks, runtime checks also being left to implement by every binding. When using GDScript at least, this is not the expected experience. Obscure issues like Objects instanced in c++ can't be bound to typed gdscript variables godot-cpp#430 could be related to this.

• C#: using rich registration as mentioned above, could allow better C# integration (if possible?), because right now I heard using a GDNative library from C# is really daunting. It's frustrating to see GDNative and C# are two built-in modules, yet they are two fragmented parts of the scripting system. This is one of the big reasons I'm still not all-in for providing C++ NativeScript plugins, preferring C++ modules instead for their universal reach (despite the hurdles of engine compilation). Such fragmentation is understandable for GDScript, but probably not for the engine's very language.

• Being able to register static functions would be nice.

• Add concept of editor/release. Currently GDNative libraries only come in one version that is supposed to be release. However there is no way to provide 2 versions for editor and release... that means any checks and editor-specific code we may want to write in a library will have to ship in the final game, and it's not clean. Right now all we can do is either tell users to recompile the library before exporting, or telling them to swap prebuilt libraries somehow.

@karroffel

make all object references be ObjectIDs and not raw pointers. This includes all calls to ptrcall or varcall

Why? Is it for safety? Wouldn't this incur a performance overhead? I have a library which uses set_pixel and get_pixel a lot on images. If I chose to use C++, I don't expect to have such indirection baked in every call.

Push more code about POD core types (such as Vector2/3, Color, Basis) into bindings.

I've seen that. In C++ I find there is no point to use method pointers to do linear algebra, other than having the exact same behavior, but the overhead becomes quite stupid. If the Godot math library can be copied around more easily, it's easier. For C++ only though, other languages may have to rewrite their own, or use other math libraries as long as they allow the same conventions as Godot