Add ExtensionDevelopmentPlugin for in-editor native extension development

[reduz]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

Extension development is relatively complex. It requires downloading a separate set of headers, generating them, setting up a compiler toolchain, etc. This happens independent of the usage of Godot from the editor, so development can be relatively difficult.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The idea is to have something called an "ExtensionDevelopmentPlugin" that eases the project generation, compilation, updating the headers and bindings. This way, everything can be properly done from the editor UI.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

The new ExtensionDevelopmentPlugin will handle extension development. To develop an extension, a tab in the ProjectSettings will allow specifying project folders as well as the plugin used. As an example:

Path	Language
res://myextension/terrain	C++ [edit]
res://anoherextension	C# [edit]
res://yetanother	Rust [edit]

Creating an extension will make the plugin set up the proper toolchains, debug protocol, build system, build command, and everything needed. If something is not present, the plugin will verify this and let you know that its misconfigured (likely showing an error icon). Running the project, the plugin will also attempt to compile the extension and connect to an IDE for debugging.

It is important to note here that extensions that you already download compiled are not managed or tracked by this system. This system is only for extension development. That said, however, the new extension development plugin system makes the extensions in development interact with the other extensions installed by generating and providing the relevant headers (or API depending on the language) for their use.

As an example, in the config menu for an extension in development, you will be able to see a list of extensions installed and tick which ones you want to have access to. In C++ this means that the proper headers will be generated for using this extension. In C#, it means the proper .cs classes will be created to interact with it.

Finally, the extension plugin will also make sure to compile as many platforms as possible on deploy (in the case of compiled languages) but it will be the developer responsibility to install the required toolchains (there is a separate discussion ongoing about supporting GitHub CI for being able to build for every single platform, as well as also using webassembly optionally to ensure what is compiled runs everywhere, but I think this is material for other proposals).

Stock Godot will only come with support for C++ and C# development, but it is expected that you can download these plugins from the asset library, allowing support for development of other languages such as Nim, Rust, etc.

If this enhancement will not be used often, can it be worked around with a few lines of script?

This is core functionality.

Is there a reason why this should be core and not an add-on in the asset library?

This feature allows the creation of add-ons.

[fire]

I recommend the use of the https://github.com/mstorsjo/llvm-mingw package for toolchains for systems other than for mac. There's a different discussion about how to build for mac.

Benefits:

1. Small
2. Supports MSVC dbg symbols via a flag
3. Fast!
4. Can cross compile
5. We can distribute the package on the asset library.

[reduz]

@fire In this case, I think the user will be in charge of installing this, but our build templates should definitely support it.

[fire]

The idea is the user can go to the asset library, click download and start immediately developing Godot extensions in like c++.

There is no need to also then install msvc and click 5 windows or similar for mingw.

[reduz]

@fire Okay but in practice you have to remember that

• On Linux you install toolchains from package manager, which depends on distribution
• On MacOS you install toolchain from asset store
• On Windows you install manually, and some developers prefer MSVC over alternatives.

We can refer to the proper Godot documentation page for this, and the plugin can also autodetect toolchains, but user work is unavoidable here. At least it only needs to be done once, since this would be an editor wide setting.

[Ansraer]

What about just having the plugin download a docker container with the necessary build tools? This would make it possible for users to get everything they need to develop e.g. c++ with literally a single click.
In theory this would also make debugging and support easier, since everyone will use the exact same version of everything.

[reduz]

@Ansraer sure that could work to some degree, but keep in mind we can't redistribute legally a lot of stuff, like Android toolchain or MacOS/iOS headers. Would still need to do scripts for users to create those themselves.

This is in part why, if your extension does not really use anything native, webassembly is a very attractive option for deploy.

[Ansraer]

True, we could probably have some sort of setup wizard that guides the user through this. Actually having a window that tells me what I need to download, with a next button I can only press once I placed the downloaded file in the correct folder actually sounds like a really nice idea.

Another reason why I would like to use docker containers is libraries and compilers. At least master itself is currently incompatible with a bunch of compilers and libraries. Getting my dev environment for it setup and working took me a few days. The situation might be better when you only want to build gdextensions, but given how cumbersome it is to get anything installed on windows, and how many linux distros ship very outdated versions the idea of a clean controlled environment is very tempting imo.

[theraot]

Please don't make Docker mandatory.

As an option, it is ok. I'm sure it will get use.

Yet, I will not use it. Last time I tried Docker on my machine performance was very bad. Yes, I'm overdue an upgrade, but reasons, ok? Instead, I'll set up the development environment. I had no problem following the instructions from the documentation to build Godot.

I have seen the idea come up before. And I said the same thing: Please don't make Docker mandatory.

On the flip side, you could provide YAML files for CI platforms. For instance I had good experience with AppVeyor for other projects (It is a good fit for .NET projects because of its NuGet integration among other things), and you can put that on Docker if you want to. It does not have to be AppVeyor, it could be Jenkins (I think that is better for Java), it could be Github actions, or something else. I don't know what solution is a better fit for Godot and its community.

[barbaros83]

i see the comments sugesting a compiler be shipped with godot or the plugin, but if this gets you to the point where you could compile the extension with 1 command i think it should already be easy enough for most people that want to write their own extensions

[reduz]

@barbaros83 Downloading a compiler for this is literally no effort and, given large part of the Godot user base will just not need this, including it with the default download would be a large waste of disk, time and bandwidth. Don't want to go the Unity way here.

[Ansraer]

@theraot Are you sure that performance problem was actually dockers fault? Dockers overhead is usually only ~5%. There are some rare cases when it can get up to 18%, but that is caused by the security measures against attacks like spectre (which only really matter when you are running on a server). If you run docker with all security features disabled you almost never get a performance impact higher than 2%.
These days I am building a LOT of my code exclusively in containers, and I never encountered any slowdown when compared to running it directly.

[theraot]

@Ansraer Listen, I blame my hardware. 5% of what? You are assuming my machine is comparable to yours. Regardless, perhaps Dockers would run better for me now (It is a more recent version of Docker on a more recent version of the OS after all), or in the future with whatever settings. I would still believe Docker should not be mandatory.

[aaronfranke]

This proposal is great! Having Godot download/generate the files needed and then only having the users supply a compiler and an IDE would be ideal. It would make GDExtension development comparable to the C# support, much more approachable and much easier to know you're doing it "correctly". We should make it very easy to get started with a "Hello World" extension.

Stock Godot will only come with support for C++ and C# development, but it is expected that you can download these plugins from the asset library, allowing support for development of other languages such as Nim, Rust, etc.

Would it be possible to design the system so that it searches for a special category on the Godot Asset Library, so that all languages can be easily grabbed from one menu in Godot?

[reduz]

@aaronfranke yes sure, that is precisely the idea

[eon-s]

The editor can have a basic "toolchain check" tool for GDE with a set of common configurations and guides, then updates to that tool could be added via addons to keep it up to date.

[neikeq]

I'll comment on some parts, but I don't really understand what this proposal is about and what you have in mind.

Creating an extension will make the plugin set up the proper toolchains, debug protocol, build system, build command, and everything needed.

We already have an editor plugins that manages build system and exporting for C#. Other languages could benefit from it though, but maybe they could also implement it as an editor plugin like it's done with C#.

As an example, in the config menu for an extension in development, you will be able to see a list of extensions installed and tick which ones you want to have access to. In C++ this means that the proper headers will be generated for using this extension. In C#, it means the proper .cs classes will be created to interact with it.

The generation of proper C# classes for GDExtension classes is something I will need. However, I want the C# build system to work even when run from the command line or an IDE, not only when launched from Godot. As such, I will likely make a MSBuild task that runs Godot to generate the api.json and use that to generate the C# classes via source generators.

[Zylann]

Does this mean the Godot Editor would become mandatory to develop extensions?

Would this force a specific build system with specific C++ compiler options? (what if I want to use something other than SCons, or a different version of the C++ standard, or other flags?)

Also how would developping editor plugins work, especially when non-reloadable?

About docker containers: I have no experience in these, but so far I prefer using toolchains I have to do the main development. I might however use it when doing releases, which I do with CI at the moment. Also I wonder if such containers would be used solely for compiling? Because usually my C++ workflow is: "Build a debug library, launch Godot with debugger attached, eventually breakpoint, step and stop, modify code, restart". Do containers intervene in this?

An extra point (probably a bit niche): my own extension should be usable by other extensions, probably as described in this proposal, but as personal preference, I plan to also make it usable by source integration. What I mean by that, is that a project using C++ too can depend on it by compiling its sources with it (or statically linking), rather than through Godot class wrappers from a precompiled version. This has the big advantage of having full idiomatic access to the C++ API of the extension, without any limitation from Godot's API, and without ABI compatibility mess. The "project" I refer to here would very likely be a game, and it would result in a single extension binary being compiled, even though the filesystem would still have two, each in their own folder. Would ExtensionDevelopmentPlugin hamper such a setup?

[kiririn39]

Sounds really great!
This is a killer feature for me and i think it will also attract more c++ devs (maybe from unreal? who knows) which may lead to more engine contributors.

Reasoning part:
Just to remind to people who thinks that c/c++ is only for performance and because of this Godot doesnt need a good c++ support. Actually that is only part of it's usecase. First of all it's about writing game in a language of your choice.
Another one would be to study c++ in a form of making games. This one is partially relatable to me. Although I'm a c++ dev i do not like the language and ecosystem. But i do like what kind of possibilities it gives me. The main reason why i want to use it in Godot is to actually is to enjoy the process of making a game (not to actually ship it, at least not now) AND improve as a programmer.

My view of implementation:
As of now (Godot 4.1 beta) these are steps user has to in a nutshell: (i may be wrong here or there, please correct me if do)

1. get the engine
2. generate api
3. get scons and stuff
4. get godot-cpp
5. combine godot-cpp, and ones actual script code in a single project
6. place appropriate platform binaries in a godot (if part 5 was done outside of godot directory)

3rd one can be easilly done by a single shell command and you only need to do it one. While Godot docs can provide an example string for less technmically savvy users, more advanced can easily configure it for their needs. i'm talking about getting specific compilers that one need
The most hard part for me by far is 5. As it involves making your own solution.

The most ideal for me would be:
0. user gets scons and stuff

1. In editor one should choose a button to create new script "module", enter name, path and some other minor config and if they want to generate any solution files like vs or cmake. Editor downloads a godot-cpp inside a project folder (I dont see any problems here. It's not like godot-cpp a third party or something) Besides godot project target specific engine version the same true for godot cpp.
2. For each script 'module' editor generates subfolder with optional visual studio or cmake solution with prepared entry point
3. hot reload changed c++ script "module"

Probably thats it. users can then open solution and press build button, or reconfigure their compilers. The only downside i can see is that a project folder will become polluted with intermediate files. Well, that can be worked around with gitignore
Basically editor handles the most complex parts up to making a working crossplatform solutions. Any platform specifics configs users should handle themselves.
On windows installing a msvc toolchain is super easy using vs installer. On MacOs i think it's also done using single command if you want to use a-clang. On Linux, well, package managers or something. Basically user should fill up a few platform or toolchain specific configs

[Calinou]

@Barugon Please don't bump issues without contributing significant new information. Use the 👍 reaction button on the first post instead.