GDExtension library cannot be reloaded while editor is running

[Bromeon]

Godot version

4.0.dev (b770fa2)

System information

Windows 10

Issue description

On Windows, the Godot editor "locks" a DLL containing a GDExtension library and releases it only after shutdown. Native code can thus not be recompiled as long as the editor remains open.

I'm not exactly sure about the current behavior on Linux and Mac. A user reported the editor would immediately crash upon swapping a .so file (see below), but I already heard that people managed to get the launched game running with an updated library. What is the state here?

This is a considerable limitation for game developers actively using GDExtension. Unlike add-ons/tools that exist during the whole lifetime of the editor, GDExtension for games lives from being frequently updated. Requiring the user to reload the editor on every change not only makes a very common workflow impossible, but is also a regression from GDNative, where reloading (for non-tool classes) could be achieved while the editor was out of focus -- even if it had its bugs.

---

Now, it isn't my style to complain without trying to find a common solution 😉

I think this problem can be split into two parts, of which the first is likely easier to achieve and might already mitigate some problems:

1. Do you think it's feasible to enable a "library swap" feature that would at least let the launched game make use of recompiled native code? The extension library loaded by the editor itself could remain the same.

2. In the original GDExtension PR, hot reloading was listed under TODO. This feature has the potential to make GDExtension a true first-class citizen, and would likely encourage a huge amount of plugin and game development in the Godot ecosystem. In particular, it would make Godot much more attractive to C++ developers.

I'm fully aware that true hot-reloading is very difficult to achieve, as such I'd appreciate some insights on the topic from Godot developers. Have there already been (rough) plans or ideas how such a feature might play along with the ClassDB? If not, would it be appreciated if I offered my help in working out a high-level design together?

Some challenges I see:

• A mechanism to mark instances of unloaded/reloaded classes.
  Define their behavior when referenced from GDScript code, scenes or Godot caches.
• A specified lifecycle, clarifying which role engine and extension play in the init/shutdown.
  User callbacks to invoke for cleanup code.
• Eventually, a safe way to reload/unload without UB.

---

This has been brought up before, but so far without a more detailed discussion. I'll try to use this issue also to summarize the efforts (but can also switch to godot-proposals in case we go into design).

Godot 3 (GDNative):

• Godot 3.3 editor locks dll for writing (GDNative) #48086
• Allow manual unloading/reloading of GDNative/GDExtention lib files (DLL, etc) godot-proposals#5324

Godot 4 (GDExtension):

• [GDExtension] updating library during editor opened, crashes the engine godot-cpp#638
• Implement live reloading in GDExtension godot-proposals#4437

Steps to reproduce

Recompile a GDExtension native library while the Godot 4 editor is open.

Minimal reproduction project

No response

[Calinou]

1. Do you think it's feasible to enable a "library swap" feature that would at least let the launched game make use of recompiled native code? The extension library loaded by the editor itself could remain the same.

This can likely be achieved by having a list of library path remaps passed as a CLI argument to the project run from the editor, with new paths being temporary locations that will never be overwritten.

[Waridley]

The hardest part of this seems to be invalidating all of the pointers the library hands to the engine, like init function pointers, GDNativeExtensionClassCreationInfo, Extension Class method pointers, etc. All references to the library need to be dropped before the library can be unloaded.

Edit: Actually my first examples aren't that hard to deal with, it's mostly about objects like nodes or custom servers.

[Bromeon]

Very rough idea, how the new DLL could at least be used for a game's launch (1st problem):

1. Godot extension config file mentions path/to/library.dll
2. Compiler generates path/to/library.dll
3. Godot keeps a watch on that file, to be notified about changes
4. On a change, Godot copies path/to/library.dll to .godot/libs/library-8172389312837.dll and loads that
5. Next time, it will be .godot/libs/library-441902036275.dll

Main takeaway: Godot never directly loads path/to/library.dll, so the lock does not exist.

Other ideas:

• just move the file (may interfere with compiler)
• copy entire DLL into memory (needs more RAM; big waste if only a small part of the library is used)
• actively release the previously loaded library on events like focus-lost
  • might need hooks to notify the extension about unloading (i.e. solution to 2nd problem)

Is something like that realistic?
Are file watches already implemented in Godot, e.g. for editor purposes?

[EngineGuy]

Main takeaway: Godot never directly loads path/to/library.dll, so the lock does not exist.

I think it's very important for the loaded file to be debuggable in the same way as the compiled one. Not sure if this change might interfere with that.

[Bromeon]

@EngineGuy That's a very good point. 👍

The cleanest solution would probably be to release the lock (either manually triggered, or on FocusLost/Minimized events).

[MachineMakesNoise]

I tried to create a plugin for Godot that uses NativeExtensionManager class in GDScript to manually control loaded extensions.

You can unload the extension, compile new dll, replace the old one and then load the extension again. Now this does really work but it breaks the engine.

It either

• crashes after hitting play with new dll in place.
• breaks gdscript somehow (my lambdas that worked fine before just break with obscure errors, memory corrupted maybe?).
• outputs '"scene/resources/packed_scene.cpp:178 - Node './TestNode2D' was modified from inside an instance, but it has vanished."' and removes the whole extension node from saved scene (ouch, this should not happen right?!).

After these I just gave up. Unfortunately I need native extensions due to performance (and somewhat disliking GDScript) and restarting engine each time is annoying workflow for rapid development and IMO somewhat undermines the idea of dynamic GDExtensions.

I understand "hot reload" is hard to get right (UE as an example) and I don't think it even needs to be pursued but replacing libraries between running the game from editor should be doable without the "hot" magic?

edit: typos

[setzer22]

Now that we have https://github.com/godot-rust/gdextension/ I was able to play around a bit with the new system and, I don't want to sound too harsh, but GDExtension feels like a substantial usability downgrade for me as a game developer due to this issue. Let me elaborate:

The way I develop games with Godot 3.x using a native language (Rust in my case, but this applies just as well to C++ devs) is pretty much the same as one would do in regular GDScript: I write script classes, only that instead of using gd files, I use gdns files. This is and incredibly productive setup, because I get the full editor experience, but all my game logic is written in a language that makes refactoring and scaling my game much easier. Note that my use case for a native language has nothing to do with developing editor extensions or speeding up critical pieces of game logic: My game logic is entirely written in a native language so I can benefit from the mature static analysis tooling around it.

Now, I know I'm not the only one using Godot like this, but I can only speak for myself. My issue with the transition to GDNative to GDExtension is that it takes away value from me as a user without really giving me much in return. Let me explain with an example:

In Godot 3.x, this is how I work when I want to create some new behavior:

• I start by creating a new rust file, writing a new struct (a godot class), and exposing some of those struct methods to godot. It looks like this. That link is a Zeppelin character controller. Does that read lightweight and simple like a version of GDSscript with curly braces and static typing? Well, that's how it feels to me 🙂.
• Next, I jump back to the editor, create a new node (in this case, a Spatial), and use the asset finder to locate the Zeppelin.gdns file. I drag and drop this file into the node, and the script is set automatically. Yup, I didn't have to do anything else, it's automated: The gdns file was generated by a build script and my dll was recompiled when saving the file (Rust has cargo watch, but C++ devs can use entr to recompile their code on file save).
• At that point, I can run the scene and start iterating on the logic. That process will typically consist of launching and closing the game multiple times, making small tweaks to the code. During this process, I usually add properties to my class (The equivalent of export vars in GDScript), and tweak those in the editor. It is very important for me to be able to add new properties on the fly, because it's impossible to predict which parameters I will want to expose beforehand.

This all works wonderfully and makes me feel as productive as it gets! But now, let's take a look at my pain points in Godot 4 via GDExtension:

• Creating a new class requires an editor restart. This is true even if the Windows DLL locking issue is solved. I'm not even on Windows. Unless GDExtension gets full hot-reloading like GDNative used to have, every time I create a new script I will have to restart the editor, loosing my state (all the open tabs, prompting to save unsaved changes I may not really want to save, ...). This breaks my flow and forces me to retreat to other styles of development that simply ignore the editor and use Godot as a rendering library, which IMO takes away most of its value.
• But even worse: Adding or removing properties also requires an editor restart. This makes exporting vars from my native scripts far less useful, when it used to be one of the biggest productivity boosters for me in 3.x.

I really don't mean any of this in a bad way. My hope is that, by explaining my workflow in detail, I will help developers understand why so many people seem to care about this issue (I mean, look at the upvotes!). There is a niche (yes, we're not that many, but we exist) of developers that has seen a considerable degradation in their workflow when transitioning from Godot 3 -> 4.

Basically, I just came to suggest that, even if the DLL locking bug is solved, what lies underneath is not the productive experience developers used to have in 3.x. I understand that might not be a priority, and I know I'm hardly Godot's target user by any means, but I'd like to raise awareness about why this is an issue and how it breaks developer's workflows exactly.

[antonWetzel]

I would like to have a Button in the Editor to compile (like C#?) which triggers

1. save all open scenes to a hidden location
2. unload "scripting" gdextension (free shared library)
3. trigger a compilation
4. reload "scripting" gdextension
5. reload all scenes from the hidden location

To work Godot needs some Information

• which gdextension to reload (path should be enough)
• how to compile (terminal command and working directory?)

Maybe a special resouces with the information is enough. The import process would trigger the workflow.
Extenal compilations could touch the resource file to automate the process. (somehow avoid recompilation)
(This feels more like a hack then a solution)

sava solution sounds similar, so I have no idea if this can work.

[dkaste]

@setzer22

Creating a new class requires an editor restart

But even worse: Adding or removing properties also requires an editor restart

This isn't exactly true (in my experience on Linux, without the file lock, at least). The editor doesn't reload the extension, but the game loads the updated one each time you launch it. Only if you need to create or modify those classes/properties in the editor do you need to restart (which admittedly does suck).

I have been using cargo watch -s scons -C . successfully to rebuild my godot-cpp extension with the editor open on Linux this way.

That being said, I think the experience on Windows should be at least as good as the current one on Linux.

Side note: Personally, I have found GDExtension most useful for code that needs to be fast and changes frequently, but isn't directly tied to the node system (i.e. classes that extend RefCounted). In my case, I use it for a binary protocol and dealing with large arrays. Then I use signals or function calls to pass data to GDScript that will handle the node stuff.

[setzer22]

The editor doesn't reload the extension, but the game loads the updated one each time you launch it.

Ah, sorry about that. I should've made it more clear in my wording. Yes, this matches my experience on Linux too with Rust. My main concern here is that for a workflow that is tied to the node system (the mainstream way to develop things in Godot), it gets much more difficult because creating a new script-like class requires an editor restart. Otherwise, there's no way to put that node on the scene. Well, technically there is, but it's not a comfortable way to do it.

I have found GDExtension most useful for code that needs to be fast and changes frequently, but isn't directly tied to the node system

Yes, I totally agree. This seems to be the main use case behind its design.