CMake: Tests crashing on Linux with "stack smashing detected"

[dsnopek]

Godot version

4.4.stable

godot-cpp version

a3f8921

System information

Ubuntu Linux 22.04

Issue description

This may be a me problem, but I'm not sure, so just making an issue to track it, just in case :-)

At the moment, if I use cmake to build the tests, and try to run them, I get this crash:

*** stack smashing detected ***: terminated
Godot Engine v4.4.stable.official.4c311cbee - https://godotengine.org

ERROR: Tests failed to complete

When I'm testing @enetheru's PRs, I do frequently run the tests, and I even tried rolling back to older versions that I thought I had tested, but at the moment they all crash, unless I go all the way back to 7d3870b

I tried doing a git bisect to find if there was a particular commit that broke it, but it's super hard to do because godot-cpp and Godot compatibility are intertwined and I'd need to pair with a Godot build from the same time period to be sure that that wasn't the cause of the crashes.

(Git bisect did conclude that 5c9529f was the first bad one, but I tried modifying current master to roll back those changes and use classdb_register_extension_class3 - it still crashed with the "stack smashing" error. So, I think my bisect was tainted by not matching with older Godot builds?)

However, if I build the tests using scons, then they run fine :-/

It's super weird!

Steps to reproduce

I'm building the tests like this:

mkdir cmake-build
cd cmake-build
cmake .. -DGODOTCPP_ENABLE_TESTING=YES -G Ninja
cmake --build . -target godot-cpp.test.template_debug
cd ..

... and then running them like this:

cd test
GODOT=/path/to/godot ./run-tests.sh

I'd love if anyone else on Linux could test and see if this is just me, or if there's a real problem here :-)

Minimal reproduction project

This is with the tests that come with the godot-cpp repo in the test/ directory

[enetheru]

I have a hunch, can you test for me compiling in Release mode like so:

cd godot-cpp
cmake -S . -B cmake-build -DGODOTCPP_ENABLE_TESTING=YES -DCMAKE_BUILD_TYPE=Release -G Ninja
cmake --build cmake-build -t godot-cpp-test
cd test
GODOT=/path/to/godot ./run-tests.sh

Using the -S <source_dir> and -B <build_dir> is just cosmetic.

[enetheru]

I've had some time to compile godot and test.

I think there really is a problem for me to solve
I'm using godot-cpp master
using Godot 4.4-stable-official, and my own compile of 4.4.1 with I believe harmless extras.

Build configuration is template_debug, with build_profile set to speed things up. but no other changes.

Scons compiled dll:
scons build_profile=build_profile.json

• godot -e --path project - Loads editor fine, Play: Test Passed

• godot --path project - Test Passed

• godot-4.4.1-custom -e --path test\project\ - Loads editor fine, Play: Test Passed

• godot-4.4.1-custom --path test\project\ - Test Passed

CMake compiled dll:
cmake -S . -B cmake-build --fresh -DGODOTCPP_ENABLE_TESTING=YES -DGODOTCPP_BUILD_PROFILE=".\test\build_profile.json"
cmake --build cmake-build -t godot-cpp-test

• godot-4.4 -e --path test\project\ - Loads editor fine, Play: No output

• godot-4.4 --path test\project\ - No output

• godot-4.4.1-custom -e --path test\project\ - Loads editor fine, Play: No editor output, console shows backtrace

• godot-4.4 --path test\project\ - Crash with Backtrace

Loaded with a godot with debug symbols, my stacktrace looks nothing like what you describe, so I'm going to go do more testing.

[enetheru]

My problem was from cmake adding /RTC1 in debug builds which is an msvc flag for runtime error checking for stack issues.
This used to be present i believe in older versions. I have a PR for it, but I dont know how much of a solution it is, feels like hiding the real problem.

[enetheru]

OK, so we discussed and discovered that adding /rtc1 to scons also causes the bug to appear.

added /RTC1 to godot-cpp

Godot	Godot-cpp
4.3-stable	4.3-stable	OK
4.4-dev2	4.3-stable	OK
4.4-stable	4.3-stable	OK
4.4.1-stable	4.3-stable	OK
4.5-dev1	4.3-stable	OK
---	---	---	---
4.4-dev2	4.4-stable	FAIL for other reasons
4.4-rc1	4.4-stable	FAIL
4.4-stable	4.4-stable	FAIL
4.5-dev1	4.4-stable	FAIL
---	---	---	---
4.4-dev2	9b98b19	OK	Update bindings to 4.4.dev2

This is what I have so far

[enetheru]

Godot	Godot-cpp
4.4-dev2	9b98b19	OK	Update bindings to 4.4.dev2
4.1-rc1	9b98b19	OK
4.5-dev1	9b98b19	OK
4.5-dev1	012b8ff	OK	CMake: Alleviate target name clashes, visibility, and grouping.
4.4-stable	012b8ff	OK	This cmake commit is the last successful one
4.4.1-stable	012b8ff	OK
4.4-stable	65046d0	FAIL	Update for virtual method compatibility system
4.4.1-stable	65046d0	FAIL
4.5-dev1	65046d0	FAIL

Which is the conclusion you came to already, and by the looks of it it just implements the changes for the updates in godot.
So the next step is figuring out why godot is smashing the stack. which I honestly have no idea how to go about.

[enetheru]

I also got a failure if i reverted 65046d0 , so i created a branch at 65046d0, reverted it, and then replayed all the commits after and it works. :|

If I then cherry pick the commit, no matter what I api file i dump I either cant get it to compile because of missing hash values when performing code generation, or it exhibits the same error.

[enetheru]

OK but hows this.. in my branch without 65046d0, i can dump godot-4.5-dev1 api, and compile, and it still exhibits the same error. WTF, so its the changes introduced in the api.json, and not necessarily the virtual method compatibility system? I'm just not that familiar with how this all fits together, its hard to understand.

This stacktrace is from my custom godot-4.4.1 build using the api dumped from it. with the godot-cpp branch with all changes except the virtual method compatibility. this is the only debug build i have on hand, all the other tests were with godot-4.5-dev1 or godot-4.4.1-stable

Godot Engine v4.4.1.stable.custom_build.24c5cbc31 (2025-03-26 23:08:49 UTC) - https://godotengine.org
Vulkan 1.4.303 - Forward+ - Using Device #0: NVIDIA - NVIDIA GeForce RTX 3050 Laptop GPU


================================================================
CrashHandlerException: Program crashed
Engine version: Godot Engine v4.4.1.stable.custom_build (24c5cbc312400af8e00a918946aa1fc4db08647a)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[0] failwithmessage (D:\a\_work\1\s\src\vctools\crt\vcstartup\src\rtc\error.cpp:213)
[1] _RTC_StackFailure (D:\a\_work\1\s\src\vctools\crt\vcstartup\src\rtc\error.cpp:263)
[2] _RTC_CheckStackVars (D:\a\_work\1\s\src\vctools\crt\vcstartup\src\rtc\stack.cpp:69)
[3] godot::internal::_call_native_mb_ret<char32_t> (C:\Godot\src\godot-cpp\include\godot_cpp\core\engine_ptrcall.hpp:62)
[4] godot::InputEventKey::get_unicode (C:\Godot\src\godot-cpp\gen\src\classes\input_event_key.cpp:103)
[5] Example::_input (C:\Godot\src\godot-cpp\test\src\example.cpp:707)
[6] godot::call_with_ptr_args_helper<Example,godot::Ref<godot::InputEvent> const &,0> (C:\Godot\src\godot-cpp\include\godot_cpp\core\binder_common.hpp:193)
[7] godot::call_with_ptr_args<Example,godot::Ref<godot::InputEvent> const &> (C:\Godot\src\godot-cpp\include\godot_cpp\core\binder_common.hpp:213)
[8] <lambda_30812651af13b7fff1c795df58205dfb>::operator() (C:\Godot\src\godot-cpp\gen\include\godot_cpp\classes\node.hpp:349)
[9] <lambda_30812651af13b7fff1c795df58205dfb>::<lambda_invoker_cdecl> (C:\Godot\src\godot-cpp\gen\include\godot_cpp\classes\node.hpp:349)
[10] Node::_call_input (C:\build\godot\w64.msvc.tracy\scene\main\node.cpp:3464)
[11] SceneTree::_call_input_pause (C:\build\godot\w64.msvc.tracy\scene\main\scene_tree.cpp:1311)
[12] Viewport::push_input (C:\build\godot\w64.msvc.tracy\scene\main\viewport.cpp:3289)
[13] call_with_variant_args_dv<Viewport,Ref<InputEvent> const &,bool> (C:\build\godot\w64.msvc.tracy\core\variant\binder_common.h:462)
[14] MethodBindT<Viewport,Ref<InputEvent> const &,bool>::call (C:\build\godot\w64.msvc.tracy\core\object\method_bind.h:348)
[15] GDScriptFunction::call (C:\build\godot\w64.msvc.tracy\modules\gdscript\gdscript_vm.cpp:2057)
[16] GDScriptInstance::callp (C:\build\godot\w64.msvc.tracy\modules\gdscript\gdscript.cpp:2053)
[17] Node::_notification (C:\build\godot\w64.msvc.tracy\scene\main\node.cpp:227)
[18] Object::notification (C:\build\godot\w64.msvc.tracy\core\object\object.cpp:914)
[19] Node::_propagate_ready (C:\build\godot\w64.msvc.tracy\scene\main\node.cpp:283)
[20] Node::_propagate_ready (C:\build\godot\w64.msvc.tracy\scene\main\node.cpp:272)
[21] Node::_set_tree (C:\build\godot\w64.msvc.tracy\scene\main\node.cpp:3292)
[22] SceneTree::initialize (C:\build\godot\w64.msvc.tracy\scene\main\scene_tree.cpp:476)
[23] OS_Windows::run (C:\build\godot\w64.msvc.tracy\platform\windows\os_windows.cpp:2078)
[24] widechar_main (C:\build\godot\w64.msvc.tracy\platform\windows\godot_windows.cpp:99)
[25] _main (C:\build\godot\w64.msvc.tracy\platform\windows\godot_windows.cpp:126)
[26] main (C:\build\godot\w64.msvc.tracy\platform\windows\godot_windows.cpp:140)
[27] __scrt_common_main_seh (D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
[28] BaseThreadInitThunk
-- END OF BACKTRACE --
================================================================

[enetheru]

OK for a laugh, i searched the api.json for get_unicode and reverted the difference, and I it worked.
same for godot-4.5-dev1

[enetheru]

OK so i dont get it. I changed back to the master branch, added my /RTC1 call, and removed the meta keyword from the get and set_unicode functions. I get a passing test. I thought i screwed up the /RTC1 thing and re-did it which is why there are probably emails for deleted messages.

The only difference is the removal of the meta value, whats with that. It changes the expression of the get/set unicode functions from generation to use uint64_t rather than char32_t ..

I can see in the godot source its intended to be char32_t, I don't understand.

[enetheru]

changing the meta value to uint32 to change the function signature doesn't trigger the stack error