Cannot build my GDExtension with Godot 4.4

[ze2j]

Tested versions

Reproducible with Godot 4.4
Not reproducible with Godot 4.3

System information

Godot v4.4.1.rc (afaa0cd4b) - Ubuntu 22.04.5 LTS 22.04 on X11 - X11 display driver, Multi-window, 2 monitors - Vulkan (Forward+) - dedicated NVIDIA GeForce GTX 1070 (nvidia; 535.183.01) - Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz (8 threads)

Issue description

I encountered an unusual build error while porting my project from Godot 4.3 to 4.4. My project consists of several custom modules registering 94 classes to ClassDB and a small GDExtension written in C++.

Godot 4.4 itself builds without errors, and godot-cpp for 4.4 also compiles successfully. However, building my extension fails with the following error:

$ scons platform=linux dev_build=yes compiledb=true precision=double disable_exceptions=false verbose=true
scons: Reading SConscript files ...
Auto-detected 8 CPU cores available for build parallelism. Using 7 cores by default. You can override it with the -j argument.
Building for architecture x86_64 on platform linux
scons: done reading SConscript files.
scons: Building targets ...
ar rc <godot-cpp>/bin/libgodot-cpp.linux.template_debug.dev.double.x86_64.a <skipping a very long list of absolute paths to *.o files>
scons: *** [<godot-cpp-private-fork>/bin/libgodot-cpp.linux.template_debug.dev.double.x86_64.a] sh: Argument list too long
scons: building terminated because of errors.

In the ar rc command, 1050 object files are listed, including the 94 from my project. The total length of the arguments is 131,107 characters. I believe the error occurs because the arguments are interpreted as a single entity, exceeding the system's limit of MAX_ARG_STRLEN (131,072).

To confirm this, I commented out some of my GDREGISTER_CLASS calls to reduce the total argument length below the threshold, and the build succeeded. Another potential workaround could be relocating my project to a directory with a shorter path (though I haven’t tested this; my current project path is only 47 characters long).

As Godot continues to grow, I suspect more users on similar Unix-like systems will encounter this limitation.

I'm unsure whether to report this issue to godot-cpp, or even scons directly, given its potential impact on Godot’s documentation. Please let me know if I should transfer this bug report elsewhere.

Notes:

• The issue occurs with both scons 4.0 and 4.9.
• Enabling the scons stack trace (--debug=stacktrace) produces the following error when the build fails:

scons: internal stack trace:
  File "/home/mla/.local/lib/python3.10/site-packages/SCons/Taskmaster/Job.py", line 737, in _work
    task.execute()
  File "/home/mla/.local/lib/python3.10/site-packages/SCons/Script/Main.py", line 224, in execute
    SCons.Taskmaster.OutOfDateTask.execute(self)
  File "/home/mla/.local/lib/python3.10/site-packages/SCons/Taskmaster/__init__.py", line 263, in execute
    raise buildError

Steps to reproduce

Increase the number of classes registered to ClassDB then build the GDextension C++ example

Minimal reproduction project (MRP)

Let me know if you need one (it's not a trivial task)

[akien-mga]

Moving to https://github.com/godotengine/godot-cpp.

[dsnopek]

Thanks for the report!

Just to make sure I'm understanding correctly: the issue is caused by the classes that were added to Godot in your custom module, not the classes added in your GDExtension? That does make sense, given that all the generated classes are dumped into the same directory, and so scons would put them in the same .a file

I'm not sure what tools (if any) scons gives us to deal with this. A potential solution is just making more directories for the generated .cpp files, such that we ensure that each directory only holds a maximum of 50 (?) files?

[ze2j]

Just to make sure I'm understanding correctly: the issue is caused by the classes that were added to Godot in your custom module, not the classes added in your GDExtension?

Yes that's correct.

That does make sense, given that all the generated classes are dumped into the same directory, and so scons would put them in the same .a file
I'm not sure what tools (if any) scons gives us to deal with this. A potential solution is just making more directories for the generated .cpp files, such that we ensure that each directory only holds a maximum of 50 (?) files?

I don’t think this would work, because the issue is with the number of files passed to the command that creates the library, and we want to reduce that number. I am trying a different approach: create small intermediate static libraries and then link them together.

Edit:
Here is an attempt trying to implement this approach (to apply in godotcpp.py in the _godot_cpp method) but it doesn't work... The resulting library is empty:

-        library = env.StaticLibrary(target=env.File("bin/%s" % library_name), source=sources)
+        libraries = []
+        remaining = sources[:]
+        n = 0
+        while len(remaining) > 0:
+            libraries.append(env.StaticLibrary(target="bin/libgodot-cpp-part_%d%s" % (n, env["suffix"] + env["LIBSUFFIX"]), source=remaining[:100]))
+            remaining = remaining[100:]
+            n += 1
+
+        library = env.StaticLibrary(target=env.File("bin/%s" % library_name), source=[], LIBS=libraries)
+        for lib in libraries:
+            env.Depends(env.File("bin/%s" % library_name), lib)
+

[ze2j]

I can't find a single out-of-the-box method that is portable:

• Linking smaller static libraries together doesn’t work as I thought because the object files they contain are not extracted and repackaged.
• Incrementally building a large static library seems possible using ar with the right options, but this is not portable.
• Extracting object files from smaller static libraries and then merging them should work, but this is also not portable.

As far as I can tell, we are left with two possible approaches, but one requirement must be dropped:

• Instead of building a single static library, build smaller static libraries and let the GDExtension shared library link against them.
• Create a custom builder, specific to Unix-like systems, to incrementally build the single large static library.

[enetheru]

does using a file for the options work?
https://www.man7.org/linux/man-pages/man1/ar.1.html

@file
           Read command-line options from file.  The options read are
           inserted in place of the original @file option.  If file does
           not exist, or cannot be read, then the option will be treated
           literally, and not removed.

           Options in file are separated by whitespace.  A whitespace
           character may be included in an option by surrounding the
           entire option in either single or double quotes.  Any
           character (including a backslash) may be included by prefixing
           the character to be included with a backslash.  The file may
           itself contain additional @file options; any such options will
           be processed recursively.

[ze2j]

Nice idea!

It works, I am making some more tests and will provide a PR in a few hours if everything is fine.