include performance

[mathieu-lacage]

Hi,

The ns-3 project uses cppyy to generate automated bindings for its c++ interfaces. In practice, it takes a little more than 10s to start on my laptop which is... A bit slow. I looked into what is going on and it appears that cppyy.include takes the crux of this time:

cppyy.include(f"ns3/{module}-module.h")

I am not entirely surprised by this so I'd like to know if there is a way to speed this up. i.e., would it be possible to cache the result of this include call so that later calls to this function for the same header is faster ?

Mathieu

[mathieu-lacage]

I'd like to point out that, intuitively, I suspect that there is something to be done with PCH. I suspect I might just need generate a specific pch from the headers of my project and then, to set the CLING_STANDARD_PCH environment variable. I guess the makepch script from cppyy-backend might be the ticket to generate the pch file...

Is this hopeless or am I on the right track ?

[wlav]

Yes, I'm afraid so. There have been several times that upstream has promised better, but none of the proposed solutions were portable and workable. I've been parroting their promises, but now feel silly making them as I wasn't able to keep them. And now the last they recommended was that I should make it easier for developers to add to the PCH... I haven't done that yet, b/c one of the neat things about modules, would be the ability to have different preprocessor defines and/or compiler flags. This would especially be important for projects that have their header files nicely organized.

Anyway, you'll find in the cppyy_backend directory in your site-packages location the directories etc/dictpch. In there, there is a file called allHeaders.h. At the end of that file, add something like:

#if __has_include("myproject.h")
#include "myproject.h"
#endif

then set CLING_REBUILD_PCH=1 or indeed set CLING_STANDARD_PCH to a file that doesn't exist yet to rebuild. Use EXTRA_CLING_ARGS to add the appropriate -I argument to find "myproject.h" (make sure to add -O2 -march=native if desired, as the envar will otherwise override those defaults). Finally, run python -c "import cppyy" to rebuild.

[mathieu-lacage]

hi @wlav

Thanks: this is enough to get me headed in the right direction. I will report with the result.

Out of curiosity, I see these warnings all the time when I start ns-3 but I am not entirely sure I understand what them mean. Do you know what emits these ?

[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { _ZN3ns3L16g_TimeStaticInitE, $.cling-module-162.__inits.0, __orc_init_func.cling-module-162, _GLOBAL__sub_I_cling_module_162, _cxx_global_var_initcling_module_162 }) }
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { __orc_init_func.cling-module-162 }) }
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { _ZZN3ns34Time14PeekResolutionEvE10resolution, _cxx_global_var_initcling_module_193, _ZGVN3ns3L27UE_MEASUREMENT_REPORT_DELAYE, _ZN3ns3L27UE_MEASUREMENT_REPORT_DELAYE, _ZN3ns34TimeD1Ev, _ZGVZN3ns34Time14PeekResolutionEvE10resolution, $.cling-module-193.__inits.0, __orc_init_func.cling-module-193 }) }
[runStaticInitializersOnce]: Failed to materialize symbols: { (main, { __orc_init_func.cling-module-193 }) }

[mathieu-lacage]

I can guess that this is the JIT that is telling me that it is unable to resolve elf symbols. I dug out the source code for runStaticInitializersOnce and went down the rabbit hole of cppy-backend and cling root itself. I understand that what is happening here is that while it is building the pch, it is parsing the headers. Some of these headers have static variables defined, so this triggers code in the interpreter to execute these global constructors and schedule corresponding destructors for later.

Ok.

So, I looked into ns-3 and, I see this in a header:

static const Time UE_MEASUREMENT_REPORT_DELAY = MicroSeconds(1);

Which, really, does not make much sense to me.

I replaced that static const with a #define because I am cheap and it got rid that annoying error...

[mathieu-lacage]

@wlav After some more testing, it turns out that your suggestion works surprisingly well for me.

Before: 12.36s
After: 1.88s

There is still about 800ms spent in cppyy during startup which is tolerable and the remaining 1s is because of crazy/amazing code in the ns3 init.py that I cannot get rid of without a lot more work.

Do you think it would be possible to release a version of cppyy_backend with the change you suggested to allHeaders.h is only to allow me to upstream my local ns-3 patches ?

For reference, here is the code I ended up writing for ns-3 to provide automatic generation of pch given a specific ns-3 configuration, in case it is useful to others.

class CppyyFactory:
    def __init__(self):
        self._smart_ptr_type = []
        self._library_path = []
        self._include_path = []
        self._defines = []
        self._includes = []
        self._libraries = []
        self._debug = False


    def AddSmartPtrType(self, name):
        self._smart_ptr_type.append(name)

    def set_debug(self, debug):
        self._debug = debug

    def add_library_path(self, path):
        self._library_path.append(path)

    def add_include_path(self, path):
        self._include_path.append(path)

    def add_defines(self, defines):
        self._defines.extend(defines)

    def include(self, header):
        self._includes.append(header)

    def load_library(self, name):
        self._libraries.append(name)

    def create(self):
        use_custom_pch = os.environ.get('NS3_PYTHON_USE_CUSTOM_PCH')
        if use_custom_pch:
            o = self._create_with_custom_pch()
        else:
            o = self._create_with_default_pch()
        return o

    def _create_with_default_pch(self):
        try:
            import cppyy
        except ModuleNotFoundError:
            print("Cppyy is required by the ns-3 python bindings.")
            print("You can install it with the following command: pip install cppyy")
            exit(-1)
        import libcppyy
        for smart_ptr_type in self._smart_ptr_type:
            libcppyy.AddSmartPtrType(smart_ptr_type)
        cppyy.set_debug(self._debug)
        for path in self._include_path:
            cppyy.add_include_path(path)
        for define in self._defines:
            cppyy.cppexec(f"""
                #ifndef {define}
                #define {define} 1
                #endif /* {define} */
            """)
        for include in self._includes:
            cppyy.include(include)
        for path in self._library_path:
            cppyy.add_library_path(path)
        for library in self._libraries:
            cppyy.load_library(library)
        return cppyy

    def _create_with_custom_pch(self):
        import json, hashlib, os
        signature_body = json.dumps({
            'smart_ptr_type': self._smart_ptr_type,
            'debug': self._debug,
            'library_path': sorted(self._library_path),
            'include_path': sorted(self._include_path),
            'cppexe': sorted(self._defines),
            'include': sorted(self._includes),
            'load_library': sorted(self._libraries)
        })
        signature = hashlib.sha256(signature_body.encode('utf-8')).hexdigest()
        directory = os.path.abspath(os.path.join(os.path.expanduser('~'), '.ns3', signature))
        try:
            os.makedirs(directory)
        except:
            pass
        includes = [f'#include "{include}"\n' for include in self._includes]
        with open(os.path.join(directory, 'myproject.h'), 'w+') as f:
            f.write(''.join(includes))
        os.environ['CLING_STANDARD_PCH'] = os.path.join(directory, 'ns3.pch')
        cling_args = [f'-I{include_path}' for include_path in self._include_path]
        cling_args += [f'-I{directory}']
        cling_args += [f'-D{define}=1' for define in uniq(self._defines)]
        cling_args += ['-O2', '-march=native']
        os.environ['EXTRA_CLING_ARGS'] = ' '.join(cling_args)
        try:
            import cppyy
        except ModuleNotFoundError:
            print("Cppyy is required by the ns-3 python bindings.")
            print("You can install it with the following command: pip install cppyy")
            exit(-1)
        import libcppyy
        for smart_ptr_type in self._smart_ptr_type:
            libcppyy.AddSmartPtrType(smart_ptr_type)
        cppyy.set_debug(self._debug)
        for path in self._library_path:
            cppyy.add_library_path(path)
        for library in self._libraries:
            cppyy.load_library(library)
        return cppyy

[wlav]

The "failed to materialize" symbols is an old one: #175. There has been a partial fix which solves it in some cases (the problematic code was in the standard STL header for std::string), but not in all, and yes, that fix was similar to what you observed and changed. It's something that only happens with Clang13, though. I think, but don't know for certain, that prior Clang ignored the initial error that in newer Clang is causing this cascade of errors.

Left-over startup time is likely from loading the .so's and the PCH from disk. Especially the PCH is not small. Further, there's some more disk access such as checking whether it up-to-date, searching for CPyCppyy's include directory (there's nothing standard about Python package installations) etc.

As for the change, you mean adding this:

#if __has_include("myproject.h")
#include "myproject.h"
#endif

to allHeaders.h? Fine in principle, except that then there's again "only one".

The makepch.py script (called here: https://github.com/wlav/cppyy-backend/blob/master/cling/python/cppyy_backend/loader.py#L223) can actually take multiple header files that then all end up in the PCH. Ie. modifying that code to take a list of headers from some config file (or envar) would be better, especially since that list of headers could be saved next to the .pch and update the latter when the former changes.