Speed up building multiple executable with onedir

[claudious96]

Hi all,

first of all thank you so much for building and maintaining this open source library.

I have an application which is composed of several tool (entrypoints). All tools share the same dependencies.

I already have a spec file for building a onedir bundle that includes all tools but I'd like to speed up the build, since what I'm doing is running the Analysis for each single tool which I believe is what causes the slowdown. In the spec the very first thing done is collecting the dependencies for all apps by parsing the poetry.lock file, including each package using collect_submodules and later on passing the resulting list to the hidden_imports arg.

So the result is that for each tool I'm actually importing the same packages but the operation is repeated for each tool in either Analysis, PYZ or EXE.

I really like that the binaries' size is just a few KB for each tool while having a single large directory that includes the dependencies.

So is there a way to create the directory of the bundle only once? Should I run the the Analysis only once and then reuse the output in the PYZ and EXE for each entrypoint/tool?

I've found the discussions #6634 and #7140 but they all seem to suffer from the same slow down of running the analysis for all entrypoints.

This is how my spec file looks like:

apps = [
    ('tool-a','app/tools/a.py'),
    ('tool-b','app/tools/b.py'),
    ('tool-c','app/tools/c.py'),
    ('tool-d','app/tools/d.py'),
    ('tool-e','app/tools/e.py'),
]

hidden_imports  = ... # parse the poetry.lock file once, I'll reuse the result for each script

compiled_apps = []

for name, script in apps:
    print(f"------ COMPILING {name} ------")
    a = Analysis(
        [script],
        pathex=[],
        binaries=[],
        datas=[],
        hiddenimports=hiddenimports,
        hookspath=[],
        hooksconfig={},
        runtime_hooks=[],
        excludes=[],
        win_no_prefer_redirects=False,
        win_private_assemblies=False,
        cipher=block_cipher,
        noarchive=True,
    )

    pyz = PYZ(a.pure, a.zipped_data)

    exe = EXE(
        pyz,
        a.scripts,
        exclude_binaries=True,
        name=name,
        debug=False,
        bootloader_ignore_signals=False,
        strip=False,
        upx=True,
        upx_exclude=[],
        runtime_tmpdir=None,
        console=True,
        disable_windowed_traceback=False,
        argv_emulation=False,
        target_arch=None,
        codesign_identity=None,
        entitlements_file=None,
    )

    compiled_apps.extend([a.binaries, a.datas, exe])

coll = COLLECT(
        *compiled_apps,
        name='my-bundled-app',
    )

[rokm]

So is there a way to create the directory of the bundle only once? Should I run the the Analysis only once and then reuse the output in the PYZ and EXE for each entrypoint/tool?

There is no officially supported, out-of-the box way for that.

If all those entry-points really have identical import dependencies, I suppose you could try running Analyis only once, on one of them, and then re-use the results, while manually creating the scripts TOC list instead of using a.scripts. Since that TOC also includes run-time hooks, you would probably have to take a.scripts, and replace the last entry with the one that corresponds to the actual script.

---

But if I were in your place, I would reconsider having multiple executables in the first place, as each of them will contain its copy of PYZ, with all collected modules. For programs with complex dependencies, that alone might unnecessarily baloon the distribution size.

I would instead opt for having a single executable whose first argument is <a|b|c|d|e>, and depending on that it would perform the functionality of corresponding entry-point script. And create wrapper scripts (shell on POSIX, .bat on Windows) corresponding to each entry-point that calls the main executable with appropriate dispatch argument and passes on the remaining command-line arguments.

[claudious96]

Rok, thanks for your answer.

There is no officially supported, out-of-the box way for that.

I see, mine might be a very specific use case.

But if I were in your place, I would reconsider having multiple executables in the first place, as each of them will contain its copy of PYZ, with all collected modules. For programs with complex dependencies, that alone might unnecessarily baloon the distribution size.

I'm not sure I follow you here, what do you mean by each of them will contain its copy of PYZ? Does this lead to sub-optimal performance or is it just about the size? Asking this because the size of each bin looks pretty small to me (the biggest is ~80 KB) while the directory is around 700 MB.

A thing I haven't said is that I'm cythonizing the code and including the result of the cython compilation as the binaries args of Analysis, I guess that is the reason for such small binary size (?)

I would instead opt for having a single executable whose first argument is <a|b|c|d|e>, and depending on that it would perform the functionality of corresponding entry-point script. And create wrapper scripts (shell on POSIX, .bat on Windows) corresponding to each entry-point that calls the main executable with appropriate dispatch argument and passes on the remaining command-line arguments.

I'm not sure it suits my situation but that can definitely be a workaround.

[rokm]

But if I were in your place, I would reconsider having multiple executables in the first place, as each of them will contain its copy of PYZ, with all collected modules. For programs with complex dependencies, that alone might unnecessarily baloon the distribution size.

I'm not sure I follow you here, what do you mean by each of them will contain its copy of PYZ? Does this lead to sub-optimal performance or is it just about the size? Asking this because the size of each bin looks pretty small to me (the biggest is ~80 KB) while the directory is around 700 MB.

A thing I haven't said is that I'm cythonizing the code and including the result of the cython compilation as the binaries args of Analysis, I guess that is the reason for such small binary size (?)

It is about the size, but if you are cythonizing python modules, it does not apply to your case.

[claudious96]

Got it. I'll try to play a bit with manually creating the scripts TOC. I'll write here if I manage to get somewhere, it may be useful to someone else.

Thanks again!