Generate shared libraries instead of .pyc files

[lpereira]

This idea is a stretch -- and can be a whole lot of work as it will differ between operating systems -- but is something that has been in the back of my mind for a bit now.

Looking at improving startup performance, especially while importing modules written in Python, I think it would be beneficial to, instead of generating .pyc files, generating shared libraries directly. These shared libraries would be similar to deepfrozen modules (without the patchup functions because they wouldn't be needed), which Python would try to load before trying to load .pyc files.

This would require us to generate these files without a compiler and/or linker, which makes this a bit tricky as this is clearly not portable (and would require at least 3 implementations: one for ELF systems, one for PE systems, and one for Mach-O systems; endianess and other ABI considerations such as alignment wouldn't be much of an issue as the code that would be generating this would have that information already and the generated shared library wouldn't be usable in a different OS/CPU arch configuration). On ELF systems, this would probably be a shared object with a single .data segment, and only one exported symbol, that points to the toplevel struct -- very similar to what the deepfreeze script does.

While complex and platform-specific, I imagine the potential improvements for this should be more meaningful in terms of startup time than the current usage of deepfreezing, as a lot of software out there imports a lot of modules and end up not using everything from them, wasting time unmarshaling things that are rarely necessary.

An alternative to this idea would be generating a file that can be mmap-ed into the process, and includes a patchup table to patch all the pointers to account for ASLR and whatnot. This would be more portable and would achieve essentially the same end result.

[lpereira]

Things I still need to think about:

• How to factor in modules with side-effects as they're being imported?

[ericsnowcurrently]

Related:

• Faster startup -- Share code objects from memory-mapped file #150
• Improve import time #158
• pyc-ng - creating serializations for modules that lends itself well to memory mapping and lazy loading #197
• Daemon mode to speed up startup? #129
• Improve startup time. #134
• https://arxiv.org/abs/2104.00254

[oraluben]

• Faster startup -- Share code objects from memory-mapped file #150

We've finished third-partying this mmap approach, and plan to publish it to PyPI. We would love to share more information after we're back from holiday. (happy tiger's year!)

[arhadthedev]

Will antiviruses be calm about generation of executable modules right under their nose?