Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bpo-45696: Deep-freeze #29118

Merged
merged 46 commits into from
Nov 11, 2021
Merged

bpo-45696: Deep-freeze #29118

merged 46 commits into from
Nov 11, 2021

Conversation

gvanrossum
Copy link
Member

@gvanrossum gvanrossum commented Oct 21, 2021 •
edited
Loading

See faster-cpython/ideas#84

This seems to be 10% faster than current main[1], so an improvement over Eric's freeze.

Downside: the data segment grows from 490 KB to 1.3 MB. (We could get ~300 KB back by not including the marshalled-frozen data.)

Another downside, for developers: whenever you touch a source file, _bootstrap_python gets rebuilt, and that triggers a rebuild of all the deep-frozen modules. (See comment below for a possible fix.)

Haven't gotten it working on Windows yet. (The magic is disabled there for now.) I might do that in a separate PR.

[1] Measured on an x86 Mac, with lto-pgo, this shaves 1.5-2 msec off a total run time of around 14 msec. I measured by using the subprocess to run python -c pass 100 times.

https://bugs.python.org/issue45696

ericsnowcurrently and others added 27 commits August 30, 2021 17:32
@ericsnowcurrently
Add placeholders for stdlib modules.
a04d702
@ericsnowcurrently
Add the frozen stdlib modules.
033acab
@ericsnowcurrently
Freeze the stdlib modules loaded during startup.
0591cbd
@ericsnowcurrently
Group the import-related configs.
72666d4
@ericsnowcurrently
Add the -X frozen_modules CLI option.
ce71894
@ericsnowcurrently
Ignore frozen modules depending on "-X frozen_modules".
ab500e2
@ericsnowcurrently
Remember the stdlib dir during startup.
932e563
@gvanrossum
First part of deepfreeze
2bb2a43 
Not generating Makefile yet
@ericsnowcurrently
Add _Py_GetMainConfig().
cb6c74d
@ericsnowcurrently
Let _Py_GetStdlibDir() fall back to a config.
e94240e
@ericsnowcurrently
Add _PyConfig_InitImportConfig().
e106e41
@ericsnowcurrently
Identify whether or not the executed is running installed.
80506bb
@ericsnowcurrently
Default to "off" if in development.
9ac5b11
@ericsnowcurrently
Remember if built with --enable-optimizations.
acfc2a2
@ericsnowcurrently
Default to "on" for PGO builds.
ef653fb
@gvanrossum
Make deepfreeze actually work
f70319b 
Still not generating Makefile
@ericsnowcurrently
Add a NEWS entry.
a381ccf
@gvanrossum
Generate all deepfreeze items in Makefile.pre.in
47e3917
@gvanrossum
Don't use GET_CODE in the extern declarations
20992e9
@gvanrossum
Merge branch 'pr-28107' into deepfreeze
692959b
@gvanrossum
Exclude encodings module for now
77812d6
@gvanrossum
Regenerate Python/frozen.c
80cf00f
@gvanrossum
Make [deep]freeze process mess noisy
1d29e95
@gvanrossum
Regenerated MSVC project files
9818c44
@gvanrossum
Merge branch 'main' into deepfreeze
9e817b2
@gvanrossum
Make deep-freezing work again
2d89d0a
@gvanrossum
Make deep-freezing *really* work
8d029ae
@gvanrossum
Copy link
Member Author

@tiran any idea?

@tiran
Copy link
Member

tiran commented Nov 9, 2021

@tiran any idea?

You could try to use the bootstrap interpreter to generate sysconfig and pybuilddir.txt: main...tiran:deepfreeze-bootstrap

In the long run we might be able to replace PYTHON_FOR_REGEN with bootstrap interpreter, too.

@gvanrossum
Copy link
Member Author

@tiran, Does it look good now? All tests pass. I'll merge origin/main again and re-run the buildbots and then I hope to land this. We'll leave several things to future generations (or at least a later PR):

  • Windows
  • Avoid deep-freezing and rebuilding after every source change
  • Replace PYTHON_FOR_REGEN with $(BOOTSTRAP)

@gvanrossum gvanrossum added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 10, 2021
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @gvanrossum for commit 0c9de73 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 10, 2021
@gvanrossum
Copy link
Member Author

The AMD64 FreeBSD buildbot is failing (https://buildbot.python.org/all/#/builders/203/builds/314/steps/3/logs/stdio), and AFAICT the cause is that its Make doesn't understand $<. Any suggestions? This part of Makefile.pre.in is generated code, so I could just expand it in the code generator (Tools/scripts/freeze_modules.py), like I had before. Or is there a trick to get this to work on FreeBSD?

@tiran
Copy link
Member

tiran commented Nov 10, 2021

I was not aware that bmake has a different syntax for autovars than gmake. I only develop with GNU make. I cannot find a common syntax that works with BSD make and GNU make. You have to go back to the old system.

I'm sorry for the extra work I caused.

@gvanrossum gvanrossum added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 10, 2021
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @gvanrossum for commit 3e881031b14dc98f7d2b579e6b06c151df886333 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 10, 2021
@gvanrossum gvanrossum added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 10, 2021
@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @gvanrossum for commit f40d138 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Nov 10, 2021
@gvanrossum gvanrossum merged commit 1cbaa50 into python:main Nov 11, 2021
@gvanrossum gvanrossum deleted the deepfreeze branch November 11, 2021 02:01
remykarem pushed a commit to remykarem/cpython that referenced this pull request Dec 7, 2021
@gvanrossum @remykarem
bpo-45696: Deep-freeze selected modules (pythonGH-29118)
e200b3e 
This gains 10% or more in startup time for `python -c pass` on UNIX-ish systems.

The Makefile.pre.in generating code builds on Eric's work for bpo-45020, but the .c file generator is new.

Windows version TBD.
remykarem pushed a commit to remykarem/cpython that referenced this pull request Jan 30, 2022
@gvanrossum @remykarem
bpo-45696: Deep-freeze selected modules (pythonGH-29118)
d5c35ec 
This gains 10% or more in startup time for `python -c pass` on UNIX-ish systems.

The Makefile.pre.in generating code builds on Eric's work for bpo-45020, but the .c file generator is new.

Windows version TBD.
@kumaraditya303 kumaraditya303 mentioned this pull request Feb 2, 2022
Merged
miss-islington pushed a commit that referenced this pull request Feb 4, 2022
@kumaraditya303
bpo-46608: exclude marshalled-frozen data if deep-freezing to save 30…
bf95ff9 
…0 KB space (GH-31074)

This reduces the size of the data segment by **300 KB** of the executable because if the modules are deep-frozen then the marshalled frozen data just wastes space. This was inspired by comment by @gvanrossum in #29118 (comment). Note: There is a new option `--deepfreeze-only` in `freeze_modules.py` to change this behavior, it is on be default to save disk space.
```console 
# du -s ./python before
27892   ./python
# du -s ./python after
27524   ./python
```

Automerge-Triggered-By: GH:ericsnowcurrently
@gvanrossum gvanrossum mentioned this pull request Apr 10, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@methane methane methane left review comments

@tiran tiran tiran left review comments

@ericsnowcurrently ericsnowcurrently ericsnowcurrently approved these changes

@encukou encukou Awaiting requested review from encukou

@ncoghlan ncoghlan Awaiting requested review from ncoghlan ncoghlan is a code owner

@warsaw warsaw Awaiting requested review from warsaw warsaw is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@gvanrossum @bedevere-bot @tiran @methane @ericsnowcurrently @the-knights-who-say-ni