Setuptools 60 slows down startup

[gvanrossum]

We discovered that if you have setuptools 60 or higher installed, a whole bunch of extra modules are imported that add ~10 msec to startup.

This caused a noticeable bump in the timings for python_startup between Dec 18 and 19, 2021.

Should we just deep-freeze all the imported stdlib modules?

Here's the list of extra modules:

import time:       278 |        278 |         types
import time:       112 |        112 |           _operator
import time:       419 |        531 |         operator
import time:       129 |        129 |             itertools
import time:       325 |        325 |             keyword
import time:       468 |        468 |             reprlib
import time:       258 |        258 |             _collections
import time:       978 |       2156 |           collections
import time:        78 |         78 |           _functools
import time:       835 |       3068 |         functools
import time:      1359 |       5235 |       enum
import time:       138 |        138 |         _sre
import time:       497 |        497 |           sre_constants
import time:       528 |       1025 |         sre_parse
import time:       512 |       1674 |       sre_compile
import time:       109 |        109 |       _locale
import time:       886 |        886 |       copyreg
import time:       671 |       8574 |     re
import time:       471 |        471 |       warnings
import time:       330 |        801 |     importlib
import time:       906 |      10279 |   _distutils_hack

Presumably there's overlap with the modules imported by runpy (used by python -m module).

[markshannon]

I guess it depends on whether most applications actually use these modules.
If they do, then deep-freeze them.
If they don't, then fix setuptools.

I suspect that deep-freezing these modules might be easier than fixing setuptools.

[markshannon]

Any idea how much of the slowdown we get back by deep-freezing those modules?

[ericsnowcurrently]

IIUC, part of that cost is from making a copy of distutils on disk.

[gvanrossum]

IIUC, part of that cost is from making a copy of distutils on disk.

I don't believe that's true -- setuptools just has a vendored copy of distutils named setuptools._distutils, and they insert a new meta-finder at the front of sys.meta_path. The meta-finder imports various expensive modules: in [setuptools/_distutils_hack/__init__.py] I see these imports:

import sys
import os
import re
import importlib
import warnings
import contextlib

That's what takes the time (including whatever those modules import).

I tried freezing everything in the list, and it gives back about half of the time (roughly 5 msec). It costs another megabyte in binary size.

I think we should talk to the setuptools maintainers.

[ericsnowcurrently]

@pablogsal opened an issue on setuptools: pypa/setuptools#3006.

[gvanrossum]

Fixed by setuptools 60.5.0.

https://speed.python.org/timeline/#/?exe=12&ben=python_startup&env=1&revs=50&equid=off&quarts=on&extr=on