Skip to content

Commit

Permalink
pythongh-89792: Limit test_tools freeze test build parallelism based …
Browse files Browse the repository at this point in the history
…on the number of cores (python#101841)

unhardcode freeze test build parallelism. base it on the number of cpus, don't use more than max(2, os.cpu_count()/3).
  • Loading branch information
gpshead authored Feb 12, 2023
1 parent da2fb92 commit dfc2e06
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 6 deletions.
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
``test_tools`` now copies up to 10x less source data to a temporary
directory during the ``freeze`` test by ignoring git metadata and other
artifacts.
``test_tools`` now copies up to 10x less source data to a temporary directory
during the ``freeze`` test by ignoring git metadata and other artifacts. It
also limits its python build parallelism based on os.cpu_count instead of hard
coding it as 8 cores.
15 changes: 12 additions & 3 deletions Tools/freeze/test/freeze.py
Original file line number Diff line number Diff line change
Expand Up @@ -163,16 +163,25 @@ def prepare(script=None, outdir=None):
if not MAKE:
raise UnsupportedError('make')

cores = os.cpu_count()
if cores and cores >= 3:
# this test is most often run as part of the whole suite with a lot
# of other tests running in parallel, from 1-2 vCPU systems up to
# people's NNN core beasts. Don't attempt to use it all.
parallel = f'-j{cores*2//3}'
else:
parallel = '-j2'

# Build python.
print(f'building python in {builddir}...')
print(f'building python {parallel=} in {builddir}...')
if os.path.exists(os.path.join(srcdir, 'Makefile')):
# Out-of-tree builds require a clean srcdir.
_run_quiet([MAKE, '-C', srcdir, 'clean'])
_run_quiet([MAKE, '-C', builddir, '-j8'])
_run_quiet([MAKE, '-C', builddir, parallel])

# Install the build.
print(f'installing python into {prefix}...')
_run_quiet([MAKE, '-C', builddir, '-j8', 'install'])
_run_quiet([MAKE, '-C', builddir, 'install'])
python = os.path.join(prefix, 'bin', 'python3')

return outdir, scriptfile, python
Expand Down

0 comments on commit dfc2e06

Please sign in to comment.