PyPerformance Results on Windows for Py310 --> Py311

[sweeneyde]

Comparing versions (installed from python.org installers):

Python 3.10.2 (tags/v3.10.2:a58ebcc, Jan 17 2022, 14:12:15) [MSC v.1929 64 bit (AMD64)] on win32
Python 3.11.0a6 (main, Mar  7 2022, 16:46:19) [MSC v.1929 64 bit (AMD64)] on win32

py -m pyperf compare_to .\310.json .\311.json -G
Slower (20):
- unpack_sequence: 109 ns +- 1 ns -> 178 ns +- 2 ns: 1.63x slower
- nbody: 161 ms +- 1 ms -> 187 ms +- 2 ms: 1.17x slower
- scimark_fft: 409 ms +- 5 ms -> 471 ms +- 7 ms: 1.15x slower
- pathlib: 187 ms +- 2 ms -> 211 ms +- 2 ms: 1.13x slower
- nqueens: 111 ms +- 1 ms -> 123 ms +- 9 ms: 1.10x slower
- fannkuch: 505 ms +- 3 ms -> 553 ms +- 1 ms: 1.10x slower
- unpickle_pure_python: 321 us +- 4 us -> 351 us +- 6 us: 1.09x slower
- meteor_contest: 119 ms +- 2 ms -> 128 ms +- 1 ms: 1.07x slower
- scimark_sparse_mat_mult: 5.59 ms +- 0.61 ms -> 5.94 ms +- 0.37 ms: 1.06x slower
- xml_etree_iterparse: 112 ms +- 2 ms -> 117 ms +- 2 ms: 1.05x slower
- xml_etree_generate: 106 ms +- 1 ms -> 110 ms +- 1 ms: 1.04x slower
- pidigits: 211 ms +- 1 ms -> 219 ms +- 2 ms: 1.04x slower
- sympy_expand: 576 ms +- 4 ms -> 592 ms +- 2 ms: 1.03x slower
- regex_v8: 28.2 ms +- 0.3 ms -> 28.8 ms +- 0.4 ms: 1.02x slower
- telco: 6.98 ms +- 0.08 ms -> 7.12 ms +- 0.06 ms: 1.02x slower
- sympy_str: 355 ms +- 3 ms -> 361 ms +- 8 ms: 1.02x slower
- regex_effbot: 3.38 ms +- 0.03 ms -> 3.43 ms +- 0.06 ms: 1.01x slower
- spectral_norm: 157 ms +- 3 ms -> 159 ms +- 1 ms: 1.01x slower
- json_loads: 27.5 us +- 0.2 us -> 27.8 us +- 0.2 us: 1.01x slower
- xml_etree_parse: 161 ms +- 2 ms -> 162 ms +- 3 ms: 1.01x slower

Faster (35):
- deltablue: 7.62 ms +- 0.17 ms -> 5.59 ms +- 0.06 ms: 1.36x faster
- go: 236 ms +- 2 ms -> 191 ms +- 1 ms: 1.23x faster
- logging_silent: 185 ns +- 6 ns -> 151 ns +- 4 ns: 1.23x faster
- pyflate: 721 ms +- 33 ms -> 616 ms +- 4 ms: 1.17x faster
- xml_etree_process: 93.2 ms +- 36.5 ms -> 79.7 ms +- 0.8 ms: 1.17x faster
- richards: 74.7 ms +- 1.1 ms -> 63.9 ms +- 1.0 ms: 1.17x faster
- scimark_sor: 210 ms +- 3 ms -> 182 ms +- 1 ms: 1.15x faster
- raytrace: 501 ms +- 6 ms -> 441 ms +- 13 ms: 1.14x faster
- python_startup: 34.4 ms +- 0.2 ms -> 30.4 ms +- 0.1 ms: 1.13x faster
- 2to3: 439 ms +- 23 ms -> 391 ms +- 3 ms: 1.12x faster
- tornado_http: 216 ms +- 3 ms -> 193 ms +- 3 ms: 1.12x faster
- scimark_lu: 173 ms +- 3 ms -> 155 ms +- 2 ms: 1.12x faster
- html5lib: 88.1 ms +- 3.8 ms -> 79.0 ms +- 2.8 ms: 1.12x faster
- scimark_monte_carlo: 108 ms +- 2 ms -> 97.5 ms +- 0.6 ms: 1.11x faster
- logging_format: 13.8 us +- 0.2 us -> 12.6 us +- 0.2 us: 1.09x faster
- mako: 16.9 ms +- 0.2 ms -> 15.6 ms +- 0.7 ms: 1.08x faster
- crypto_pyaes: 118 ms +- 1 ms -> 110 ms +- 1 ms: 1.07x faster
- unpickle: 15.9 us +- 1.0 us -> 14.9 us +- 0.2 us: 1.07x faster
- logging_simple: 12.4 us +- 0.2 us -> 11.6 us +- 0.5 us: 1.07x faster
- sqlite_synth: 3.39 us +- 0.03 us -> 3.20 us +- 0.03 us: 1.06x faster
- pickle_pure_python: 470 us +- 5 us -> 447 us +- 6 us: 1.05x faster
- unpickle_list: 5.41 us +- 0.06 us -> 5.15 us +- 0.05 us: 1.05x faster
- float: 128 ms +- 1 ms -> 123 ms +- 3 ms: 1.04x faster
- pickle_list: 5.52 us +- 0.05 us -> 5.30 us +- 0.05 us: 1.04x faster
- pickle_dict: 36.5 us +- 0.2 us -> 35.1 us +- 0.3 us: 1.04x faster
- python_startup_no_site: 26.3 ms +- 0.2 ms -> 25.4 ms +- 0.1 ms: 1.04x faster
- json_dumps: 14.7 ms +- 0.2 ms -> 14.2 ms +- 0.3 ms: 1.03x faster
- regex_compile: 186 ms +- 1 ms -> 182 ms +- 3 ms: 1.03x faster
- sympy_sum: 209 ms +- 4 ms -> 203 ms +- 6 ms: 1.03x faster
- hexiom: 10.1 ms +- 0.2 ms -> 9.93 ms +- 0.07 ms: 1.02x faster
- pickle: 13.8 us +- 0.1 us -> 13.5 us +- 0.1 us: 1.02x faster
- chameleon: 10.6 ms +- 0.2 ms -> 10.4 ms +- 0.2 ms: 1.01x faster
- django_template: 52.4 ms +- 0.9 ms -> 51.8 ms +- 0.9 ms: 1.01x faster
- sympy_integrate: 25.8 ms +- 0.4 ms -> 25.6 ms +- 0.4 ms: 1.01x faster
- regex_dna: 212 ms +- 1 ms -> 211 ms +- 3 ms: 1.01x faster

Benchmark hidden because not significant (1): chaos

Geometric mean: 1.03x faster
Ignored benchmarks (2) of .\310.json: genshi_text, genshi_xml

I replicated the nbody results on another run of each benchmark:

Performance version: 1.0.4
Python version: 3.10.2 (64-bit) revision a58ebcc
Report on Windows-10-10.0.19044-SP0
Number of logical CPUs: 12
Start date: 2022-03-07 21:07:25.016445
End date: 2022-03-07 21:07:41.532414

### nbody ###
Mean +- std dev: 161 ms +- 2 ms

Performance version: 1.0.4
Python version: 3.11.0a6 (64-bit)
Report on Windows-10-10.0.19044-SP0
Number of logical CPUs: 12
Start date: 2022-03-07 21:06:44.511546
End date: 2022-03-07 21:07:03.186641

### nbody ###
Mean +- std dev: 189 ms +- 7 ms

I did have to change a couple of lines of pyperformance to get it to run:

In _pythoninfo.py, change base_executable = os.path.join(os.path.dirname(stdlib_dir), 'python') to python.exe.

[sweeneyde]

Apparently it's the PGO that is hurting nbody more than it is helping :(

Results on main right now for .\PCBuild\build.bat versus .\PCBuild\build.bat --pgo:

Mean +- std dev: [main_nbody_release] 160 ms +- 2 ms -> [main_nbody_pgo] 183 ms +- 2 ms: 1.14x slower

Could be related to https://bugs.python.org/issue45116

[gvanrossum]

This could also be bpo-45116. Is there an experiment that you could do to force inlining in ceval.c?

3% faster is really disappointing.

[brandtbucher]

We should also recalculate that geometric mean without the unpack_sequence result. That slowdown is because inline caching pushed the size of that function slightly beyond our old quickening threshold. It was fixed on main earlier today.

[sweeneyde]

py -m pyperf compare_to .\310.json .\311.json --benchmark nbody --benchmark scimark_fft --benchmark pathlib --benchmark nqueens --benchmark fannkuch --benchmark unpickle_pure_python --benchmark meteor_contest --benchmark scimark_sparse_mat_mult --benchmark xml_etree_iterparse --benchmark xml_etree_generate --benchmark pidigits --benchmark sympy_expand --benchmark regex_v8 --benchmark telco --benchmark sympy_str --benchmark regex_effbot --benchmark spectral_norm --benchmark json_loads --benchmark xml_etree_parse --benchmark deltablue --benchmark go --benchmark logging_silent --benchmark pyflate --benchmark xml_etree_process --benchmark richards --benchmark scimark_sor --benchmark raytrace --benchmark python_startup --benchmark 2to3 --benchmark tornado_http --benchmark scimark_lu --benchmark html5lib --benchmark scimark_monte_carlo --benchmark logging_format --benchmark mako --benchmark crypto_pyaes --benchmark unpickle --benchmark logging_simple --benchmark sqlite_synth --benchmark pickle_pure_python --benchmark unpickle_list --benchmark float --benchmark pickle_list --benchmark pickle_dict --benchmark python_startup_no_site --benchmark json_dumps --benchmark regex_compile --benchmark sympy_sum --benchmark hexiom --benchmark pickle --benchmark chameleon --benchmark django_template --benchmark sympy_integrate --benchmark regex_dna -G

(all the same stuff, except unpack_sequence was printed here)

Geometric mean: 1.04x faster

[kumaraditya303]

Perhaps instead of msvc, clang-cl can be used instead as it seems supported.

https://docs.microsoft.com/en-us/cpp/build/clang-support-msbuild?view=msvc-170#configure-a-windows-project-to-use-clang-tools

[gvanrossum]

@kumaraditya303 If you're interested, it's worth at least trying the Clang stuff, to see just how much we'd have to change to get a running binary with PGO+LTO that can still load 3rd party extensions (DLLs) built using the standard MSVC toolchain (e.g. wheels).

[kumaraditya303]

There is already a PR for it but it has conflicts python/cpython#18371

[gvanrossum]

Do you want to use that as a starting point? IMO most of the work will be experimenting with how easy it is to use (e.g. make it easy for a dinosaur like myself to switch).

[gvanrossum]

I am still struggling to repro this result, mostly because pyperformance has a few issues.

• @sweeneyde you nailed one (base_executable is wrong for Windows, although the right fix is more complex).
• Another I ran into is that psutil is required to do something to the process priority, and it's not trivial to get this installed in the sub-venvs where it is needed. @sweeneyde Did you run into this? Did you manage to fix it?
• A third is Creating venvs takes very long on Windows python/pyperformance#146, there's too much venv overhead.

Eric is working on the latter two now.

In the meantime I am trying to think of a fix. Mark suggested that we might redefine Py_INCREF/DECREF as macros at the top of the file to force the compiler to "inline" them. Are there other suggestions/ideas?

[sweeneyde]

• Another I ran into is that psutil is required to do something to the process priority, and it's not trivial to get this installed in the sub-venvs where it is needed. @sweeneyde Did you run into this? Did you manage to fix it?

I got the process priority warnings but ignored them. I did not do anything to change process priority (I never tried too hard to get pyperf system tune to work), but I did turn my laptop to airplane mode (after everything was installed, while the first benchmark was running) and then avoided doing much with it while running those benchmarks.

[gvanrossum]

I get a different set of results. The geometric mean is 1.13x faster. (But I have 14 failing benchmarks.) This is on yesterday's main branch (sadly I failed to get the exact commit). A few random comments:

• nbody: 1.16x slower, comparable to Dennis' result
• unpack_sequence: 1.23x faster, whereas Dennis had it 1.63x slower
• The other four that are slower in my run are similarly slower in Dennis' run
• python_startup and and python_startup_no_site are much faster in my run (maybe deepfreeze works better now?)
• raytrace is much faster, as is unpickle (why?)
• deltablue is only a little bit faster (1.40x vs. 1.36x)

I gave up at this point, but you get the picture -- I believe my results aren't completely off. Possibly the age or architecture of the hardware differs?

Slower (5):
- nbody: 138 ms +- 9 ms -> 160 ms +- 7 ms: 1.16x slower
- scimark_fft: 398 ms +- 28 ms -> 449 ms +- 21 ms: 1.13x slower
- scimark_sparse_mat_mult: 5.59 ms +- 0.57 ms -> 6.21 ms +- 0.71 ms: 1.11x slower
- meteor_contest: 123 ms +- 7 ms -> 134 ms +- 8 ms: 1.09x slower
- fannkuch: 497 ms +- 15 ms -> 541 ms +- 13 ms: 1.09x slower

Faster (33):
- python_startup_no_site: 45.5 ms +- 9.6 ms -> 28.2 ms +- 6.9 ms: 1.61x faster
- python_startup: 51.4 ms +- 7.0 ms -> 33.7 ms +- 9.3 ms: 1.52x faster
- raytrace: 641 ms +- 74 ms -> 441 ms +- 18 ms: 1.45x faster
- unpickle: 20.6 us +- 5.2 us -> 14.5 us +- 0.9 us: 1.42x faster
- deltablue: 7.84 ms +- 0.63 ms -> 5.62 ms +- 0.25 ms: 1.40x faster
- richards: 88.8 ms +- 7.1 ms -> 64.6 ms +- 3.5 ms: 1.38x faster
- scimark_sor: 240 ms +- 28 ms -> 187 ms +- 18 ms: 1.29x faster
- regex_v8: 34.4 ms +- 4.2 ms -> 27.6 ms +- 1.6 ms: 1.25x faster
- regex_effbot: 3.77 ms +- 0.40 ms -> 3.03 ms +- 0.16 ms: 1.24x faster
- unpickle_list: 6.36 us +- 1.42 us -> 5.14 us +- 0.50 us: 1.24x faster
- unpack_sequence: 99.1 ns +- 8.9 ns -> 80.6 ns +- 3.9 ns: 1.23x faster
- scimark_monte_carlo: 125 ms +- 12 ms -> 102 ms +- 6 ms: 1.23x faster
- go: 249 ms +- 16 ms -> 203 ms +- 13 ms: 1.23x faster
- chaos: 124 ms +- 10 ms -> 102 ms +- 5 ms: 1.22x faster
- sqlite_synth: 3.72 us +- 0.32 us -> 3.08 us +- 0.21 us: 1.21x faster
- regex_dna: 252 ms +- 22 ms -> 211 ms +- 10 ms: 1.19x faster
- 2to3: 530 ms +- 42 ms -> 445 ms +- 36 ms: 1.19x faster
- logging_silent: 181 ns +- 10 ns -> 153 ns +- 20 ns: 1.18x faster
- scimark_lu: 197 ms +- 17 ms -> 167 ms +- 11 ms: 1.18x faster
- logging_simple: 11.6 us +- 0.7 us -> 9.91 us +- 0.47 us: 1.17x faster
- pyflate: 749 ms +- 26 ms -> 642 ms +- 22 ms: 1.17x faster
- pickle_pure_python: 521 us +- 109 us -> 447 us +- 31 us: 1.17x faster
- float: 125 ms +- 10 ms -> 112 ms +- 5 ms: 1.12x faster
- xml_etree_process: 96.8 ms +- 5.9 ms -> 88.0 ms +- 6.1 ms: 1.10x faster
- xml_etree_parse: 197 ms +- 16 ms -> 182 ms +- 11 ms: 1.08x faster
- json_loads: 27.1 us +- 1.3 us -> 25.2 us +- 2.1 us: 1.08x faster
- pickle: 13.3 us +- 0.9 us -> 12.4 us +- 0.6 us: 1.07x faster
- spectral_norm: 169 ms +- 14 ms -> 159 ms +- 9 ms: 1.06x faster
- xml_etree_iterparse: 140 ms +- 15 ms -> 132 ms +- 8 ms: 1.06x faster
- telco: 7.98 ms +- 0.74 ms -> 7.62 ms +- 0.39 ms: 1.05x faster
- json_dumps: 14.7 ms +- 0.8 ms -> 14.1 ms +- 0.7 ms: 1.04x faster
- pidigits: 269 ms +- 16 ms -> 259 ms +- 11 ms: 1.04x faster
- pickle_dict: 32.9 us +- 1.7 us -> 32.1 us +- 1.8 us: 1.03x faster

Benchmark hidden because not significant (7): hexiom, logging_format, nqueens, pathlib, pickle_list, unpickle_pure_python, xml_etree_generate

Geometric mean: 1.13x faster
Ignored benchmarks (14) of .\310.json: chameleon, crypto_pyaes, django_template, genshi_text, genshi_xml, html5lib, mako, sqlalchemy_declarative, sqlalchemy_imperative, sympy_expand, sympy_integrate, sympy_str, sympy_sum, tornado_http

I would attach my JSON files but GitHub doesn't believe in JSON attachments.

[gvanrossum]

More details: the log of all changes to ceval.c since 3.11a6.

C:> g log v3.11.0a6..main .\Python\ceval.c 
commit 49e1e1e1bd59cac1855b1ef4dec05d649ebcd81a
Author: Mark Shannon <mark@hotpy.org>
Date:   Wed Mar 16 00:08:37 2022 +0000

    bpo-46841: Don't scan backwards in bytecode (GH-31901)

commit 099f75614100e88ed90b68d20a51a8d9c22f81a7
Author: Mark Shannon <mark@hotpy.org>
Date:   Tue Mar 15 17:06:21 2022 +0000

    bpo-45923: Decouple suspension of tracing from tracing flag. (GH-31908)

commit 13b041222399152acb555337572bd1d571734984
Author: Oleg Iarygin <oleg@arhadthedev.net>
Date:   Mon Mar 14 19:04:22 2022 +0300

    bpo-46920: Remove code that has explainers why it was disabled (GH-31813)

commit a89c29fbcc7e7e85848499443d819c3fab68c78a
Author: Brandt Bucher <brandtbucher@microsoft.com>
Date:   Fri Mar 11 16:31:50 2022 -0800

    bpo-46841: Add a _Py_SET_OPCODE macro (GH-31780)

commit 5498a61c7c25db6f9e76032aa9c5153d79e09889
Author: Brandt Bucher <brandtbucher@microsoft.com>
Date:   Tue Mar 8 07:53:22 2022 -0800

    bpo-46841: Don't use an oparg counter for `STORE_SUBSCR` (GH-31742)

commit f193631387bfee99a812e39b05d5b7e6384b57f5
Author: Brandt Bucher <brandtbucher@microsoft.com>
Date:   Mon Mar 7 11:45:00 2022 -0800

    bpo-46841: Use inline caching for calls (GH-31709)

[gvanrossum]

Another build (a day later) is 5% faster again, but I don't really believe these results. But that means I also don't really believe Dennis' results. We're waiting on a stable machine to run tests.

[sweeneyde]

I'd be more than happy to be proven wrong! I can try to re-run things within the next couple of days as well.