Py311 is 10-20% slower in loops of empty and variant assignments than Py38, Py39 and Py310

[wangyi041228]

EDITED:
Py311 is 10-20% slower in big loops of empty or global variant assignments than Py38, Py39 and Py310, measuring with time or timeit.
Even though is's faster in other functions and libs.

[gvanrossum]

Can you show the code you used to measure this?

[wangyi041228]

env:
Win10 cmd

video(just upload):
https://www.youtube.com/watch?v=zhdF9eyi10M

result:
Python 3.11.0.b3
empty 216 193 216 204
a=i 388 357 431 391
sin(i) 1115 1050 1093 1066

Python 3.10.5
empty 168 176 177 178 171
a=i 290 309 308 315 292
sin(i) 1100 1081 1069 1073 1095

Python 3.9.13
Python 3.8.10
(just like Py310)

code:
import time
import math

t0 = time.perf_counter_ns()
for i in range(10_000_000):
...
t1 = time.perf_counter_ns()
print(t1 - t0)

t0 = time.perf_counter_ns()
for i in range(10_000_000):
pass
t1 = time.perf_counter_ns()
print(t1 - t0)

t0 = time.perf_counter_ns()
for i in range(10_000_000):
a = i
t1 = time.perf_counter_ns()
print(t1 - t0)

t0 = time.perf_counter_ns()
for i in range(10_000_000):
a = math.sin(i)
t1 = time.perf_counter_ns()
print(t1 - t0)

[gvanrossum]

Can you try again with the whole thing inside a function (which you call just once)? Your current code writes globals (i and a) and that may be slowing things down.

[wangyi041228]

They're all faster, but Python311 is still 10-20% slower than Python310 for empty loops.

For empty loops:
Python311 0.096s
Python310 0.089s

For a=i:
Python311 0.123s
Python310 0.122-0.132s

For sin:
Python311 0.70s
Python310 0.850s

[methane]

please use pyperformance or similar tool to call function hundreds of times.
Many optimizations will not work with functions called only once.

[notatallshaw]

I think it would still be good to know if Python 3.11 has performance regressions for code that doesn't call the same function multiple times (such as in simple scripts) so I tried creating OPs test cases in a way that others can hopefully more easily reproduce.

Test environment:
AMD Ryzen 9 5900X
2 x 32 GBs DDR4 clocked at 3200 MT/s
Windows 10 Version 21H2 OS Build 19044.1766
Python 3.10.5, Python 3.11.0b3 (both clean installs)

I have not set up anything special in terms of performance control, I am just running the tests several times to see if there is a consistent and big enough difference to make any conclusions (though my PC is well ventilated so there is no thermal throttling).

Test execution format:

py -{python_version} -m timeit [-s "{setup_code}"] "{run_code}"

Test 1
Run code: for i in range(10_000_000): pass
High level result: Python 3.11 is consistently 5-10% slower

Test 2
Run code: for i in range(10_000_000): a=i
High level result: After several dozen runs no significant statistical difference

Test 3
Setup code: import math
Run code: for i in range(10_000_000): a=math.sin(i)
High level result: Python 3.11 is consistently 10-20% faster

Not sure this information helps anyone, but there it is.

[gvanrossum]

The math.sin() test is pretty uninteresting,
but the ‘pass’ case deserves more investigation.
@sweeneyde ?

[sweeneyde]

It always seems to be very hard to get stable Windows benchmarks. I got these results:

## Python 3.10.4 ##
py -3.10 -m pyperf timeit -s "global i" "for i in range(10_000_000): pass" --rigorous

(I did that 5 times)
Mean +- std dev: 315 ms +- 7 ms
Mean +- std dev: 320 ms +- 10 ms
Mean +- std dev: 317 ms +- 10 ms
Mean +- std dev: 321 ms +- 10 ms
Mean +- std dev: 320 ms +- 8 ms

## Python 3.11.0b3 ##
py -3.11 -m pyperf timeit -s "global i" "for i in range(10_000_000): pass" --rigorous

(I did that 5 times)
Mean +- std dev: 322 ms +- 12 ms
Mean +- std dev: 320 ms +- 8 ms
Mean +- std dev: 322 ms +- 7 ms
Mean +- std dev: 323 ms +- 6 ms
Mean +- std dev: 316 ms +- 14 ms

[wangyi041228]

AMD Ryzen 9 5900HX with Radeon Graphics 3.30 GHz
Windows 10 Pro 21H2 19044.1766

I got "WARNING: unable to increase process priority" while measuring with py311.

## Python 3.10.5 ##
py -3.10 -m pyperf timeit -s "global i" "for i in range(10_000_000): pass" --rigorous

Mean +- std dev: 195 ms +- 9 ms
Mean +- std dev: 193 ms +- 6 ms

py -3.11 -m pyperf timeit -s "global i" "for i in range(10_000_000): pass" --rigorous
WARNING: unable to increase process priority
Mean +- std dev: 216 ms +- 9 ms
Mean +- std dev: 214 ms +- 6 ms

[gvanrossum]

Does installing psutil for 3.11 make a difference?

[gvanrossum]

I can repro the results from @wangyi041228 on my Windows laptop (although I wasn't very rigorous). I'm not sure why we see such different results where @sweeneyde saw basically no difference (at least, the difference of the means was a fraction of the std dev) -- it'd be surprising if the background noise always affected 3.11 more than 3.10.

We talked about this in the Faster CPython perf team and did not come up with any good explanation (although we expect the issue doesn't affect real code, since there's always some code in the loop). (@markshannon @brandtbucher)

I'm sure my Python 3.10.5 and 3.11.0b3 binaries all came from python.org, and I have to assume they were built using the same toolchain.

I reproduced (at least some of) the difference on my Mac as well, so I don't think it's the MSVC compiler.

I looked at the disassembly, and the heart of the loop is equivalent in 3.10 and 3.11:

3.10:

        >>    8 FOR_ITER                 2 (to 14)
             10 STORE_GLOBAL             1 (i)
             12 JUMP_ABSOLUTE            4 (to 8)

3.11:

        >>   32 FOR_ITER                 2 (to 38)
             34 STORE_GLOBAL             1 (i)
             36 JUMP_BACKWARD            3 (to 32)

Interestingly, getting rid of global i did not change things. So it's also not something in the global dict that slows down 3.11 here.

Here's a thought. Could it be the _PyCode_Warmup() call in JUMP_BACKWARD? That's an inline function, but still, it could be an extra memory load?

static inline void
_PyCode_Warmup(PyCodeObject *code)
{
    if (code->co_warmup != 0) {
        code->co_warmup++;
        if (code->co_warmup == 0) {
            _PyCode_Quicken(code);
        }
    }
}

[notatallshaw]

It always seems to be very hard to get stable Windows benchmarks.

Agreed, though I think it helps my machine has a high count of homogeneous CPU cores, there is no thermal throttling, and I have all virus scanning turned off. I created 2 virtual environments with pyperf and psutils (and both my Python installs are from python.org), I ran the pyperf several times and interleaved the runs between Python 3.10 and 3.11, I never saw the mean change more than 2% for any of the runs:

Python 3.10

(.venv310) PS C:\Users\damia> py  -m pyperf timeit -s "global i" "for i in range(10_000_000): pass" --rigorous
.........................................
Mean +- std dev: 168 ms +- 5 ms
(.venv310) PS C:\Users\damia> py  -m pyperf timeit -s "global i" "for i in range(10_000_000): pass" --rigorous
.........................................
Mean +- std dev: 169 ms +- 6 ms
(.venv310) PS C:\Users\damia> py  -m pyperf timeit -s "global i" "for i in range(10_000_000): pass" --rigorous
.........................................
Mean +- std dev: 168 ms +- 5 ms

Python 3.11

(.venv311) PS C:\Users\damia> py  -m pyperf timeit -s "global i" "for i in range(10_000_000): pass" --rigorous
.........................................
Mean +- std dev: 187 ms +- 12 ms
(.venv311) PS C:\Users\damia> py  -m pyperf timeit -s "global i" "for i in range(10_000_000): pass" --rigorous
.........................................
Mean +- std dev: 188 ms +- 16 ms
(.venv311) PS C:\Users\damia> py  -m pyperf timeit -s "global i" "for i in range(10_000_000): pass" --rigorous
.........................................
Mean +- std dev: 186 ms +- 13 ms

Though on my machine I can not reproduce anything other than an empty loop where 3.11 is slower, as soon as the loop has code in it I don't see any performance regression.

[brandtbucher]

Here's a thought. Could it be the _PyCode_Warmup() call in JUMP_BACKWARD? That's an inline function, but still, it could be an extra memory load?

To see what's actually being run on 3.11+, pass adaptive=True to dis. The JUMP_BACKWARD is quickened into a JUMP_BACKWARD_QUICK, which skips the _PyCode_Warmup dance.

[brandtbucher]

For those who can reproduce: how does the slowdown vary with the number of iterations? Is it constant, or does it grow larger/smaller for different range sizes?

[brandtbucher]

The single-digit fast-path for PyLong_FromLong did change slightly between 3.10 and 3.11. I don't see any obvious reason why the new one would be slower, though.

Do we see a similar slowdown when dropping the loop into C with:

pyperf timeit -s "from collections import deque; r = range(10_000_000)" "deque(r, 0)" --rigorous

This version simply exhausts the iterator with as little overhead as possible. It might give us an idea if this slowdown is happening in the VM or not.