Regression in Python 3.11 with significative performance loss

[Ousret]

Bug report

I am reporting a regression regarding performance that I recently noticed in Python 3.11.0b1.
Got a five times slower performance regarding charset-normalizer and I currently do not know why.

Your environment

Tested on a Linux platform (Fedora 35).

Reproducer

git clone https://github.com/Ousret/charset_normalizer.git
cd charset_normalizer
pip install -U pip setuptools
pip install -r dev-requirements.txt
pytest

Python 3.10

==== 126 passed in 3.43s ====

Python 3.11.alpha4

==== 126 passed in 6.32s ====

Python 3.11.0b1

==== 126 passed in 20.15s ====

I can concur that the shown results are reproducible at will.

[rhettinger]

I just ran this using the standard python.org Mac builds running on an M1 Pro and show a 13% improvement from Py3.10 to Py3.11beta.

I suspect you have a build issue. Python performance now relies heavily on PGO/LTO builds and on the compiler inlining frequently functions that used to be macros.

[Ousret]

Thanks for the quick response,

I have re-run and re-built from source with no warning (verbose build) Linux Fedora 35 and still the same results. One scenario on direct host and the second using a VM.

Then I fetched the official build for Windows (no-VM) and got the same results. (info. Windows 11)

----------- coverage: platform win32, python 3.11.0-beta-1 -----------
....
===== 126 passed in 19.08s =====

[hugovk]

I also see Python 3.11.0b1 is slower than 3.10.4 on macOS M1 Monterey 12.3.1, both using the https://www.python.org/downloads/macos/ installers.

Running the slowest test file and printing the 10 slowest tests:

$ /Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10 -m pytest tests/test_full_detection.py --durations 10
...
======================================================================= slowest 10 durations =======================================================================
0.51s call     tests/test_full_detection.py::test_elementary_detection[sample-french-1.txt-cp1252-French]
0.21s call     tests/test_full_detection.py::test_elementary_detection[sample-turkish.txt-cp1254-Turkish]
0.10s call     tests/test_full_detection.py::test_elementary_detection[sample-arabic-1.txt-cp1256-Arabic]
0.05s call     tests/test_full_detection.py::test_elementary_detection[sample-russian.txt-mac_cyrillic-Russian]
0.05s setup    tests/test_full_detection.py::test_elementary_detection[sample-arabic-1.txt-cp1256-Arabic]
0.04s call     tests/test_full_detection.py::test_elementary_detection[sample-greek.txt-cp1253-Greek]
0.04s call     tests/test_full_detection.py::test_elementary_detection[sample-spanish.txt-utf_8-Spanish]
0.03s call     tests/test_full_detection.py::test_elementary_detection[sample-hebrew-2.txt-cp1255-Hebrew]
0.03s call     tests/test_full_detection.py::test_elementary_detection[sample-french.txt-utf_8-French]
0.02s call     tests/test_full_detection.py::test_elementary_detection[sample-bulgarian.txt-utf_8-Bulgarian]
======================================================================== 17 passed in 1.33s ========================================================================

$ /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11 -m pytest tests/test_full_detection.py --durations 10
...
======================================================================= slowest 10 durations =======================================================================
3.55s call     tests/test_full_detection.py::test_elementary_detection[sample-french-1.txt-cp1252-French]
1.53s call     tests/test_full_detection.py::test_elementary_detection[sample-turkish.txt-cp1254-Turkish]
0.40s call     tests/test_full_detection.py::test_elementary_detection[sample-arabic-1.txt-cp1256-Arabic]
0.37s call     tests/test_full_detection.py::test_elementary_detection[sample-russian.txt-mac_cyrillic-Russian]
0.27s call     tests/test_full_detection.py::test_elementary_detection[sample-greek.txt-cp1253-Greek]
0.19s call     tests/test_full_detection.py::test_elementary_detection[sample-hebrew-2.txt-cp1255-Hebrew]
0.19s call     tests/test_full_detection.py::test_elementary_detection[sample-spanish.txt-utf_8-Spanish]
0.17s call     tests/test_full_detection.py::test_elementary_detection[sample-french.txt-utf_8-French]
0.15s call     tests/test_full_detection.py::test_elementary_detection[sample-chinese.txt-big5-Classical Chinese]
0.12s call     tests/test_full_detection.py::test_elementary_detection[sample-korean.txt-cp949-Korean]
======================================================================== 17 passed in 7.51s ========================================================================

---

We can also see 3.11-dev tests are slower (27s) than 3.4-3.10 (5-8s) with Ubuntu on GitHub Actions, but the on-flight Python setup might be different for 3.11-dev than the official releases:

https://github.com/Ousret/charset_normalizer/actions/runs/2291918919

[neonene]

I can confirm this on Windows release builds from 2bde682 (Quicken code in-place).

cc @brandtbucher @markshannon

[neonene]

What I know right now is that 2bde682 and newer don't install site-packages/coverage/tracer.cp311-win_amd64.pyd with the reproducer. Older commits can get the same slowdown when the dll is missing.

[neonene]

python -m pip install coverage would be enough to verify.

[vstinner]

coverage is not fully compatible with Python 3.11 yet: nedbat/coveragepy#1368 is not merged yet (and so is not part of any public coverage release). It uses a C extensions (tracer) for better performance. If the C extension cannot be built (which is the case right now on Python 3.11), well, you get worse performance.

This issue doesn't seem to be related to Python itself. I suggest closing it and continue the discussion in the coverage project.

[Ousret]

With more testing, I can confirm it too. I was confused with the alpha results, never suspected it.
Thank you for the investigation.

I am closing it now.