Unable to install on Python 3.11

[mohd-akram]

Describe the bug

Due to the use of the cchardet dependency, which seems to have been abandoned (PyYoshi/cChardet#77), BlackSheep fails to install on Python 3.11:

/usr/bin/clang -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -Isrc/ext/uchardet/src -I/opt/local/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c src/cchardet/_cchardet.cpp -o build/temp.macosx-12.0-x86_64-cpython-311/src/cchardet/_cchardet.o
src/cchardet/_cchardet.cpp:196:12: fatal error: 'longintrepr.h' file not found
  #include "longintrepr.h"
           ^~~~~~~~~~~~~~~
1 error generated.
error: command '/usr/bin/clang' failed with exit code 1

EDIT: After manually installing Cython, I can bypass the above error, but now get the same error for blacksheep directly:

/usr/bin/clang -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -pipe -Os -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/opt/local/Library/Frameworks/Python.framework/Versions/3.11/include/python3.11 -c blacksheep/url.c -o build/temp.macosx-12.0-x86_64-cpython-311/blacksheep/url.o -O2
blacksheep/url.c:198:12: fatal error: 'longintrepr.h' file not found
  #include "longintrepr.h"
           ^~~~~~~~~~~~~~~
1 error generated.
error: command '/usr/bin/clang' failed with exit code 1

Seems to be related to cython/cython#4461.

[RobertoPrevato]

Hi @mohd-akram,
Thanks for opening the issue. I already verified last weekend that it's sufficient to upgrade dependencies (pinned versions) and after this, BlackSheep already supports Python 3.11 (the full test suite passes).

I'm going to push a new release ASAP (probably tonight) with the necessary changes to use BlackSheep with Python 3.11.

[RobertoPrevato]

Fixed by #277

[rodolfolottin]

hi @RobertoPrevato , thanks for all your work!

Just wanted to flag that I still have the same error:

Installing collected packages: cchardet, PyYAML, MarkupSafe, itsdangerous, httptools, Jinja2, essentials-openapi, blacksheep
  DEPRECATION: cchardet is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for cchardet ... error
  error: subprocess-exited-with-error

  × Running setup.py install for cchardet did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      running install
      /Users/rod/.pyenv/versions/3.11.0/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.macosx-12.5-x86_64-cpython-311
      creating build/lib.macosx-12.5-x86_64-cpython-311/cchardet
      copying src/cchardet/version.py -> build/lib.macosx-12.5-x86_64-cpython-311/cchardet
      copying src/cchardet/__init__.py -> build/lib.macosx-12.5-x86_64-cpython-311/cchardet
      running build_ext
      building 'cchardet._cchardet' extension
      creating build/temp.macosx-12.5-x86_64-cpython-311
      creating build/temp.macosx-12.5-x86_64-cpython-311/src
      creating build/temp.macosx-12.5-x86_64-cpython-311/src/cchardet
      creating build/temp.macosx-12.5-x86_64-cpython-311/src/ext
      creating build/temp.macosx-12.5-x86_64-cpython-311/src/ext/uchardet
      creating build/temp.macosx-12.5-x86_64-cpython-311/src/ext/uchardet/src
      creating build/temp.macosx-12.5-x86_64-cpython-311/src/ext/uchardet/src/LangModels
      clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/usr/local/opt/openssl@3/include -Isrc/ext/uchardet/src -I/Users/rod/.pyenv/versions/3.11.0/include/python3.11 -c src/cchardet/_cchardet.cpp -o build/temp.macosx-12.5-x86_64-cpython-311/src/cchardet/_cchardet.o
      src/cchardet/_cchardet.cpp:196:12: fatal error: 'longintrepr.h' file not found
        #include "longintrepr.h"
                 ^~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> cchardet

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

[RobertoPrevato]

Hi @rodolfolottin,
Thank You for the heads up - I suspect it's only because I still need to release a new version to PyPI.
I am going to release to PyPI in the next hours, unless I encounter any issue.

[UPDATE]
ah no, you are right indeed. Weird, because I have a Python 3.11 virtual environment in which I installed cchardet, but I get again an error in a new virtual environment. Investigating... ⌚

[RobertoPrevato]

pip is still a mystery to me sometimes. I have now a Python 3.11 virtual environment in which I could install, after several attempts, cchardet the last weekend. I cannot install it in a new virtual environment, even using the same commands I used last Sunday.

[RobertoPrevato]

@rodolfolottin - just to share something curious with you.

This is what's happening: if you first install Cython, then it's possible to install cchardet.
This of course doesn't work in a distribution package - and it might be caused by some too smart / ambitious logic happening in cchardet setup.py (I like to keep these files as simple and stupid as possible).

~/projects/github/blacksheep (feature/python311)$ pip install cchardet
Collecting cchardet
  Using cached cchardet-2.1.7.tar.gz (653 kB)
  Preparing metadata (setup.py) ... done
Installing collected packages: cchardet
  DEPRECATION: cchardet is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for cchardet ... done
Successfully installed cchardet-2.1.7

Thank You for your comment! I will revert making cchardet an optional dependency: after all it's only used in a single line of code to detect an encoding, when text is not utf8.

[RobertoPrevato]

It seems to be fixed now. 👍🏼
Now I am investigating support for Python 3.11 in GitHub Workflows. #278

[mohd-akram]

@RobertoPrevato You might consider switching to charset_normalizer.

[RobertoPrevato]

@mohd-akram thanks for the information, I'll look into charset_normalizer.
For now, this specific issue can be closed. Before I saw your comment, I released the new version 1.2.8 to PyPI, and it includes wheel packages for Python 3.11.

[mohd-akram]

@RobertoPrevato Makes sense. I can confirm that the new version works. Thank you for the prompt fix!

[rodolfolottin]

@RobertoPrevato thank you very much!!