Latest pip prevents installation with Python > 3.9 due to cchardet

[mementum]

Issue #276 was opened for Python 3.11 but this also affects, for example installing under Windows with Python 3.10

The current determination of whether to use cchardet or cchardet happens here

BlackSheep/blacksheep/messages.pyx

Lines 7 to 10 in 34ae494

	try:
	import cchardet as chardet
	except ImportError:
	import chardet

And it is then used here

BlackSheep/blacksheep/messages.pyx

Line 157 in 34ae494

return body.decode(chardet.detect(body)['encoding'])

The current requirements.txt is somehow conflicting, because it allows skipping installation of cchardet for Python 3.11 and the same time says that chardet (the non C version, i.e.: pure python) is also requirement for all versions.

BlackSheep/requirements.txt

Lines 3 to 8 in 34ae494

	cchardet==2.1.7; python_version < '3.11'
	mock==5.0.1; python_version < '3.8'
	certifi==2022.12.7
	cffi==1.15.1
	chardet==3.0.4
	charset-normalizer==2.1.1

And charset-normalizer, as suggested in issue #276 should be used as a replacement for cchardet. Surprisingly it is a requirement but it is not used.

Suggested actions

1. Drop cchardet as a hard requirement for anything above 3.9. Because there are wheels up to that version. i.e.
   Change cchardet==2.1.7; python_version < '3.11' to cchardet==2.1.7; python_version < '3.10'

2. Change the imports to also import charset-normalizer and use it. The detect function is chardet compatible.

      try:
          import cchardet as chardet
      except ImportError:
          try:
              import charset_normalizer as chardet
          except ImportError
              import chardet

   This is actually redundant given that charset-normalizer has been added a a hard requirement to requirements.txt. A better approach would be

      try:
          import cchardet as chardet
      except ImportError:
          import charset_normalizer as chardet

   And drop the pure python chardet altogether from requirements.txt. Both blacksheep and charset-normalizer are good for python 3.8 => 3.11 (which is the minimum common denominator, taking blacksheep as the restriction)

The detect portion of the code needs no change.

Should you need a pull-request, don't hesitate to say: yes and I will submit one

[RobertoPrevato]

Hi @mementum
Thank You for the analysis - I must have done a mistake with the requirements file. And Yes, I'll be grateful if you can submit a PR for the cchardet issue.

Please note anyway that the requirements.txt file describes the development dependencies. Those include libraries used in integration tests like Flask, requests, etc. The setup.py file describes the actual dependencies of blacksheep. Side note: already have in mind to migrate to pyproject.toml, and I started doing it for other projects, but I didn´t take the time to look into packing Cython extensions that way, yet.

[RobertoPrevato]

@mementum
It seems the problem is not only about cchardet. It's the new version of pip that doesn't accept packages without wheels. Also, the CI build in Windows doesn't work anymore because the --install-options has been deprecated.

I'll need to look into this anyway - I was planning to move to pyproject.toml as I did for other libraries, but I didn't follow the (build) news so much to know all details. I'll be grateful if you submit a PR for the cchardet issue, I'll look into the other packages and into migrating to pyproject.toml.

[RobertoPrevato]

@mementum I hope you didn´t start preparing a PR in the last couple of hours, because I ended up fixing the build issue and upgrading several dependencies. After reading a bit about charset-normalizer, I decided to replace altogether both chardet and cchardet with it. It looks ideal!

Sorry for the three mentions in my comments. Thank You for the heads up about charset-normalizer!

[RobertoPrevato]

Fixed on main - still need to fix on v1 branch and publish to PyPi.

[RobertoPrevato]

Hi @mementum
This has been fixed and released with

• https://github.com/Neoteroi/BlackSheep/releases/tag/v1.2.14

It wasn´t affecting only Windows, all platforms with Python 3.10 and 3.11, because of a breaking change in the latest pip.

Thank You for reporting the issue and recommending charset-normalizer!

[93578237]

You could use https://github.com/faust-streaming/cChardet as active fork