yarl==1.6.0 causing problems with Python 3.6

[YashTatkondawar]

Using latest yarl==1.6.0 is causing multiple issues with the Python 3.6

[webknjaz]

We need proof.

[cy8aer]

I can confirm problems but with Python 3.8 used via https://github.com/poljar/matrix-nio. There calling login in https://github.com/poljar/matrix-nio/blob/master/nio/client/async_client.py line 885 excepts an

 _make_netloc() got an unexpected keyword argument 'requote'

This error does not occur with yarl 1.5.1. Yes this is not as informational as needed but because the error occured from a CI/CD loopthis might be some additional information.

[webknjaz]

@cy8aer sounds similar to #485 but nobody succeeded coming up with a reproducer for it. Mind giving a try? Also, if it's in public CI, please post a link to the job. If not, try to find all the logs available, including those with traceback. We need to understand what's happening.

Without details, it's hard to investigate something like this. The CI is green FWIW.

P.S. Nobody in this issue specified an installation method (wheels vs sdist) and also if you use C-extensions vs pure-python fallbacks.

[cy8aer]

The main problem I have: The CI/CD creates a package for a smartphone app (SailfishOS). Testing is only possible in a virtual device or hardware. The whole thing is quite complex and the output and tracebacks can only be seen partly because nio is async and only partly shown.
I saw the problem because I built my system about 3 hours after deploying 1.6.0. And then nothing was working anymore. I try to get more information. Stand by...

[webknjaz]

Installation logs would also be necessary.

[serhiy-storchaka]

Could you please provide a traceback for this exception?

This error is possible if either the code calls _make_netloc() directly or implements a subclass of URL with overridden _make_netloc, but I do not see enything in the code of matrix-nio.

[cy8aer]

First of all (that is simple) two installation logs:
https://gitlab.com/cy8aer/bluepill/-/jobs/753973912 (armv7hl)
https://gitlab.com/cy8aer/bluepill/-/jobs/753973911 (amd64)

[parsa-saadatpanah]

I don't know if this is really related to this issue, but this is what we observed:
When yarl was updated to version 1.6.0, the aiohttp client started auto-generating the header 'Content-Type': 'text/plain; charset=utf-8' in GET requests (even the ones without any body). And some web servers were not happy about that header and were returning 400.

[webknjaz]

@serhiy-storchaka there was a similar problem with 1.5.0 and it somehow looked like the dist on PyPI was packaged wrong. I thought that it may be something similar but there's no module moves in 1.6.0.

@parsa-saadatpanah I don't see how your problem is related atm.

@cy8aer

First of all (that is simple) two installation logs:

these logs have

Collecting yarl<2.0,>=1.0
Downloading yarl-1.6.0.tar.gz (175 kB)

which effectively means that your build the source yourself instead of using platform-specific wheels that are available for at least x86_64 (none for arm though):

Building wheels for collected packages: multidict, yarl
Building wheel for multidict (PEP 517): started
Building wheel for multidict (PEP 517): finished with status 'done'
Created wheel for multidict: filename=multidict-4.7.6-cp38-cp38-linux_armv7l.whl size=183342 sha256=79bc78d24eef29a429f05ae6660c37209c85d72e94a65f7145f2201eab388c1c
Stored in directory: /tmp/pip-ephem-wheel-cache-selfy4nm/wheels/78/82/3a/e149e5c0e123a9094d46ec7249c95ceff6d63c90dcdd2c295d
Building wheel for yarl (PEP 517): started
Building wheel for yarl (PEP 517): finished with status 'done'
Created wheel for yarl: filename=yarl-1.6.0-cp38-cp38-linux_armv7l.whl size=309497 sha256=a366fac84e77365f92527150303eb8c1636936893eb2b47a5b7d4ea8bae97a4f
Stored in directory: /tmp/pip-ephem-wheel-cache-selfy4nm/wheels/d2/96/e7/cfe883f66405a91804b1e4cfbbccd32f40e97aa53150c941dc
Successfully built multidict yarl
Installing collected packages: unpaddedbase64, aiofiles, pyrsistent, attrs, six, jsonschema, logbook, hpack, hyperframe, h2, multidict, chardet, idna, yarl, async-timeout, aiohttp, h11, future, pycryptodome, peewee, atomicwrites, pycparser, cffi, python-olm, cachetools, matrix-nio

(this is for arm, your log for i486 is similar)

I've also noticed that you use a pure-Python flavor of aiohttp without C-extensions:

Collecting aiohttp<4.0.0,>=3.6.2
Downloading aiohttp-3.6.2-py3-none-any.whl (441 kB)

Now, depending on whether you have YARL_NO_EXTENSIONS=1 env var set in your build env, it may have produced yarl wheel with or without C-extensions. Since you have gcc, it's probably the former but it's still worth checking.

[cy8aer]

In my case it is no standard linux machine: It's the SDK for SailfishOS. There some parts are stripped down with python and self-compiling of wheels may be a good idea for an unkown hardware. This is why you have an arm image too and that is the image used on the devices. i486 is used for the SDK vm (and for special tablets).

[cy8aer]

Source of the installation process is here: https://gitlab.com/cy8aer/bluepill/-/blob/nio/build/build-ci.sh
But just ignore the last two lines: The first "upgrades" to 1.5.1, the second is the little fix for the problem I have with 1.5.1...
Yarl itself is installed with matrix-nio as a dependency.

[cy8aer]

That is the yarl dir inside the emulated image:

-rw-r--r-- 1 root root    154 Sep 24 09:34 __init__.py
-rw-r--r-- 1 root root   3636 Sep 24 09:34 __init__.pyi
drwxr-xr-x 2 root root   4096 Sep 24 14:51 __pycache__
-rw-r--r-- 1 root root    519 Sep 24 09:34 _quoting.py
-rw-r--r-- 1 root root 476396 Sep 24 09:34 _quoting_c.c
-rwxr-xr-x 1 root root 621484 Sep 24 09:34 _quoting_c.cpython-38-i386-linux-gnu.so
-rw-r--r-- 1 root root    447 Sep 24 09:34 _quoting_c.pyi
-rw-r--r-- 1 root root  11812 Sep 24 09:34 _quoting_c.pyx
-rw-r--r-- 1 root root   6516 Sep 24 09:34 _quoting_py.py
-rw-r--r-- 1 root root  34574 Sep 24 09:34 _url.py
-rw-r--r-- 1 root root     13 Sep 24 09:34 py.typed

(with quoting so.)

[cy8aer]

... and I ask myself how to compile the aiohttp...

[webknjaz]

Your build-ci.sh breaks your installation with
sed -i "s/, requote=False//g" /home/nemo/bluepill/build/lib/python3.8/site-packages/yarl/_url.py by removing requote=False from _make_netloc() method declaration here https://github.com/aio-libs/yarl/blob/663a3c7/yarl/_url.py#L759. And so when other parts of code legitimately call it with the arg, Python tells you that there's no arg (which is expected).

When you patch your env, you don't get what we ship. What you get is unsupported and you're solely responsible for breaking your own env, this has nothing to do with yarl.

Looks like this case is closed.

[cy8aer]

hm, then I need to open: Breaks 1.5.1 - without this patch I get the error message there. Last post: Ignore the last two lines...

[webknjaz]

Just upgrade then, we don't support old releases anyway.

[cy8aer]

https://gitlab.com/cy8aer/bluepill/-/tree/test-0.0.39
https://gitlab.com/cy8aer/bluepill/-/pipelines/194893395

stand by...

[cy8aer]

hm ashes over my head.
test-0.0.39 works on emulation and hardware.
So: works for me - but I did not open this issue...

[webknjaz]

Nobody else provided anything actionable, no demos, no proofs, no STRs, not even an explanation for what they meant under "something doesn't work". I have a proof in CI that everything supported works.

[cy8aer]

hm, next compilation, crash on hardware:

[D] unknown:0 - "PyOtherSide error: Traceback (most recent call last):\n\n  File \"/usr/share/harbour-bluepill/qml/components/ClientHandler.py\", line 15, in <module>\n    from bluepill.client import BluepillClientFactory\n\n  File \"/usr/share/harbour-bluepill/python/bluepill/client.py\", line 22, in <module>\n    import nio\n\n  File \"/usr/share/harbour-bluepill/lib/python3.8/site-packages/nio/__init__.py\", line 2, in <module>\n    from .client import *\n\n  File \"/usr/share/harbour-bluepill/lib/python3.8/site-packages/nio/client/__init__.py\", line 6, in <module>\n    from .async_client import AsyncClient, AsyncClientConfig, DataProvider\n\n  File \"/usr/share/harbour-bluepill/lib/python3.8/site-packages/nio/client/async_client.py\", line 47, in <module>\n    from aiohttp import (\n\n  File \"/usr/share/harbour-bluepill/lib/python3.8/site-packages/aiohttp/__init__.py\", line 6, in <module>\n    from .client import BaseConnector as BaseConnector\n\n  File \"/usr/share/harbour-bluepill/lib/python3.8/site-packages/aiohttp/client.py\", line 30, in <module>\n    from yarl import URL\n\n  File \"/usr/share/harbour-bluepill/lib/python3.8/site-packages/yarl/__init__.py\", line 1, in <module>\n    from ._url import URL, cache_clear, cache_configure, cache_info\n\n  File \"/usr/share/harbour-bluepill/lib/python3.8/site-packages/yarl/_url.py\", line 59, in <module>\n    class URL:\n\n  File \"/usr/share/harbour-bluepill/lib/python3.8/site-packages/yarl/_url.py\", line 131, in URL\n    _QUOTER = _Quoter(requote=False)\n\n  File \"yarl/_quoting.pyx\", line 192, in yarl._quoting._Quoter.__init__\n\nTypeError: __init__() got an unexpected keyword argument 'requote'\n"
[D] onError:190 - python error: Cannot import module: ClientHandler (Traceback (most recent call last):

  File "/usr/share/harbour-bluepill/qml/components/ClientHandler.py", line 15, in <module>
    from bluepill.client import BluepillClientFactory

  File "/usr/share/harbour-bluepill/python/bluepill/client.py", line 22, in <module>
    import nio

  File "/usr/share/harbour-bluepill/lib/python3.8/site-packages/nio/__init__.py", line 2, in <module>
    from .client import *

  File "/usr/share/harbour-bluepill/lib/python3.8/site-packages/nio/client/__init__.py", line 6, in <module>
    from .async_client import AsyncClient, AsyncClientConfig, DataProvider

  File "/usr/share/harbour-bluepill/lib/python3.8/site-packages/nio/client/async_client.py", line 47, in <module>
    from aiohttp import (

  File "/usr/share/harbour-bluepill/lib/python3.8/site-packages/aiohttp/__init__.py", line 6, in <module>
    from .client import BaseConnector as BaseConnector

  File "/usr/share/harbour-bluepill/lib/python3.8/site-packages/aiohttp/client.py", line 30, in <module>
    from yarl import URL

  File "/usr/share/harbour-bluepill/lib/python3.8/site-packages/yarl/__init__.py", line 1, in <module>
    from ._url import URL, cache_clear, cache_configure, cache_info

  File "/usr/share/harbour-bluepill/lib/python3.8/site-packages/yarl/_url.py", line 59, in <module>
    class URL:

  File "/usr/share/harbour-bluepill/lib/python3.8/site-packages/yarl/_url.py", line 131, in URL
    _QUOTER = _Quoter(requote=False)

  File "yarl/_quoting.pyx", line 192, in yarl._quoting._Quoter.__init__

TypeError: __init__() got an unexpected keyword argument 'requote'
)
[D] unknown:0 - "PyOtherSide error: Traceback (most recent call last):\n\n  File \"<string>\", line 1, in <module>\n\nNameError: name 'ClientHandler' is not defined\n"
[D] :78 - ClientHandler is now imported
[D] onError:190 - python error: Function not found: 'ClientHandler.do_init' (Traceback (most recent call last):

  File "<string>", line 1, in <module>

NameError: name 'ClientHandler' is not defined
)

[cy8aer]

I think I try the yarl test suite on the device. But not today...

[cy8aer]

Ok, with YARL_NO_EXTENSIONS=1 it works and I will work with this now (so no wheel for me).

[YashTatkondawar]

Nobody else provided anything actionable, no demos, no proofs, no STRs, not even an explanation for what they meant under "something doesn't work". I have a proof in CI that everything supported works.

Hi, sorry for the late reply. We have a very tight setup using yarl and so couldn't provide exact logs of what went wrong. But now it seems that the issue we were facing is logged as issue #509 . We have even seen that this behavior/issue is only with python 3.6 and not with python 3.7 So hope this may get resolved early.

[serhiy-storchaka]

See also aio-libs/aiohttp#4972.