Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WebsocketsProviderV2 AttributeError: 'NoneType' object has no attribute 'method' #3075

Closed
dmkulazhenko opened this issue Aug 14, 2023 · 6 comments

Comments

@dmkulazhenko
Copy link
Contributor

dmkulazhenko commented Aug 14, 2023

  • Version: 6.8.0
  • Python: 3.9.16
  • OS: Linux 6.1.31-2-MANJARO

What was wrong?

I'm just trying to subscribe to a node using WS and eth_subscribe to get notifications about new blocks.
Took code from docs.

Modified it a little bit, so came up with:

import asyncio
import time
from typing import cast

from eth_typing import HexStr
from web3 import AsyncWeb3
from web3.providers import WebsocketProviderV2

start = time.time()


async def ws_v2_subscription_context_manager_example():
    async with AsyncWeb3.persistent_websocket(
        WebsocketProviderV2(
            "wss://..."
        )
    ) as w3:
        # subscribe to new block headers
        # noinspection PyTypeChecker
        subscription_id: HexStr = cast(
            HexStr, (await w3.eth.subscribe("newHeads"))["result"]
        )
        print(f"subscribed: {subscription_id}")

        unsubscribed = False
        while not unsubscribed:
            async for response in w3.listen_to_websocket():
                try:
                    print(
                        "block_number",
                        int(response["params"]["result"]["number"], 16),
                    )
                except Exception as e:
                    print("ERROR", str(e))

                if time.time() - start > 30:
                    print("unsubscribing...")
                    unsubscribed = await w3.eth.unsubscribe(subscription_id)
                    print("unsubscribed:", unsubscribed)
                    break


async def main():
    await ws_v2_subscription_context_manager_example()


if __name__ == "__main__":
    asyncio.run(main())

It crashes:

subscribed: 0xcab36d5d4b328aa605891a693fb01118
block_number 17914393
Traceback (most recent call last):
  File "/home/...../test.py", line 48, in <module>
    asyncio.run(main())
  File "/home/.../.pyenv/versions/3.9.16/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/.../.pyenv/versions/3.9.16/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/...../test.py", line 44, in main
    await ws_v2_subscription_context_manager_example()
  File "/home/...../test.py", line 27, in ws_v2_subscription_context_manager_example
    async for response in w3.listen_to_websocket():
  File "/home/..../venv/lib/python3.9/site-packages/web3/manager.py", line 315, in _ws_recv_stream
    if request_info.method == "eth_subscribe" and "result" in response.keys():
AttributeError: 'NoneType' object has no attribute 'method'

Process finished with exit code 1

How can it be fixed?

Seems to be in https://github.com/ethereum/web3.py/blob/main/web3/manager.py#L296

async def _ws_recv_stream(self) -> AsyncGenerator[RPCResponse, None]:
     ...
     request_info = self._provider._get_request_information_for_response(response)

     if request_info is None:
         self.logger.debug("No cache key found for response, returning raw response")
         yield response
     
     if request_info.method == "eth_subscribe" and "result" in response.keys():
     ...
     yield apply_result_formatters(result_formatters, partly_formatted_response)

should look like

async def _ws_recv_stream(self) -> AsyncGenerator[RPCResponse, None]:
     ...
     request_info = self._provider._get_request_information_for_response(response)

     if request_info is None:
         self.logger.debug("No cache key found for response, returning raw response")
         yield response
     else:
         if request_info.method == "eth_subscribe" and "result" in response.keys():
         ...
         yield apply_result_formatters(result_formatters, partly_formatted_response)

To be honest, I dont have a lot of time to invest in this problem 😞.
But self._provider._get_request_information_for_response tries to look for some cache by md5hash(subscription_id). I haven't digged in this caching mechanism and why is it needed, but seems to be if request_info is None — result should be yielded right away without something like applying of result_formatters and re-yielding after returning of scope to generator.

btw, after this dirty fix, at least for me, code works as expected 😄


Also, there is another minor issues https://github.com/ethereum/web3.py/blob/main/web3/eth/async_eth.py#L680 async def subscribe(...) -> HexStr returns full-response (dict), but not HexStr as expected to be subscription_id.

@fselmo fselmo mentioned this issue Aug 14, 2023
1 task
@fselmo
Copy link
Collaborator

fselmo commented Aug 14, 2023

Hey @dmkulazhenko, thanks for the bug report. That yield should definitely be isolated. I'm not sure what's going on with your eth_subscribe response yet as I can't reproduce that on my end but I will take a closer look. If you are connecting to a particular chain / provider that kind of information might help as well. Maybe even what the response object (dict) looks like.

@dmkulazhenko
Copy link
Contributor Author

I'm not sure what's going on with your eth_subscribe response yet as I can't reproduce that on my end but I will take a closer look. If you are connecting to a particular chain / provider that kind of information might help as well. Maybe even what the response object (dict) looks like.

@fselmo, Im using ankr node.

Code

import asyncio
import time

from eth_typing import HexStr
from web3 import AsyncWeb3
from web3.providers import WebsocketProviderV2

start = time.time()


async def ws_v2_subscription_context_manager_example():
    async with AsyncWeb3.persistent_websocket(
        WebsocketProviderV2(
            "wss://rpc.ankr.com/eth/ws/<token>"
        )
    ) as w3:
        subscription = await w3.eth.subscribe("newHeads")
        print(f"Subscription full: {subscription}")
        # noinspection PyTypeChecker
        subscription_id: HexStr = subscription["result"]
        print(f"subscribed: {subscription_id}")

        unsubscribed = False
        while not unsubscribed:
            async for response in w3.listen_to_websocket():
                try:
                    print(
                        "block_number",
                        int(response["params"]["result"]["number"], 16),
                    )
                except Exception as e:
                    print("ERROR", str(e))

                if time.time() - start > 30:
                    print("unsubscribing...")
                    unsubscribed = await w3.eth.unsubscribe(subscription_id)
                    print("unsubscribed:", unsubscribed)
                    break


async def main():
    await ws_v2_subscription_context_manager_example()


if __name__ == "__main__":
    asyncio.run(main())

Stdout:

Subscription full: {'jsonrpc': '2.0', 'id': None, 'result': '0xe902b9e85c33e30194b7a41e024a0574'}
subscribed: 0xe902b9e85c33e30194b7a41e024a0574
block_number 17919747
block_number 17919748
block_number 17919749
unsubscribing...
unsubscribed: True

Also, let me test on my own node.

@dmkulazhenko
Copy link
Contributor Author

dmkulazhenko commented Aug 15, 2023

@fselmo, so I tested it with my own node (erigon version 2.42.0-stable-808225f7), and results are interesting:

import asyncio
import time

from web3 import AsyncWeb3
from web3.providers import WebsocketProviderV2

start = time.time()


async def ws_v2_subscription_context_manager_example():
    async with AsyncWeb3.persistent_websocket(
        WebsocketProviderV2(
            "<node_url>"
        )
    ) as w3:
        subscription = await w3.eth.subscribe("newHeads")
        print(f"Subscription full: {subscription}")
        try:
            subscription_id = subscription["result"]
        except TypeError:
            subscription_id = subscription

        unsubscribed = False
        while not unsubscribed:
            async for response in w3.listen_to_websocket():
                try:
                    print(response)
                except Exception as e:
                    print("ERROR", str(e))

                if time.time() - start > 30:
                    print("unsubscribing...")
                    unsubscribed = await w3.eth.unsubscribe(subscription_id)
                    print("unsubscribed:", unsubscribed)
                    break


async def main():
    await ws_v2_subscription_context_manager_example()


if __name__ == "__main__":
    asyncio.run(main())

Erigon:

Subscription full: 0x587f28f5142b1016bc9315ebf4a327cb
AttributeDict({'parentHash': HexBytes('0xce9267517aa60c5ac3af4ca4f2fc08a70d1da34a6072038b8f0bd0c958dc38a6'), 'sha3Uncles': HexBytes('0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347'), 'miner': '0x1f9090aaE28b8a3dCeaDf281B0F12828e676c326', 'stateRoot': HexBytes('0x00627514e11f214fee22dbba8523342ac4a329e9e122da59280a51c6b9ddf45d'), 'transactionsRoot': HexBytes('0x6d2ad669a233ffe7d56582efb3e08b041706e6e8e555122f354af4f221f9dd98'), 'receiptsRoot': HexBytes('0x219a0eefbc8f84f8cf3e2780cbd543d6af30997c0fbece83bf2750434d42dc16'), 'logsBloom': HexBytes('0xf977adbc576f2ceeba8b5f96faf5ff36efbb9df5ee5fb237ac59fe94ff6decbb8e5587e5f3aefb377ac37de7affebf7feed577d9ebcef7ff7be56ce64138a6777d76c73fcfdbb9ab5efbff2fed1d38e7bef3b89a3af87db0ebe63f4b9563ff3d93f64ef7ffdf75b73fdbdffa44e02f78f6bafa6b27dd3ff9f67d3fb1f9eb0f2b7fc2ebede69d87d997f9d3fd3fb69e7f7f37a1b1b3799effff4cd776fcbf7cb5fbeff1eb9fbcefe1b43f71ff17e6a6cccf7b7f7f6a5a21a5b97bc7ff3e8ef77f76f01527e5db2fffed1b6bea4bfe755f3ffc5eef55e62b7f4341efafda9dec0da77bfc0ebdbf4cf6d0ff2d6ab3fd7ffcdd35f7dc3bf1ebe4dcd8a59f7d87cfee'), 'difficulty': 0, 'number': 17919806, 'gasLimit': 30000000, 'gasUsed': 29997695, 'timestamp': 1692097211, 'extraData': HexBytes('0x7273796e632d6275696c6465722e78797a'), 'mixHash': HexBytes('0x821b1d6b7599ea1d3fab78e52141ba7e44e9e3e1919710982dbf166cc5c34329'), 'nonce': HexBytes('0x0000000000000000'), 'baseFeePerGas': 14483682681, 'withdrawalsRoot': HexBytes('0x09c2bf023d84758a1595ba86f0b7e4765bbd4c936fb9a965acd76cbf2899ff46'), 'hash': HexBytes('0x33ddbbcf10424b8a017ebfe23acc644ab366b66f4a921879141b7478e41cce8a')})
AttributeDict({'parentHash': HexBytes('0x33ddbbcf10424b8a017ebfe23acc644ab366b66f4a921879141b7478e41cce8a'), 'sha3Uncles': HexBytes('0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347'), 'miner': '0x95222290DD7278Aa3Ddd389Cc1E1d165CC4BAfe5', 'stateRoot': HexBytes('0x9c37ce9de099b154d11c5c8440e66d3705216c7389954f6e18616935aa46eaa6'), 'transactionsRoot': HexBytes('0xa9c79ca159ef826ab1468055bec1daaa0a2d690cfeb48fab87656ab83cbfbc10'), 'receiptsRoot': HexBytes('0x133a76aaa9123079fe426dc82a920a16a28ba5e4691b2973d0adeebe37073bb1'), 'logsBloom': HexBytes('0xffe7f5ff6f57f75fdbdffff7e3b4f7be7578d5fbffd5cff7ef6fdf6fefedbb76feaedbdfd7eddf7bfdfff7772dfdbffdf6ffbff9bd84ef7afb0ff68ffdac7bff3feee5db2ffeefffebf37edfff8c3eeda7bf563ddbefefafebfcefbf8ffe3c4fff5fddf7d647ef77bbfdf9f75bfadffdcbfae2fcbfffdefedf4cfe7ffa3d7fb9f73dffdfe777fffbffd8b57c5b1bdbbfbbbebffbfbddbc7f5ffffff291f5f771dbfeadeaff6f6fffea3e9bf7ddfbefaf77af3fefdbffa5bfbfdfffb3b6cb3fe5cab3f5b769df7ef3fde9fbfb4f77fffbbf7fb7ffdffef3db7fffafff6efbf97cfffdefffbfd6bca6c65ffeac7bdddebdff7fdbbf7f757f6e7f4fdff9e7f77efe'), 'difficulty': 0, 'number': 17919807, 'gasLimit': 30000000, 'gasUsed': 29630611, 'timestamp': 1692097223, 'extraData': HexBytes('0x6265617665726275696c642e6f7267'), 'mixHash': HexBytes('0x991dff3d107bb7e837ec521ab30148d692a8b0d885dcddf577ad76bbb770b4f6'), 'nonce': HexBytes('0x0000000000000000'), 'baseFeePerGas': 16293864808, 'withdrawalsRoot': HexBytes('0x8a08ca6b0cf9e76d1fef5af3e25cccea880248b18d4f26d0e57253cb665289e9'), 'hash': HexBytes('0xb0ab75517ebe2c0afa6768d6ae198d80742d1a5ae219c45e7a4a328bec9dff86')})
AttributeDict({'parentHash': HexBytes('0xb0ab75517ebe2c0afa6768d6ae198d80742d1a5ae219c45e7a4a328bec9dff86'), 'sha3Uncles': HexBytes('0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347'), 'miner': '0x388C818CA8B9251b393131C08a736A67ccB19297', 'stateRoot': HexBytes('0xfce576917051c44a788796466fa99061d2782fff6eae9eb62679b2c0009e7130'), 'transactionsRoot': HexBytes('0xc9faebc0f3c6488354496f1b4530b8a1cd09a579daf8c16e3dc137152bcb313d'), 'receiptsRoot': HexBytes('0xc4fe0def95af8e416eff5d0f7fb28aeee33b142f1c80bf273e2577d9f5e2268b'), 'logsBloom': HexBytes('0x36e3510f5f6d171db8148922a68d73a35931c6f9511590150b0d20047fab1b2011e723a292c12be124107f2092154d344f0321719a647995ba97d5971c6c8631cbb27608a478daa8fa024a8bd9552ab80424209a9d5c1a930438ff4d8f229240aa42cef8e706830d2aee10a042aa98d1803019f1231e2c5296c68e116a8c932376304bf7bef9402161f921e60a96165b9034b3198b7080ca492888f7b9b147b882e001cbd9ed377a0a8650e9d9531e430f702b5540b234ffb813d73f948a315dd58b06820306a7696eb411020f1b07c72fcd55cae5a4d8f92745751b686fe6c49f30e84897ee44ee4e1e6a9aaa01721895ba92733b70b85e45a2a0410605f443'), 'difficulty': 0, 'number': 17919808, 'gasLimit': 29970705, 'gasUsed': 18911237, 'timestamp': 1692097235, 'extraData': HexBytes('0x546974616e2028746974616e6275696c6465722e78797a29'), 'mixHash': HexBytes('0x54fb419f02fae08b75e2de7523be2a4ed4c77f331ccc9f05dbb80ff8cf0ecbc2'), 'nonce': HexBytes('0x0000000000000000'), 'baseFeePerGas': 18280441455, 'withdrawalsRoot': HexBytes('0x4a56a65facff04f3f413de2a02807d28924d9c2d613538b29c4ff7d10dc5c620'), 'hash': HexBytes('0xd4f27c784a0e49cfd2b1c2096dfdfae186de4b95682cb658140997214f1c682e')})
unsubscribing...
unsubscribed: True

Ankr:

Subscription full: {'jsonrpc': '2.0', 'id': None, 'result': '0x5bdd3f0abf32f1d567de78c65b212416'}
{'jsonrpc': '2.0', 'method': 'eth_subscription', 'params': {'subscription': '0x5bdd3f0abf32f1d567de78c65b212416', 'result': {'parentHash': '0x44036c250dbb589dc238ab8a2b86d49bdefaf4ae0bd9ca440e3279e26e933cc6', 'sha3Uncles': '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347', 'miner': '0xdafea492d9c6733ae3d56b7ed1adb60692c98bc5', 'stateRoot': '0x887aae094bf81d890618278dceb0e138e3e76f761c39954275cb819fae288e3b', 'transactionsRoot': '0x0402ec01eba4484c589591d52bb515c814274abe9ff6eafd2b8b3970fa938ef5', 'receiptsRoot': '0xf3496c673c0805f3ac93eb4171081dec06f5019eafdf29a9c9bc6891095601cc', 'logsBloom': '0x622911c879a9baa9d822352a972435b03dbf46704a49030c1681cd526c33084eb485999dcd006269a2d00722c8d08f759679b8199e036ba94d10d34518690b2c0ca66e7948502baa7a1a4a9fcaf2f866e9b1293dbbd40aa0082e125eeea2ac68ee0ce4e8135a8ea7851c593070e84d47a8f9a0102186a6b487d38f185a0a67a66d7a84f40b020d27e274642c9f47817f21441ff1cfd4cc5e67b0036b4a98070c8a082d48f3f864702a1048cdd3136648ed2c2e22999bd4f2828986b6321a18419d5891a61390124a6d00bc064732b3853458a8434d64323c6eefb70282ada9245d10a19df75608a250c612f978d18448b3ac9b96dbd863de3f23e22b209d1c21', 'difficulty': '0x0', 'number': '0x1116f46', 'gasLimit': '0x1c9c380', 'gasUsed': '0xf3861c', 'timestamp': '0x64db5b1b', 'extraData': '0x496c6c756d696e61746520446d6f63726174697a6520447374726962757465', 'mixHash': '0x9578af2df0f6717d1254f63b0dcbacdf40eb046a3bd2ccec79e35dc2a6d3f43e', 'nonce': '0x0000000000000000', 'baseFeePerGas': '0x484f4e1b6', 'withdrawalsRoot': '0x451cacd2f6c5eb2e2f93607eff17076f35e9178fb5a9eb1d3d841ff89feb1db3', 'excessDataGas': None, 'hash': '0xb067f962c62d8ae70914d1cf2d676c8946bda5463623f804af3c471ee8ed1dc2'}}}
{'jsonrpc': '2.0', 'method': 'eth_subscription', 'params': {'subscription': '0x5bdd3f0abf32f1d567de78c65b212416', 'result': {'parentHash': '0xb067f962c62d8ae70914d1cf2d676c8946bda5463623f804af3c471ee8ed1dc2', 'sha3Uncles': '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347', 'miner': '0x1f9090aae28b8a3dceadf281b0f12828e676c326', 'stateRoot': '0x455a1305a90383b481dfbeac9e7d5695afed2cf147bb66acde52784ae6833319', 'transactionsRoot': '0x80e2082a74d9cde40fcd2f16bfeca755b053ba650e1959e01de5c34b699016a0', 'receiptsRoot': '0x68d8cd1f543c6cf2579ee944d102c76522f9209b65669568d9f150eb2dabeba5', 'logsBloom': '0x743349ab69a0c28f3f1416248ca4aae64021433348696800ae6b0666dc034149fb3173ed04300e6540101f02cc34053c2fa930019e8e3c225b37abc0326e4a498031720c680e092de9de628b5a6d1efea983862a5de48890ae35444c9e699e40aa9d44e29ac2aa0904d55c240a48ecd311483848251b0cc4f34a3c758828c736dda1b7dcaaf6d5e1c24d416645aea62c78fc57e1d73688cf60ada847e1b00a3487d301833ba8e4520bfa25e999978e0a9d440ad898afb5c416614f07748c3c2101d860f227a313016240c1035d0ad5f402cc3483e92515fe2e223b26de2b7d244dfbef029300c5d80bf6d4e9f104f9a9c5a058432b3381640085684305c8d521', 'difficulty': '0x0', 'number': '0x1116f47', 'gasLimit': '0x1c9c380', 'gasUsed': '0xf3248b', 'timestamp': '0x64db5b27', 'extraData': '0x7273796e632d6275696c6465722e78797a', 'mixHash': '0x89ad2580625c1f3d78538cbe60eb0ca2385c22dbc58347f4360a9a8f4f2a9cfb', 'nonce': '0x0000000000000000', 'baseFeePerGas': '0x48e354ac7', 'withdrawalsRoot': '0x1504d67dfb5a9e689fd1f5f67739142166e39b9627c71f51e0d8a1425ca457d3', 'excessDataGas': None, 'hash': '0x1e3d4aea21edf97f1e8869bad232e918160d941e57d71f10cd2bd6f0ba96f0c7'}}}
{'jsonrpc': '2.0', 'method': 'eth_subscription', 'params': {'subscription': '0x5bdd3f0abf32f1d567de78c65b212416', 'result': {'parentHash': '0x1e3d4aea21edf97f1e8869bad232e918160d941e57d71f10cd2bd6f0ba96f0c7', 'sha3Uncles': '0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347', 'miner': '0x1f9090aae28b8a3dceadf281b0f12828e676c326', 'stateRoot': '0x7e40e410b61f076d7395df8c26f26423a41b40d91bf819197220a92d164878c5', 'transactionsRoot': '0xc49d2244b9b74008f2a8456366aee8463e41c74797ef08f4ff8f71094421634b', 'receiptsRoot': '0x437061019f20d077b3716836cbd27849e5cfd1e864b5e9e20bac118906d0efc4', 'logsBloom': '0xce6193462d418244587d0236a0fcbd8019800f350a248810466d294002a59561244ebba55c02119063c69a928199097c1ac5c0379e3a28d1845eca4e943da25e44624018c1cbdd2acb4e5aeff86926be0023bc0691f5ce02ac194c4fea7cc7195a444528db27e60923801001131038dd6ab8107801be04522637e47eaa1a02a64ce0b47214460921414505e27fb6a48b7964b58dfb0b52fe0c251a60d8d323209211d950952b30fb38b554f3c8728e4216489888eae96c740490cb3f72354021c29020864a10d2c82004c5031c0fc7c5aad03a226672011e69a209aa4823ec0fea74619c0f4aeb8840c4c4bdf4124135a5fe52181b10654607437049a147bc4b', 'difficulty': '0x0', 'number': '0x1116f48', 'gasLimit': '0x1c9c380', 'gasUsed': '0xf98511', 'timestamp': '0x64db5b33', 'extraData': '0x7273796e632d6275696c6465722e78797a', 'mixHash': '0xc5606b85769914575c73d38f88d2b759a14ccac0ca1d5badd6e91c4a7d8e11c2', 'nonce': '0x0000000000000000', 'baseFeePerGas': '0x4974a803f', 'withdrawalsRoot': '0x48f5cda9bf6ff5ba200cf2383e2338d9e6311dcc3cf9e9e592aa8f87c1e92bb1', 'excessDataGas': None, 'hash': '0x8780b3ea5346d70b12f4dd673153812636647c01a147ff1005621b5f497e4ede'}}}
unsubscribing...
unsubscribed: True

Seems to be Ankr node (don't know what they are using geth/erigon/besu) and Erigon have different API.

@fselmo
Copy link
Collaborator

fselmo commented Aug 15, 2023

@fselmo I see the issue already. As per the JSON-RPC 2.0 spec, each request has an id and its response should contain the same id so a user understands what request the response is for. web3.py sends an id with each request but the id for the Ankr node is None. This is quite important for a websocket provider so one can differentiate between responses. This is how the WebsocketProviderV2 knows how to format a response. So, in your case, you are getting the raw response back because there's no way to tell what request this response is for.

Perhaps there is an ankr node that follows the spec that you might be able to use. I can add some details about this to the documentation but this doesn't seem to be an issue with the library at this point.

@dmkulazhenko
Copy link
Contributor Author

dmkulazhenko commented Aug 15, 2023

@fselmo

no problems, thanks for explanation, I will resolve it on my side :)

p.s. I have nothing to add, you can close issue, when #3077 will be merged. Have a nice day <3

@fselmo
Copy link
Collaborator

fselmo commented Aug 15, 2023

Sounds good. Thanks again for reporting 👍🏼

@fselmo fselmo closed this as completed Aug 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants