TypeError: Cannot convert None of type <class 'NoneType'> to bytes (in formatting middleware?)

[nullp0tr]

• Version: 5.24.0
• Python: 3.8
• OS: linux
• pip freeze output

aiohttp==3.8.1
aiosignal==1.2.0
astroid==2.11.5
async-timeout==4.0.2
attrs==21.4.0
base58==2.1.1
bitarray==1.2.2
black==21.12b0
blinker==1.4
boto3==1.23.0
boto3-stubs==1.23.0
botocore==1.26.0
botocore-stubs==1.26.0.post1
CacheControl==0.12.11
cachy==0.3.0
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.12
cleo==0.8.1
click==8.0.4
clickclick==20.10.2
clikit==0.6.2
common==0.1.0
connexion==2.13.1
crashtest==0.3.1
cryptography==37.0.2
cytoolz==0.11.2
dill==0.3.4
distlib==0.3.4
ecdsa==0.17.0
eth-abi==2.1.1
eth-account==0.5.7
eth-hash==0.3.2
eth-keyfile==0.5.1
eth-keys==0.3.4
eth-rlp==0.2.1
eth-typing==2.3.0
eth-utils==1.10.0
filelock==3.7.0
flasgger==0.9.5
Flask==2.1.2
Flask-Pydantic==0.9.0
frozenlist==1.3.0
hexbytes==0.2.2
html5lib==1.1
idna==3.3
importlib-metadata==4.11.3
importlib-resources==5.6.0
inflection==0.5.1
iniconfig==1.1.1
ipfshttpclient==0.8.0a2
isort==5.10.1
itsdangerous==2.1.2
jeepney==0.8.0
Jinja2==3.1.2
jmespath==1.0.0
jsonschema==3.2.0
keyring==23.5.0
lazy-object-proxy==1.7.1
lockfile==0.12.2
lru-dict==1.1.7
MarkupSafe==2.1.1
mccabe==0.7.0
mistune==2.0.2
msgpack==1.0.3
multiaddr==0.0.9
multidict==6.0.2
mypy==0.910
mypy-extensions==0.4.3
netaddr==0.8.0
openapi-schema-validator==0.2.3
openapi-spec-validator==0.3.1
packaging==20.9
parsimonious==0.8.1
pastel==0.2.1
pathspec==0.9.0
pendulum==2.1.2
pexpect==4.8.0
pip==22.0.4
pkginfo==1.8.2
platformdirs==2.5.2
pluggy==1.0.0
poetry==1.1.13
poetry-core==1.0.8
protobuf==3.20.1
ptyprocess==0.7.0
py==1.11.0
pyasn1==0.4.8
pycparser==2.21
pycryptodome==3.14.1
pydantic==1.9.0
pylev==1.4.0
pylint==2.13.9
pyparsing==3.0.9
pyrsistent==0.16.1
pytest==7.1.2
python-dateutil==2.8.2
python-dotenv==0.19.2
python-jose==3.3.0
pytzdata==2020.1
PyYAML==5.4.1
requests==2.27.1
requests-toolbelt==0.9.1
rlp==2.0.1
rsa==4.8
s3transfer==0.5.2
SecretStorage==3.3.2
sentry-sdk==1.5.12
setuptools==62.1.0
shellingham==1.4.0
six==1.16.0
swagger-ui-bundle==0.0.9
toml==0.10.2
tomli==1.2.3
tomlkit==0.10.2
toolz==0.11.2
types-boto3==1.0.2
types-requests==2.27.25
types-urllib3==1.26.14
typing_extensions==4.2.0
urllib3==1.26.9
varint==1.0.2
virtualenv==20.14.1
web3==5.24.0
webencodings==0.5.1
websockets==9.1
Werkzeug==2.1.2
wheel==0.37.1
wrapt==1.14.1
yarl==1.7.2
zipp==3.8.0

What was wrong?

on a company project we're getting the following error once every few moons

Traceback (most recent call last):
  .... (company code that calls contract.functions.functionName().call())
  File "/usr/local/lib/python3.8/site-packages/web3/contract.py", line 959, in call
    return call_contract_function(
  File "/usr/local/lib/python3.8/site-packages/web3/contract.py", line 1503, in call_contract_function
    return_data = web3.eth.call(
  File "/usr/local/lib/python3.8/site-packages/web3/module.py", line 57, in caller
    result = w3.manager.request_blocking(method_str,
  File "/usr/local/lib/python3.8/site-packages/web3/manager.py", line 186, in request_blocking
    response = self._make_request(method, params)
  File "/usr/local/lib/python3.8/site-packages/web3/manager.py", line 147, in _make_request
    return request_func(method, params)
  File "cytoolz/functoolz.pyx", line 254, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "/usr/local/lib/python3.8/site-packages/web3/middleware/formatting.py", line 76, in apply_formatters
    response = make_request(method, params)
  File "/usr/local/lib/python3.8/site-packages/web3/middleware/gas_price_strategy.py", line 90, in middleware
    return make_request(method, params)
  File "cytoolz/functoolz.pyx", line 254, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "/usr/local/lib/python3.8/site-packages/web3/middleware/formatting.py", line 74, in apply_formatters
    response = make_request(method, formatted_params)
  File "/usr/local/lib/python3.8/site-packages/web3/middleware/attrdict.py", line 33, in middleware
    response = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 254, in cytoolz.functoolz.curry.__call__
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "/usr/local/lib/python3.8/site-packages/web3/middleware/formatting.py", line 83, in apply_formatters
    formatter(response["result"]),
  File "/usr/local/lib/python3.8/site-packages/hexbytes/main.py", line 23, in __new__
    bytesval = to_bytes(val)
  File "/usr/local/lib/python3.8/site-packages/hexbytes/_utils.py", line 30, in to_bytes
    raise TypeError(f"Cannot convert {val!r} of type {type(val)} to bytes")
TypeError: Cannot convert None of type <class 'NoneType'> to bytes

unfortunately we haven't managed to find code to reproduce this issue consistently, it just happens sometimes on a pretty standard call that we issue quite often and that almost always succeeds. any pointers would be appreciated !

[fselmo]

Hey @nullp0tr, it looks like the call response has a "result" in the json but the result is null / None. web3.py has some middleware that make the output more readable / user friendly and specifically the formatting middleware is attempting to turn this None into HexBytes.

I would review a PR that looks for response["result"] having a value. I think this can be addressed in a global sense by changing this line to:

if "result" in response and response["result"] is not None and method in result_formatters

... with a similar change applied to the error formatters lines directly below it and ideally adds some appropriate tests around the None results.

I do think this would more broadly only attempt formatting a response or error if the value was present. In your case, since this seems to be only when there is a None or null value, you can also try / except this exception and move on when it's None if that helps you get unstuck for now.

If you or someone else doesn't get to it soon enough I think one of us can come by and add this PR soon. I'll leave this open for now so we can track it.

[nullp0tr]

Hi @fselmo thanks a lot for the feedback, I'll see if I can write a test case for this to facilitate a PR 🙏

[fselmo]

@nullp0tr Any time! Let us know if you need any help. I'd be happy to collaborate on adding some tests too.