how can I set the enlarge the pool_connections and pool_maxsize for urllib3 in web3?

[ayeteng]

• Version: 5.2.0
• Python: 3.6
• OS: osx

What was wrong?

If I submit the transaction in multi-threading environment, I got the error message as below:

WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: localhost

I think it's because of the connection not released until I get the transaction receipt. So, is it possible to set larger the pool_connections and pool_maxsize for urllib3 in web3 to avoid such error?

def send_tx(unit):
    while True:
            try:
                tx_hash = self.contract_instance[recorder.sol'].functions.add_value(
                    str(unit['sys_id']).encode('utf-8'),
                    ).transact({'from': self.w3.toChecksumAddress(unit['iot_working_update_account'])})
                return tx_hash
            except Exception as e:
                print("Cannot connect to geth and tx fail. Reason: %s, Retry: %d" % (e, count))

def get_receipt(tx_hash):
    while True:
            try:
                ret_value = self.w3.eth.waitForTransactionReceipt(tx_hash, timeout=600)
                return ret_value
            except Exception as e:
                print("Cannot get the transaction receipt. Reason: %s, Retry: %d" % (e, count))

tx_hash = send_tx(unit)
receipt = get_receipt(tx_hash)

[kclowes]

Sorry @ayeteng this was raised right at the beginning of Devcon and I missed it. Are you using the HTTPProvider? I'm not aware of a way to increase pool_connections or pool_maxsize in web3. I believe the urllib3 error is actually being bubbled up from the requests library and it looks like they don't do any requests that are non-blocking. @pipermerriam do you have any ideas here?

[ayeteng]

Thanks @kclowes ,
About this issue, because the DEFAULT_POOLSIZE is set to 10 in requests package, I'd like to check if possible to set the pool connection to 100 with following trick in web3 request connection.

import requests
sess = requests.Session()
adapter = requests.adapters.HTTPAdapter(pool_connections=100, pool_maxsize=100)
sess.mount('http://', adapter)
resp = sess.get("/mypage")

[EdNoepel]

Web3 seems to assume that kwargs passed to the request method (likely just timeout) are the ones which should be hashed in the cache_key. I'd argue the following points:

• HttpProvider should accept arguments passed to the HttpAdapter constructor.
• generate_cache_key should hash both the request arguments and those used for creation of the HttpAdapter.

I'm meditating on a clean way to accomplish this without drastically redesigning request.py.
Suggestions would be appreciated.

[ayeteng]

@EdNoepel ,
Given I don't know how to pass the kwargs to request via web3, I did the workaround to modify the request default setting by myself. It works good recently so I don't work on this issue for the time been.
Thanks for your comment.

[ayeteng]

I found this was closed in 55297bc