stack_handler crashes due to asyncio.exceptions.TimeoutError

[j1404]

Today (Tue 27 Feb) stack_handler crashes with the following error. This is pretty weird, since I haven't changed or updated anything and last evening everything was just fine. I'm running pst 1.71, ib-insync 0.9.86 and IB Gateway 10.24, this setup has been running fine a few months.

I can see from the IB Gateway log that client connect six times and each time lots of data is pulled from IB but for some reason the client doesn't agree. What might have caused this issue? Can it really be a timeout issue and if yes, where do I change the timeout?

sysproduction.run_stack_handler.run_stack_handler:
 None 
 Arguments: []
2024-02-27 17:48:47 DEBUG config {'type': 'config', 'stage': 'config'} Adding config defaults
2024-02-27 17:48:47 DEBUG config {'type': 'config', 'stage': 'config'} Adding config defaults
2024-02-27 17:48:47 DEBUG stack_handler check_external_position_break will run every 0 minutes until process ends with heartbeats every 10 minutes
2024-02-27 17:48:47 DEBUG config {'type': 'config', 'stage': 'config'} Adding config defaults
2024-02-27 17:48:47 DEBUG stack_handler spawn_children_from_new_instrument_orders will run every 0 minutes until process ends with heartbeats every 10 minutes
2024-02-27 17:48:47 DEBUG config {'type': 'config', 'stage': 'config'} Adding config defaults
2024-02-27 17:48:47 DEBUG stack_handler generate_force_roll_orders will run every 0 minutes at most 1 times with heartbeats every 10 minutes
2024-02-27 17:48:47 DEBUG config {'type': 'config', 'stage': 'config'} Adding config defaults
2024-02-27 17:48:47 DEBUG stack_handler create_broker_orders_from_contract_orders will run every 0 minutes until process ends with heartbeats every 10 minutes
2024-02-27 17:48:47 DEBUG config {'type': 'config', 'stage': 'config'} Adding config defaults
2024-02-27 17:48:47 DEBUG stack_handler process_fills_stack will run every 0 minutes until process ends with heartbeats every 10 minutes
2024-02-27 17:48:47 DEBUG config {'type': 'config', 'stage': 'config'} Adding config defaults
2024-02-27 17:48:47 DEBUG stack_handler handle_completed_orders will run every 0 minutes until process ends with heartbeats every 10 minutes
2024-02-27 17:48:47 DEBUG config {'type': 'config', 'stage': 'config'} Adding config defaults
2024-02-27 17:48:47 DEBUG stack_handler safe_stack_removal will run once only on process completion
2024-02-27 17:48:47 DEBUG config {'type': 'config', 'stage': 'config'} Adding config defaults
2024-02-27 17:48:47 DEBUG stack_handler refresh_additional_sampling_all_instruments will run every 60 minutes until process ends with heartbeats every 10 minutes
2024-02-27 17:48:47 DEBUG config {'type': 'config', 'stage': 'config'} Adding config defaults
2024-02-27 17:48:48 DEBUG stack_handler {'type': 'run_stack_handler'} check_external_position_break still alive, done 0 of unlimited executions every 0 minutes
2024-02-27 17:48:48 DEBUG stack_handler {'component': 'mongoIbBrokerClientIdData'} Locked IB client ID 1
2024-02-27 17:48:52 DEBUG stack_handler {'component': 'mongoIbBrokerClientIdData'} Locked IB client ID 2
2024-02-27 17:48:57 DEBUG stack_handler {'component': 'mongoIbBrokerClientIdData'} Locked IB client ID 3
2024-02-27 17:49:02 DEBUG stack_handler {'component': 'mongoIbBrokerClientIdData'} Locked IB client ID 4
2024-02-27 17:49:07 DEBUG stack_handler {'component': 'mongoIbBrokerClientIdData'} Locked IB client ID 5
2024-02-27 17:49:11 DEBUG stack_handler {'component': 'mongoIbBrokerClientIdData'} Locked IB client ID 6
2024-02-27 17:49:16 DEBUG stack_handler {'component': 'mongoIbBrokerClientIdData'} Locked IB client ID 15
2024-02-27 17:49:16 DEBUG stack_handler {'component': 'mongoIbBrokerClientIdData'} Released IB client ID 1
2024-02-27 17:49:16 DEBUG stack_handler {'component': 'mongoIbBrokerClientIdData'} Released IB client ID 2
2024-02-27 17:49:16 DEBUG stack_handler {'component': 'mongoIbBrokerClientIdData'} Released IB client ID 3
2024-02-27 17:49:16 DEBUG stack_handler {'component': 'mongoIbBrokerClientIdData'} Released IB client ID 4
2024-02-27 17:49:16 DEBUG stack_handler {'component': 'mongoIbBrokerClientIdData'} Released IB client ID 5
2024-02-27 17:49:16 DEBUG stack_handler {'component': 'mongoIbBrokerClientIdData'} Released IB client ID 6
2024-02-27 17:49:16 CRITICAL stack_handler {'type': 'run_stack_handler'} Error  couldn't evaluate ibFxPricesData(self.ib_conn, self, log = self.log.setup(component = ibFxPricesData)) This might be because (a) IB gateway not running, or (b) import is missing                         or (c) arguments don't follow pattern
/home/alpha/pysystemtrade/venv/3.8.17/lib/python3.8/site-packages/arctic/_util.py:6: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  from pandas.util.testing import assert_frame_equal
/home/alpha/pysystemtrade/venv/3.8.17/lib/python3.8/site-packages/arctic/store/_pandas_ndarray_store.py:6: FutureWarning: The Panel class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
  from pandas import DataFrame, Series, Panel
Traceback (most recent call last):
  File "/home/alpha/pysystemtrade/sysdata/data_blob.py", line 125, in _add_ib_class
    resolved_instance = class_object(self.ib_conn, self, log=log)
  File "/home/alpha/pysystemtrade/sysdata/data_blob.py", line 270, in ib_conn
    ib_conn = self._get_new_ib_connection()
  File "/home/alpha/pysystemtrade/sysdata/data_blob.py", line 293, in _get_new_ib_connection
    raise e
  File "/home/alpha/pysystemtrade/sysdata/data_blob.py", line 282, in _get_new_ib_connection
    ib_conn = connectionIB(client_id, log=self.log)
  File "/home/alpha/pysystemtrade/sysbrokers/IB/ib_connection.py", line 57, in __init__
    self._init_connection(
  File "/home/alpha/pysystemtrade/sysbrokers/IB/ib_connection.py", line 82, in _init_connection
    ib.connect(ipaddress, port, clientId=client_id, account=account)
  File "/home/alpha/pysystemtrade/venv/3.8.17/lib/python3.8/site-packages/ib_insync/ib.py", line 279, in connect
    return self._run(self.connectAsync(
  File "/home/alpha/pysystemtrade/venv/3.8.17/lib/python3.8/site-packages/ib_insync/ib.py", line 318, in _run
    return util.run(*awaitables, timeout=self.RequestTimeout)
  File "/home/alpha/pysystemtrade/venv/3.8.17/lib/python3.8/site-packages/ib_insync/util.py", line 341, in run
    result = loop.run_until_complete(task)
  File "/home/alpha/.pyenv/versions/3.8.17/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/alpha/pysystemtrade/venv/3.8.17/lib/python3.8/site-packages/ib_insync/ib.py", line 1782, in connectAsync
    await asyncio.wait_for(self.reqExecutionsAsync(), timeout)
  File "/home/alpha/.pyenv/versions/3.8.17/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/alpha/pysystemtrade/sysproduction/linux/scripts/run.py", line 67, in <module>
    func(*args, **kwargs)
  File "/home/alpha/pysystemtrade/sysproduction/run_stack_handler.py", line 11, in run_stack_handler
    price_process.run_process()
  File "/home/alpha/pysystemtrade/syscontrol/run_process.py", line 106, in run_process
    self._main_loop_over_methods()
  File "/home/alpha/pysystemtrade/syscontrol/run_process.py", line 132, in _main_loop_over_methods
    timer_class.check_and_run(**kwargs)
  File "/home/alpha/pysystemtrade/syscontrol/timer_functions.py", line 124, in check_and_run
    self.run_function(**kwargs)
  File "/home/alpha/pysystemtrade/syscontrol/timer_functions.py", line 299, in run_function
    self._function(**kwargs)
  File "/home/alpha/pysystemtrade/sysexecution/stack_handler/checks.py", line 31, in check_external_position_break
    data_broker = dataBroker(self.data)
  File "/home/alpha/pysystemtrade/sysproduction/data/generic_production_data.py", line 11, in __init__
    data = self._add_required_classes_to_data(data)
  File "/home/alpha/pysystemtrade/sysproduction/data/broker.py", line 54, in _add_required_classes_to_data
    data.add_class_list(broker_class_list)
  File "/home/alpha/pysystemtrade/sysdata/data_blob.py", line 82, in add_class_list
    self.add_class_object(class_object)
  File "/home/alpha/pysystemtrade/sysdata/data_blob.py", line 88, in add_class_object
    resolved_instance = self._get_resolved_instance_of_class(class_object)
  File "/home/alpha/pysystemtrade/sysdata/data_blob.py", line 93, in _get_resolved_instance_of_class
    resolved_instance = class_adding_method(class_object)
  File "/home/alpha/pysystemtrade/sysdata/data_blob.py", line 133, in _add_ib_class
    self._raise_and_log_error(msg)
  File "/home/alpha/pysystemtrade/sysdata/data_blob.py", line 326, in _raise_and_log_error
    raise Exception(error_msg)
Exception: Error  couldn't evaluate ibFxPricesData(self.ib_conn, self, log = self.log.setup(component = ibFxPricesData)) This might be because (a) IB gateway not running, or (b) import is missing                         or (c) arguments don't follow pattern

Same thing

$ python
Python 3.8.17 (default, Jul 31 2023, 16:56:30) 
[GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from sysbrokers.IB.ib_connection import connectionIB                                   
Configuring sim logging
>>> conn = connectionIB( 999, ib_ipaddress = "127.0.0.1", ib_port=4001, account="xxx")
2024-02-27 18:06:56 DEBUG config {'type': 'config', 'stage': 'config'} Adding config defaults

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/alpha/pysystemtrade/sysbrokers/IB/ib_connection.py", line 57, in __init__
    self._init_connection(
  File "/home/alpha/pysystemtrade/sysbrokers/IB/ib_connection.py", line 82, in _init_connection
    ib.connect(ipaddress, port, clientId=client_id, account=account)
  File "/home/alpha/pysystemtrade/venv/3.8.17/lib/python3.8/site-packages/ib_insync/ib.py", line 279, in connect
    return self._run(self.connectAsync(
  File "/home/alpha/pysystemtrade/venv/3.8.17/lib/python3.8/site-packages/ib_insync/ib.py", line 318, in _run
    return util.run(*awaitables, timeout=self.RequestTimeout)
  File "/home/alpha/pysystemtrade/venv/3.8.17/lib/python3.8/site-packages/ib_insync/util.py", line 341, in run
    result = loop.run_until_complete(task)
  File "/home/alpha/.pyenv/versions/3.8.17/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/alpha/pysystemtrade/venv/3.8.17/lib/python3.8/site-packages/ib_insync/ib.py", line 1782, in connectAsync
    await asyncio.wait_for(self.reqExecutionsAsync(), timeout)
  File "/home/alpha/.pyenv/versions/3.8.17/lib/python3.8/asyncio/tasks.py", line 501, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError
>>> 

[robcarver17]

It's a timeout, nothing to do with pysystemtrade, so you should be asking eithier on an insync forum or a generic TWS forum

Create a minimum example using just ibsync first.

[gdoscher]

Did you ever resolve this? Weirdly I just began having the same issue out of the blue. I can see IB accepting the incoming connection on the proper API port and client ID, yet it just continually disconnects from that and retries new client IDs (all of which are accepted in TWS) until it claims a timeout. I even updated the time from 0 seconds, to 5 and then 10 but to no avail. Not much info on the IB-insync or TWS forums so I thought I'd check if you had figured this out since my issue is identical.

[j1404]

Couldn't figure out why it happens but found a workaround: temporarily pointed pst to IB Gateway on Windows. IB API log tells that lots of data is downloaded by the client but at some point the connection just breaks. So my solution was to either use Windows temporarily or wait for the next day.

I've seen this issue a few times. It happened always either Monday or Tuesday afternoon. I would argue that it must have something to do with the Linux-based Gateway, because when the timeout occurs, Windows-based Gateway works just fine.