run_daily_price_updates crashes due to socket disconnection

[emretezel]

Hi Everyone,

All of a sudden last week, almost every day the run_daily_price_updates process crashed in the evening when it was time to load US and EU prices. This process was running smooth for many months.

I know of IB Server daily reset times, and I make sure the process starts after the US server restart time, 1am Eastern Time.

Could this be due to anything other than genuine socket connection issue.

I am using latest master code and IB Gateway 10.25.

 2023-11-03 20:00:14 DEBUG update_historical_prices {'instrument_code': 'NIKKEI', 'contract_date': '20240300'} Now it's time to download region EMEA: ['BOBL', 'BONO', 'BTP
', 'BUND', 'DAX', 'EUROSTX', 'OAT', 'SMI']
2023-11-03 20:00:14 DEBUG update_historical_prices {'component': 'ibFuturesContractData'} Reqid -1: 2107 HMDS data farm connection is inactive but should be available upo
n demand.hkhmds for None
2023-11-03 20:00:14 DEBUG update_historical_prices {'component': 'ibFuturesContractPriceData'} Reqid -1: 2107 HMDS data farm connection is inactive but should be availabl
e upon demand.hkhmds for None
2023-11-03 20:00:14 ERROR ib_insync.client [Errno 104] Connection reset by peer
/home/pysystemtrade/anaconda3/envs/pysystemtrade-dev/lib/python3.8/site-packages/arctic/_util.py:6: FutureWarning: pandas.util.testing is deprecated. Use the functions in
 the public API at pandas.testing instead.
  from pandas.util.testing import assert_frame_equal
/home/pysystemtrade/anaconda3/envs/pysystemtrade-dev/lib/python3.8/site-packages/arctic/store/_pandas_ndarray_store.py:6: FutureWarning: The Panel class is removed from p
andas. Accessing it from the top-level namespace will also be removed in the next version
  from pandas import DataFrame, Series, Panel
/home/pysystemtrade/opt/pysystemtrade/sysobjects/futures_per_contract_prices.py:35: UserWarning: Pandas doesn't allow columns to be created via a new attribute name - see
 https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute-access
  self._as_df = price_data_as_df
Traceback (most recent call last):
  File "/home/pysystemtrade/opt/pysystemtrade/syscore/cache.py", line 48, in _get_from_store
    value = self.store[key]
KeyError: "_get_ib_futures_contract_from_broker/(BOBL/20231200,)/{'trade_list_for_multiple_legs': None, 'allow_expired': False}"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pysystemtrade/opt/pysystemtrade/syscore/cache.py", line 29, in get
    value_from_store = self._get_from_store(key)
  File "/home/pysystemtrade/opt/pysystemtrade/syscore/cache.py", line 50, in _get_from_store
    raise missingData("Missing cache element %s" % key) from e
syscore.exceptions.missingData: Missing cache element _get_ib_futures_contract_from_broker/(BOBL/20231200,)/{'trade_list_for_multiple_legs': None, 'allow_expired': False}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pysystemtrade/opt/pysystemtrade/sysproduction/linux/scripts/run.py", line 67, in <module>
    func(*args, **kwargs)
  File "/home/pysystemtrade/opt/pysystemtrade/sysproduction/run_daily_price_updates.py", line 12, in run_daily_price_updates
    price_process.run_process()
  File "/home/pysystemtrade/opt/pysystemtrade/syscontrol/run_process.py", line 106, in run_process
    self._main_loop_over_methods()
  File "/home/pysystemtrade/opt/pysystemtrade/syscontrol/run_process.py", line 132, in _main_loop_over_methods
    timer_class.check_and_run(**kwargs)
  File "/home/pysystemtrade/opt/pysystemtrade/syscontrol/timer_functions.py", line 124, in check_and_run
    self.run_function(**kwargs)
  File "/home/pysystemtrade/opt/pysystemtrade/syscontrol/timer_functions.py", line 299, in run_function
    self._function(**kwargs)
  File "/home/pysystemtrade/opt/pysystemtrade/sysproduction/update_historical_prices.py", line 52, in update_historical_prices
    update_historical_prices_with_data(data, download_by_zone=download_by_zone)
  File "/home/pysystemtrade/opt/pysystemtrade/sysproduction/update_historical_prices.py", line 61, in update_historical_prices_with_data
    manage_download_over_multiple_time_zones(
  File "/home/pysystemtrade/opt/pysystemtrade/sysproduction/update_historical_prices.py", line 91, in manage_download_over_multiple_time_zones
    manage_download_given_dict_of_instrument_codes(
  File "/home/pysystemtrade/opt/pysystemtrade/sysproduction/update_historical_prices.py", line 177, in manage_download_given_dict_of_instrument_codes
    update_historical_prices_for_list_of_instrument_codes(
  File "/home/pysystemtrade/opt/pysystemtrade/sysproduction/update_historical_prices.py", line 286, in update_historical_prices_for_list_of_instrument_codes
    update_historical_prices_for_instrument(
  File "/home/pysystemtrade/opt/pysystemtrade/sysproduction/update_historical_prices.py", line 320, in update_historical_prices_for_instrument
    update_historical_prices_for_instrument_and_contract(
  File "/home/pysystemtrade/opt/pysystemtrade/sysproduction/update_historical_prices.py", line 344, in update_historical_prices_for_instrument_and_contract
    get_and_add_prices_for_frequency(
  File "/home/pysystemtrade/opt/pysystemtrade/sysproduction/update_historical_prices.py", line 370, in get_and_add_prices_for_frequency
    broker_data_source.get_cleaned_prices_at_frequency_for_contract_object(
  File "/home/pysystemtrade/opt/pysystemtrade/sysproduction/data/broker.py", line 133, in get_cleaned_prices_at_frequency_for_contract_object
    broker_prices_raw = self.get_prices_at_frequency_for_contract_object(
  File "/home/pysystemtrade/opt/pysystemtrade/sysproduction/data/broker.py", line 159, in get_prices_at_frequency_for_contract_object
    return self.broker_futures_contract_price_data.get_prices_at_frequency_for_contract_object(
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/ib_futures_contract_price_data.py", line 166, in get_prices_at_frequency_for_contract_object
    prices = self._get_prices_at_frequency_for_contract_object_no_checking(
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/ib_futures_contract_price_data.py", line 181, in _get_prices_at_frequency_for_contract_object_no_checking
    return self._get_prices_at_frequency_for_contract_object_no_checking_with_expiry_flag(
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/ib_futures_contract_price_data.py", line 208, in _get_prices_at_frequency_for_contract_object_no_checking_with_expiry_flag
    self.futures_contract_data.get_contract_object_with_IB_data(
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/ib_futures_contracts_data.py", line 84, in get_contract_object_with_IB_data
    futures_contract_with_ib_data.update_expiry_dates_one_at_a_time_with_method(
  File "/home/pysystemtrade/opt/pysystemtrade/sysobjects/contracts.py", line 246, in update_expiry_dates_one_at_a_time_with_method
    new_expiries = [
  File "/home/pysystemtrade/opt/pysystemtrade/sysobjects/contracts.py", line 247, in <listcomp>
    method(single_contract, **kwargs)
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/ib_futures_contracts_data.py", line 122, in _get_actual_expiry_date_given_single_contract_with_ib_metadata
    expiry_date = self.ib_client.broker_get_single_contract_expiry_date(
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/client/ib_contracts_client.py", line 81, in broker_get_single_contract_expiry_date
    ibcontract = self.ib_futures_contract(
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/client/ib_contracts_client.py", line 347, in ib_futures_contract
    ibcontract_with_legs = self.ib_futures_contract_with_legs(
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/client/ib_contracts_client.py", line 375, in ib_futures_contract_with_legs
    ibcontract_with_legs = self._get_stored_or_live_contract(
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/client/ib_contracts_client.py", line 390, in _get_stored_or_live_contract
    ibcontract_with_legs = self.cache.get(
  File "/home/pysystemtrade/opt/pysystemtrade/syscore/cache.py", line 31, in get
    value_from_store = self._calculate_and_store(
  File "/home/pysystemtrade/opt/pysystemtrade/syscore/cache.py", line 38, in _calculate_and_store
    value = function_instance(*args, **kwargs)
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/client/ib_contracts_client.py", line 434, in _get_ib_futures_contract_from_broker
    ibcontract_with_legs = self._get_vanilla_ib_futures_contract_with_legs(
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/client/ib_contracts_client.py", line 449, in _get_vanilla_ib_futures_contract_with_legs
    ibcontract = self._get_vanilla_ib_futures_contract(
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/client/ib_contracts_client.py", line 516, in _get_vanilla_ib_futures_contract
    ibcontract_list = self.ib_get_contract_chain(
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/client/ib_contracts_client.py", line 592, in ib_get_contract_chain
    new_contract_details_list = self.get_contract_details(
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/client/ib_client.py", line 207, in get_contract_details
    contract_details = self._get_contract_details(
  File "/home/pysystemtrade/opt/pysystemtrade/sysbrokers/IB/client/ib_client.py", line 230, in _get_contract_details
    return self.ib.reqContractDetails(ib_contract_pattern)
  File "/home/pysystemtrade/anaconda3/envs/pysystemtrade-dev/lib/python3.8/site-packages/ib_insync/ib.py", line 948, in reqContractDetails
    return self._run(self.reqContractDetailsAsync(contract))
  File "/home/pysystemtrade/anaconda3/envs/pysystemtrade-dev/lib/python3.8/site-packages/ib_insync/ib.py", line 318, in _run
    return util.run(*awaitables, timeout=self.RequestTimeout)
  File "/home/pysystemtrade/anaconda3/envs/pysystemtrade-dev/lib/python3.8/site-packages/ib_insync/util.py", line 341, in run
    result = loop.run_until_complete(task)
  File "/home/pysystemtrade/anaconda3/envs/pysystemtrade-dev/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
ConnectionError: Socket disconnect

[robcarver17]

Have you had a look at your IB logs? It does look like something that is outside the fault of pst.