Edit: Timezone discrepancy causing trading hour issues

[craigmediaservices]

I had a few live trades execute flawlessly last night and a position closed this AM.

However new orders are being reported as LOCKED

2024-09-11 13:01:14 DEBUG stack_handler {'type': 'run_stack_handler', 'strategy_name': 'dynamic_system', 'instrument_code': 'SOYMEAL', 'contract_order_id': '', 'instrument_order_id': 120} Put child order (Order ID:no order ID) Type best for dynamic_system/SOYMEAL/20250100, qty [-1], fill [0]@ price, None Parent:120 Children:no_children LOCKED on stack with ID 120 from parent order (Order ID:120) Type best for dynamic_system SOYMEAL, qty [-1], fill [0]@ price, None Parent:no parent Children:no_children

If I try to unlock under 32 or 33 (Lock/unlock order and instrument code) any change to lock status crashes

2024-09-11 13:28:56 DEBUG connectionIB {'type': 'Interactive-Order-Stack', 'broker': 'IB', 'clientid': 115} Terminating {'ipaddress': '127.0.0.1', 'port': 4001, 'client': 115}
2024-09-11 13:28:56 DEBUG Interactive-Order-Stack {'component': 'mongoIbBrokerClientIdData'} Released IB client ID 115
Traceback (most recent call last):
File "/home/trader/pysystemtrade/sysproduction/linux/scripts/run.py", line 66, in
func(*args, **kwargs)
File "/home/trader/pysystemtrade/sysproduction/interactive_order_stack.py", line 61, in interactive_order_stack
interactive_order_stack_with_ib_conn(ib_conn)
File "/home/trader/pysystemtrade/sysproduction/interactive_order_stack.py", line 70, in interactive_order_stack_with_ib_conn
menu.run_menu()
File "/home/trader/pysystemtrade/syscore/interactive/menus.py", line 64, in run_menu
method_chosen(*self._args, **self._kwargs)
File "/home/trader/pysystemtrade/sysproduction/interactive_order_stack.py", line 810, in order_locking
if order.is_order_locked():
AttributeError: 'named_object' object has no attribute 'is_order_locked'

I can also try to unlock under ALGO

(Order ID:120) Type best for dynamic_system/SOYMEAL/20250100, qty [-1], fill [0]@ price, None Parent:120 Children:no_children

It is reported as released

Order ID 120
Controlled by None; releasing now
Released

However I see no messages in the API or Client ID log regarding this order, and it immediately returns to LOCKED status.

There were no trade limits setup at this point. I tried using interactive_order_stack to manually place an order, but also not going through and no updates. I also restarted the system, ran shutdown processes, cleared the orders and regenerated.

Now I don't believe that's the root problem regardless, I am just not clear at what stage these locks are occurring and I don't see any logs related to the lock.

I verified that I am capable of manually placing the order in TWS though.

Any help on where to dig into this would be greatly appreciated!

[tgibson11]

Locking an order is a normal part of the order creation & execution process. They should not remain locked indefinitely, of course, but just seeing a log message stating that an order is locked is not indicative of any problem.

So let's back up a little. What led you to investigate the logs regarding this order?

Maybe you noticed that this order had not been executed when you expected it to have been? If so, I should point out that SOYMEAL, and other CME agricultural contracts have trading hours that are different from most other CME futures. When an order cannot be executed because of trading hours, there are no log messages telling you this, it just looks like stack_handler is ignoring the order. If this more or less describes what happened, then I would first check your trading hour configuration (interactive_diagnostics --> View instrument configuration --> View trading hours for all instruments).

If that is not what happened, please provide more context. The other likely way I can see an order getting stuck in locked status is if there was a stack handler crash, in which case you should have a stack trace for that.

[craigmediaservices]

Thank you so much.

Does that function work for you? Mine keeps raising "missingData" exception in current_contract_dict (from multiple_prices.py)

I added some logging and it is a large number of instruments, not just one.

I was able to bypass it though by modifying the function

def get_trading_hours_for_all_instruments(data=arg_not_supplied):
    if data is arg_not_supplied:
        data = dataBlob()

    diag_prices = diagPrices(data)
    list_of_instruments = diag_prices.get_list_of_instruments_with_contract_prices()

    p = progressBar(len(list_of_instruments))
    all_trading_hours = {}
    for instrument_code in list_of_instruments:
        print(f"Processing instrument: {instrument_code}")  # Log current instrument
        p.iterate()
        try:
            trading_hours = get_trading_hours_for_instrument(data, instrument_code)
        except missingContract:
            print(f"*** NO TRADING HOURS FOR {instrument_code} ***")
            continue
        except missingData:
            print(f"*** MISSING DATA FOR {instrument_code}, skipping ***")
            continue

        # Use the first available trading day
        check_trading_hours(trading_hours, instrument_code)
        all_trading_hours[instrument_code] = trading_hours

    p.close()

    return all_trading_hours

Some of the trades have since gone through as you expected! So good news. I was worried it was an API or account problem.

As you mentioned, trading hours are restricted to a much shorter window than I expected

SOYOIL : 11/09 15:30 to 11/09 18:20 (2.8 hours)

Now I am the proud owner of a more diversified set of instruments (only BTP was owned when I posted)

  instrument_code contract_date          expiry_date  position
7         BITCOIN      20240927  2024-09-27 00:00:00      -2.0
9             BTP      20241206  2024-12-06 00:00:00       1.0
1             CHF      20241216  2024-12-16 00:00:00       1.0
3    COPPER-micro      20241126  2024-11-26 00:00:00      -1.0
4             DOW      20241220  2024-12-20 00:00:00       1.0
8         GAS-PEN      20241028  2024-10-28 00:00:00      -1.0
6             GBP      20241216  2024-12-16 00:00:00       1.0
2             MXP      20241216  2024-12-16 00:00:00      -1.0
5         RUSSELL      20241220  2024-12-20 00:00:00      -1.0
0            US30      20241219  2024-12-19 00:00:00       1.0

Can I buy you a coffee/beer?

[tgibson11]

It does work for me.

I think that exception occurs because you have some instruments with contract prices that don't have multiple prices. That is a bug - the code should use one or the other consistently - but you could also avoid it by deleting the contract prices for instruments you don't use. And now that we're using parquet, that is just a matter of deleting some files, or moving them to a different location if you do want to keep the data around for some reason.

[craigmediaservices]

I think part of my issue was that I had not set gmt_offset for my server time (EST).

So it looks like the trading hours weren't corresponding to what they should have actually been, and trades were locked when it was regular or even good market hours.

[craigmediaservices]

@tgibson11 Thanks again for your help. So I noticed that my CORN and SOYOIL trade is still not going through.

Here is the output of trading hours:

SOYOIL              : 

Now I have dug in, and both the saved trading hours object and the trading hours pulled from IB have data, however it appears there is no intersection between the two hour sets.

Here is an example:

IB Trading Hours: tradingHours(opening_time=datetime.datetime(2024, 9, 19, 11, 30), closing_time=datetime.datetime(2024, 9, 19, 14, 20))
Weekly Trading Hours: {'Monday': [tradingHoursAnyDay(opening_time=datetime.time(15, 0), closing_time=datetime.time(20, 0))], 'Tuesday': [tradingHoursAnyDay(opening_time=datetime.time(15, 0), closing_time=datetime.time(20, 0))], 'Wednesday': [tradingHoursAnyDay(opening_time=datetime.time(15, 0), closing_time=datetime.time(20, 0))], 'Thursday': [tradingHoursAnyDay(opening_time=datetime.time(15, 0), closing_time=datetime.time(20, 0))], 'Friday': [tradingHoursAnyDay(opening_time=datetime.time(15, 0), closing_time=datetime.time(20, 0))], 'Saturday': [tradingHoursAnyDay(opening_time=datetime.time(15, 0), closing_time=datetime.time(20, 0))], 'Sunday': [tradingHoursAnyDay(opening_time=datetime.time(15, 0), closing_time=datetime.time(20, 0))]}
Intersected Trading Hours: []

IB trading hours are listed as 11:30 to 14:20
Saved trading hours are 15:00 to 20:00

It looks like a conflict with the stored YAML. I wonder if running my server in EST and trying to use the gmt_offset is causing some issue somewhere in the chain. Might be easier to just reset server time to UTC?

Or did you find you needed to manually edit the YAML?

[tgibson11]

Yes, the issue is that you need to update the configured trading hours for your local timezone.

The defaults assume you live in GMT.

[craigmediaservices]

A-ha. Thank you!

[tgibson11]

The defaults are defined here.

You probably just need to subtract 4 hours (or whatever) from all the open/close times.

You probably don't need to bother with the instrument-specific hours - but that is supported in case you do need it.

[craigmediaservices]

Do you know the logic behind US hours? They seem odd, but there might be something Futures specific.

US/Eastern:

• '14:00'
• '19:00'

Corresponds to 10am to 3pm EST in the summer and 9am to 2pm in the Winter.

[tgibson11]

US equity markets open at 9:30am EST, close at 4pm, so that is presumed to be the most liquid hours to trade.

Rob doesn't want to trade right at the open or close (presumably due to volatility, and also maybe the risk of not getting filled at the close). So he's conservative with the hours he's willing to trade.

Also, if you live in an area that doesn't have the same DST schedule as an exchange, using conservative hours makes it unnecessary to worry about that too.

[craigmediaservices]

So my logic for switching the server to UTC was because GMT_offset is only used in the code base to deal with IBKR prices. All the defaults are setup to assume the server is using UTC, and I don't want to set GMT_offset twice per year to adjust for DST. Linux/Unix itself defaults to UTC.

I thought it would make sense for a true "global trading system" rather than being primarily EST focused.

IBKR prices appear to be received in the relevant timezone.

[craigmediaservices]

There might also be an issue with having the start time > end time. Rob has this in his config too, but maybe he doesn't actually trade anything in Australia (I don't either). And notice that his other Asia timezones aren't configured like that.

You are a genius!

SPI200              : 16/09 01:00 to 16/09 03:00 (2.0 hours) 17/09 01:00 to 17/09 03:00 (2.0 hours) 18/09 01:00 to 18/09 03:00 (2.0 hours)

That solved the issue. It wasn't actually my EST/UTC setting that time. Thank you for all your insights.

I've been paper trading for about 5-months and thought I knew what I was doing, but obviously I'm an idiot.

[craigmediaservices]

This is offtopic, but you might be able to clarify:

By default the processes run in this order

process_configuration_start_time:
default: '00:01'
run_stack_handler: '00:01'
run_capital_update: '01:00'
run_daily_prices_updates: '20:00'
run_daily_fx_and_contract_updates: '07:00'
run_daily_update_multiple_adjusted_prices: '23:00'
run_systems: '20:05'
run_strategy_order_generator: '20:10'
run_backups: '20:15'
run_cleaners: '20:20'
run_reports: '20:25'

But shouldn't run_daily_update_multiple_adjusted_prices be ran prior to run_systems and run_strategy_order_generator or we're using old prices for generated system positions?

[tgibson11]

You are correct about the ideal order of things, and 23:00 does seem like an odd choice relative to the other processes. Probably a mistake.

However, consider that we are collecting data from markets all around the globe. There is a very narrow window (if there is one at all) where you could expect to have end-of-day data for the current day for all markets. Some prices are likely to be almost 24-hours old no matter what.

Bottom line is you should adjust the times according to your own needs. Also see the previous_process config a bit further down for a better idea of the expected order of running things. Edit: well, never mind, that doesn't clarify much either.

[craigmediaservices]

Thank you, I wanted to make sure I wasn't making another mistake here.

I had modified it to price collect prior to run_systems, but went back to defaults to just make sure everything was good-to-go live as changes can be finicky.

The signals used are slow enough that it might not make much difference, though maybe in extreme tail circumstances it could help.