-
|
Hello All Example: stack_handler_echos_output_2022_07_13.txt After Stack_handler shut down I re-ran run_strategy_order_generator manually and attempted to trade manually using Interactive_order_stack: Of course you may ask, how the hell are you only seeing this now? What I have checked and tried.
All pretty standard so looks ok.
Any other suggestions from the Hive Mind? |
Beta Was this translation helpful? Give feedback.
Answered by
rorymac
Jul 15, 2022
Replies: 5 comments 16 replies
-
|
Just to quickly help you not waste time: the risk overlay is part of
running the system. If instrument orders are being generated, risk overlay
has nothing to do with it.
My first thought was trading hours.
Take a look at
sysbrokers/IB/ib_trading_hours.py: get_conservative_trading_time_for_time_zone
& get_time_difference.
Rob has those based on UTC/GMT (London time). If you're in the US, I think
you'll need to modify those (not sure if that is intentional, or an
oversight - I've never heard Rob mention the need to modify it). In any
case, I have modified it in my repo based on US/Mountain (or maybe it was
US/Pacific - I'm in Arizona where we don't believe in Daylight Savings
Time, so it changes).
…On Thu, Jul 14, 2022 at 7:56 AM Rory Mackay ***@***.***> wrote:
Hello All
I am putting this here because I think it is user error, not a code issue.
I am running SHA de8a348
<de8a348>
Linux Mint
*TLDR:*
Running stack handler some instruments trade automatically others do not,
trying to then do manual trade the same instruments do not trade. No error
messages are produced.
*Example:*
In the text file is the dump of the echo file for stack_handler.
BOBL trades
BUT the following do not.
GOLD_micro
MXP
NASDAQ_micro
BITCOIN
The contract orders are created, no error messages are created.
stack_handler_echos_output_2022_07_13.txt
<https://github.com/robcarver17/pysystemtrade/files/9112494/stack_handler_echos_output_2022_07_13.txt>
After Stack_handler shut down I re-ran run_strategy_order_generator
manually and attempted to trade manually using Interactive_order_stack:
option 1 then option 12 and then specify contract number, no trade and no
error message.
CONTRACT STACK
(Order ID:1811) Type best for live_300/GOLD_micro/20220800, qty [-1], fill [0]@ price, None Parent:1946 Children:no_children
(Order ID:1812) Type best for live_300/MXP/20220900, qty [-1], fill [0]@ price, None Parent:1947 Children:no_children
(Order ID:1813) Type best for live_300/NASDAQ_micro/20220900, qty [-1], fill [0]@ price, None Parent:1948 Children:no_children
(Order ID:1814) Type best for live_300/BITCOIN/20220700, qty [-3], fill [0]@ price, None Parent:1949 Children:no_children
Which contract order ID? <RETURN for for all> 1811
Are you sure? (Y/other)Y
If stack process not running, your next job will be to get the fills from IB
10: Spawn contract orders from instrument orders
11: Create force roll contract orders
12: Create (and try to execute...) IB broker orders
13: Balance trade: Create a series of trades and immediately fill them (not actually executed)
14: Balance instrument trade: Create a trade just at the strategy level and fill (not actually executed)
15: Manual trade: Create a series of trades to be executed
16: Cash FX trade
Of course you may ask, how the hell are you only seeing this now?
For reasons that are neither intelligent or interesting, I have been
physically trading then booking the deals manually for a while.
*What I have checked and tried.*
1.
Config files not set up properly
I checked my private_config.yaml
The instruments are not on the any of the restricted lists.
and the private_control_config.yaml
The flag for preventing stack handler from trading is not commented
out. (long shot but thought I should check)
2.
So could be a risk override
My risk parameters are set up as follows in the live_system yaml file:
## RISK overlay
risk_overlay:
max_risk_fraction_normal_risk: 1.4
max_risk_fraction_stdev_risk: 3.6
max_risk_limit_sum_abs_risk: 3.4
max_risk_leverage: 13.0
All pretty standard so looks ok.
1. May be a trading hours issue.
I run the stack_handler from 00:01 to 11:01 CET (US)
also looked into rob's code and that appears pretty robust.
Any other suggestions from the Hive Mind?
—
Reply to this email directly, view it on GitHub
<#687>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA3JWKQME2P7TSY23CLDR7TVUATBFANCNFSM53SRW7CQ>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Beta Was this translation helpful? Give feedback.
2 replies
-
|
Thanks Todd |
Beta Was this translation helpful? Give feedback.
-
|
I run on EST and I had the same issue until I modified the time definitions to match my local time |
Beta Was this translation helpful? Give feedback.
-
|
Check out sysexecution/stack_handler/create_broker_orders_from_contract_orders.py So I'd place a breakpoint and step through to see whether the order is surviving the pre process stage. Amongst other things, this checks to see if the market is open. If it makes it past there, then is the algo able to place the order? Again there are a few things that could go wrong here. NOTE you may ask why there isn't more logging here to say what's going on. The problem is, for example, a market is closed then we'd spam the logs every 5 seconds saying this. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Thanks to the hive mind, the problem was indeed that the system as written is geared to GMT, as it should be given that the BDFL is resident in that time zone. Thanks to Todd and Meldinman for pointing out the time issue, and thanks to Rob for showing me how to run a trace and where. Here is the hack that fixed the time issue: Now I do believe if I submitted this a pull request I would invoke the justifiable wrath of the BDFL. This would have to include an entry in the private.config.YAML Two points:
|
Beta Was this translation helpful? Give feedback.
4 replies
-
|
I've noticed this week that this doesn't completely solve the issue. I had a LEANHOG contract order that was not being submitted to the broker. |
Beta Was this translation helpful? Give feedback.
-
|
meldinman |
Beta Was this translation helpful? Give feedback.
-
|
I just shifted them all five hours for me in EST I think it's happening to LEANHOG because its Market hours are different than most other us contract where it closes during the day. Index futures for instance show an open time of 18:00 - 16:00 if I remember correctly so adjusting them conservatively doesn't block them out. |
Beta Was this translation helpful? Give feedback.
-
|
Great, Thanks, so by implication for me in central I would need to shift -6. Thank you very much. |
Beta Was this translation helpful? Give feedback.
Answer selected by
rorymac
-
|
I would be glad to test if Rob approves this change (or something similar).
BTW, I haven't reviewed this code, so this isn't an endorsement. But I do
think this functionality is needed.
Also, couldn't the local timezone be detected and used to calculate the
offset without needing config?
…On Fri, Jul 15, 2022, 7:39 AM Rory Mackay ***@***.***> wrote:
Thanks to the hive mind, the problem was indeed that the system as written
is geared to GMT, as it should be given that the BDFL is resident in that
time zone.
Thanks to Todd and Meldinman for pointing out the time issue, and thanks
to Rob for showing me how to run a trace and where.
Here is the hack that fixed the time issue:
def get_time_difference(time_zone_id: str) -> int:
# Doesn't deal with DST. We will be conservative and only trade 1 hour
# after and 1 hour before
# confusingly, IB seem to have changed their time zone codes in 2020
time_diff_dict = {
"CST (Central Standard Time)": 6,
"MET (Middle Europe Time)": -1,
"EST (Eastern Standard Time)": 5,
"JST (Japan Standard Time)": -8,
"US/Eastern": 5,
"MET": -1,
"EST": 5,
"JST": -8,
"Japan": -8,
"US/Central": 6,
"GB-Eire": 0,
"Hongkong": -7,
"": 0,
}
# would rather that this in private.config.yaml
# lets run this and see if it works
GMT_offset = -6
for k, v in time_diff_dict.items():
time_diff_dict[k] = v + GMT_offset
diff_hours = time_diff_dict.get(time_zone_id, None)
if diff_hours is None:
raise Exception("Time zone '%s' not found!" % time_zone_id)
return diff_hours
Now I do believe if I submitted this a pull request I would invoke the
justifiable wrath of the BDFL.
With that in mind here is what I would like to submit, and I would welcome
comments on it:
import datetime
from ib_insync import ContractDetails as ibContractDetails
from syscore.dateutils import adjust_trading_hours_conservatively, openingTimesAnyDay, openingTimes, listOfOpeningTimes
from sysdata.config.production_config import get_production_config
def get_conservative_trading_hours(ib_contract_details: ibContractDetails) -> listOfOpeningTimes:
time_zone_id = ib_contract_details.timeZoneId
conservative_times = get_conservative_trading_time_for_time_zone(time_zone_id)
trading_hours = get_trading_hours(ib_contract_details)
trading_hours_adjusted_to_be_conservative = adjust_trading_hours_conservatively(
trading_hours, conservative_times=conservative_times
)
return trading_hours_adjusted_to_be_conservative
def get_trading_hours(ib_contract_details: ibContractDetails) -> listOfOpeningTimes:
try:
time_zone_id = ib_contract_details.timeZoneId
time_zone_adjustment = get_time_difference(time_zone_id)
one_off_adjustment = one_off_adjustments(ib_contract_details.contract.symbol)
trading_hours_string = ib_contract_details.tradingHours
list_of_open_times = parse_trading_hours_string(
trading_hours_string,
adjustment_hours=time_zone_adjustment,
one_off_adjustment=one_off_adjustment,
)
except Exception as e:
raise e
return list_of_open_times
NO_ADJUSTMENTS = 0, 0
CLOSED_ALL_DAY = object()
def parse_trading_hours_string(
trading_hours_string: str,
adjustment_hours: int = 0,
one_off_adjustment: tuple = NO_ADJUSTMENTS,
) -> listOfOpeningTimes:
day_by_day = trading_hours_string.split(";")
list_of_open_times = [
parse_trading_for_day(
string_for_day,
adjustment_hours=adjustment_hours,
one_off_adjustment=one_off_adjustment,
)
for string_for_day in day_by_day
]
list_of_open_times = [
open_time for open_time in list_of_open_times if open_time is not CLOSED_ALL_DAY
]
list_of_open_times = listOfOpeningTimes(list_of_open_times)
return list_of_open_times
def parse_trading_for_day(
string_for_day: str,
adjustment_hours: int = 0,
one_off_adjustment: tuple = NO_ADJUSTMENTS,
) -> openingTimes:
start_and_end = string_for_day.split("-")
if len(start_and_end) == 1:
# closed
return CLOSED_ALL_DAY
start_phrase = start_and_end[0]
end_phrase = start_and_end[1]
# Doesn't deal with DST. We will be conservative and only trade 1 hour
# after and 1 hour before
adjust_start = 1 + one_off_adjustment[0]
adjust_end = -1 + one_off_adjustment[-1]
start_dt = parse_phrase(
start_phrase, adjustment_hours=adjustment_hours, additional_adjust=adjust_start
)
end_dt = parse_phrase(
end_phrase, adjustment_hours=adjustment_hours, additional_adjust=adjust_end
)
return openingTimes(start_dt, end_dt)
def parse_phrase(phrase: str, adjustment_hours: int = 0, additional_adjust: int = 0)\
-> datetime.datetime:
total_adjustment = adjustment_hours + additional_adjust
original_time = datetime.datetime.strptime(phrase, "%Y%m%d:%H%M")
adjustment = datetime.timedelta(hours=total_adjustment)
return original_time + adjustment
def get_conservative_trading_time_for_time_zone(time_zone_id: str) -> openingTimesAnyDay:
# ALthough many things are liquid all day, we want to be conservative
# confusingly, IB seem to have changed their time zone codes in 2020
# times returned are in UTC
start_times = {
## US
"CST (Central Standard Time)": 15,
"US/Central": 15,
"CST": 15,
"EST (Eastern Standard Time)": 14,
"US/Eastern": 14,
"EST": 14,
## UK
"GB-Eire": 9,
"": 9,
## Middle European
"MET (Middle Europe Time)": 8,
"MET": 8,
## Asia
"JST (Japan Standard Time)": 1,
"JST": 1,
"Japan": 1,
"Hongkong": 1,
}
end_times = {
## US
"CST (Central Standard Time)": 20,
"US/Central": 20,
"CST": 20,
"EST (Eastern Standard Time)": 19,
"US/Eastern": 19,
"EST": 19,
## UK
"GB-Eire": 16,
"": 16,
## Middle European
"MET (Middle Europe Time)": 15,
"MET": 15,
## Asia
"JST (Japan Standard Time)": 6,
"JST": 6,
"Japan": 6,
"Hongkong": 6,
}
conservative_start_time = datetime.time(start_times[time_zone_id])
conservative_end_time = datetime.time(end_times[time_zone_id])
return openingTimesAnyDay(conservative_start_time,
conservative_end_time)
def get_GMT_offset_hours():
# this needs to be in private_config.YAML
# where are the defaults stored that needs to be
# GMT_offset_hours = 0
try:
production_config = get_production_config()
GMT_offset_hours = production_config.GMT_offset_hours
except:
raise Exception("Default is zero, have it in private_config")
return GMT_offset_hours
def get_time_difference(time_zone_id: str) -> int:
# Doesn't deal with DST. We will be conservative and only trade 1 hour
# after and 1 hour before
# confusingly, IB seem to have changed their time zone codes in 2020
time_diff_dict = {
"CST (Central Standard Time)": 6,
"MET (Middle Europe Time)": -1,
"EST (Eastern Standard Time)": 5,
"JST (Japan Standard Time)": -8,
"US/Eastern": 5,
"MET": -1,
"EST": 5,
"JST": -8,
"Japan": -8,
"US/Central": 6,
"GB-Eire": 0,
"Hongkong": -7,
"": 0,
}
GMT_offset_hours = GMT_offset_hours
for k, v in time_diff_dict.items():
time_diff_dict[k] = v + GMT_offset_hours
diff_hours = time_diff_dict.get(time_zone_id, None)
if diff_hours is None:
raise Exception("Time zone '%s' not found!" % time_zone_id)
return diff_hours
def one_off_adjustments(symbol: str) -> tuple:
## Instrument specific - none needed any more
## Leave code unless have problems again
return NO_ADJUSTMENTS
This would have to include an entry in the private.config.YAML
as follows :
GMT_offset_hours: '-6' # for US CST , the default would be '0'
Two points:
1. would this pass muster ?
2. I have no way of testing this (as I am on a good day an average
coder), how do I test it before submitting the pull request?
—
Reply to this email directly, view it on GitHub
<#687 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AA3JWKT2CPHQKOJ26O7DTDLVUFZ3LANCNFSM53SRW7CQ>
.
You are receiving this because you commented.Message ID:
***@***.***
com>
|
Beta Was this translation helpful? Give feedback.
3 replies
-
|
If someone submits a PR, and it's not horribly hacky, and it doesn't break my system I'll approve obviously |
Beta Was this translation helpful? Give feedback.
-
|
I wanted to follow up on this issue. I've tried changing the dict to what meldinman is using, since I also live in EST. And it works for most instruments, except I get no trading hours for LUMBER. Also, LEANHOG/FEEDCOW/LIVECOW are all in a brief window 15:30-16:00. Has anyone else seen that behavior or have some suggestion to fix it? |
Beta Was this translation helpful? Give feedback.
-
|
Hello So there are two parts to the issue, first is the original adjustment to the first dict time_diff_dict. The second adjustment is needed for the other two dicts start and end times I hacked it like so : The problem with submitting it as a fix is that each timezone is different and you do not want negative hours in the dict then it crashes. Hope that helps |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your help. I assume I would change 6 to 5 for EST in if start_times[k]<=5: correct? |
Beta Was this translation helpful? Give feedback.
7 replies
-
|
Hi @oldlore, Did you end up finding any insight on the trading hours spanning the entire day rather than a smaller, more conservative window? My paper system is taking big residual hits with orders filled during overnight hours that are a lot less liquid than the middle-of-the-day hours. I was hoping that applying conservative trading hours would limit my trades to the most liquid hours of the day. I, too, traced the code down to the snippet you posted and it seems to me like adjust_start_time_conservatively isn't working as intended when faced with trading hours available overnight, however my understanding of BDFL's intentions with this code could be at fault here. |
Beta Was this translation helpful? Give feedback.
-
|
Quoting @oldlore:
This is an issue with how the conservative trading hours are applied, because the trading hours from IB can start and end on different days. Or to put it another way, the start time can be after the end time, if you look at it independently of the date, which is what happens when conservative trading hours are applied. I have a proposed solution to automatically adjust the conservative hours using the GMT offset (see discussion #830), but this is a separate issue. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it is a separate issue; for someone in US EST, most of the instruments' trading hours start and end on different days and the conservative trading hours are not applied (as you can see in oldlore's trading hours for each instrument). I noticed you mentioned this in discussion #830 - "existing code doesn't handle cases where trading hours span midnight in user's timezone (although we could add that part from my proposed changes)" I think the method adjust_start_time_conservatively may need to be rewritten to incorporate the date - have you tackled this already in your proposed changes? Or should I get on it :D |
Beta Was this translation helpful? Give feedback.
-
|
My comment in discussion #830 was referring to the fact that the conservative trading hours might span midnight in some combinations of user timezone and exchange timezone. My change addresses how the conservative hours are adjusted for the GMT offset, but I don't attempt to address the issue you're dealing with there. Edit: or maybe I'm addressing half of it by splitting the conservative hours into pre-midnight and post-midnight, but there is still the problem with how those ranges are applied, in adjust_start_time_conservatively and adjust_end_time_conservatively. |
Beta Was this translation helpful? Give feedback.
-
This is what I've come up with so far. This is in addition to rorymac's changes above. It's ugly but it sort of works. I just tried different numbers for different instruments until I got hours that I thought made sense. This is for EST FYI. The resulting trading hours list (EST) is: |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thanks to the hive mind, the problem was indeed that the system as written is geared to GMT, as it should be given that the BDFL is resident in that time zone.
Thanks to Todd and Meldinman for pointing out the time issue, and thanks to Rob for showing me how to run a trace and where.
Here is the hack that fixed the time issue: