mt_modify.py to support modifying various formats

[kenorb]

This task is about expanding mt_modify.py script in dev branch, to support the modification of the following file formats. The easiest way to reproduce the tasks is to use Docker (ideally Docker Compose) using the below steps, otherwise I can try to provide these files manually.

• FXT header (modify records)
• FCC header (modify records)
• HST header (modify records)
• SEL (modify records)
• SRV (modify records)
• symbols.raw (add, modify, delete records); this works, but it should be confirmed as still working
• symgroups (adding, deleting modify records)
• ticks.raw (modify records)

Most of the code is already there, so it shouldn't be too complicated to implement it.

---

FXT format

Preparation

Example commands to execute to achieve the desired output.

Preparation using Docker (locating FXT file):

1. Clone this repo, switch to dev branch.
2. Run: docker-compose run Shell to enter Docker container (scripts/py folder is going to be mounted as /opt/scripts, so you can edit them outside of the container). Alternatively run: docker run -it ea31337/ea-tester:dev bash (better works on Windows).
3. Download FXT files by this command inside the Docker: /opt/scripts/bt_data_get.sh EURUSD 2018 DS.
4. Inside container, source these files to load the shell variables: source /opt/scripts/.vars.inc.sh && source /opt/scripts/.funcs.cmds.inc.sh.
5. Go to the folder with FXT files by: cd "$TICKDATA_DIR". You should see the .fxt file by ls *.fxt.
6. Run script either by mt_modify or /opt/scripts/py/mt_modify.py.

Preparation without using Docker (locating FXT file):

1. Download FXT file manually from here and unpack it.

Task

The task is to expand mt_modify.py to be able to modify various values by specifying key, in formats such as string, integer, floats and dates.

Reading and modifying file by specifying the new value for the given record:

1. Read the file header by: mt_read -i EURUSD1_0.fxt -t fxt-header (mt_read script works).
2. Modify the file (TO FIX): mt_modify -i EURUSD1_0.fxt -t fxt-header -m symbol=XXXYYY (mt_modify script needs work).

Expected results

Expected behaviour after modifying the file (string type):

$ mt_read -i EURUSD1_0.fxt -t fxt-header | grep ^symbol
symbol = EURUSD
$ mt_modify -i EURUSD1_0.fxt -t fxt-header -m symbol=XXXYYY # TODO
# TODO: Some confirmation message.
$ mt_read -i EURUSD1_0.fxt -t fxt-header | grep ^symbol # Expected output
symbol = XXXYYY

Modifying date type:

$ mt_read -i EURUSD1_0.fxt -t fxt-header | grep ^testBeginDate
testBeginDate = 2018-01-01 22:00:00
$ mt_modify -i EURUSD1_0.fxt -t fxt-header -m "testBeginDate=2018-01-02 23:00:00" # TODO
# TODO: Some confirmation message.
$ mt_read -i EURUSD1_0.fxt -t fxt-header | grep ^testBeginDate # Expected output
testBeginDate = 2018-01-02 23:00:00

Modifying float type:

$ mt_read -i EURUSD1_0.fxt -t fxt-header | grep pointSize    
pointSize = 1e-05 # Ideally this should show as 0.00001
$ mt_modify -i EURUSD1_0.fxt -t fxt-header -m pointSize=0.01
$ mt_read -i EURUSD1_0.fxt -t fxt-header | grep ^pointSize # Expected output
pointSize = 0.01

The same should work for the integer values.

Note: FXT format consist header and data. Executing mt_modify with -t fxt-header should only modify the header values. Ideally we shouldn't read or write the whole file for performance reasons.

HST format

Preparation

You can download .hst files in the same method as above.

With Docker:

1. Run Docker container by: docker-compose run Shell. Alternatively run: docker run -it ea31337/ea-tester:dev bash (better works on Windows).
2. Download HST files by this command inside the Docker: /opt/scripts/bt_data_get.sh EURUSD 2018 DS.
3. Inside container, source these files to load the shell variables: source /opt/scripts/.vars.inc.sh && source /opt/scripts/.funcs.cmds.inc.sh.
4. Go to the history folder by: cd "$HISTORY_DIR"/default (which is ~/.wine/drive_c/Program Files/MetaTrader 4/history/default). You should see the .hst file by ls *.hst.
5. Run script either by mt_modify or /opt/scripts/py/mt_modify.py.

Preparation without using Docker (locating FXT file):

1. Download FXT file manually from here and unpack it.

Task

The task is to expand mt_modify.py to be able to modify various values by specifying key, in formats such as string, integer, floats and dates.

Reading and modifying file by specifying the new value for the given record:

1. Read the file header by: mt_read -i EURUSD1.hst -t hst-header (this works).
2. Modify the file (TO FIX): mt_modify -i EURUSD1.hst -t hst-header -m key=value (this needs work).

Expected results

Expected results after modifying string:

$ mt_read -i EURUSD1.hst -t hst-header | grep symbol
symbol = EURUSD
$ mt_modify -i EURUSD1.hst -t hst-header -m symbol=XXXYYY
$ mt_read -i EURUSD1.hst -t hst-header | grep symbol
symbol = XXXYYY

Test the shorter or longer strings. This should also work for other fields which store integers and dates (as in previous examples).

HCC format

Preparation

To generate .hcc file, you've to convert it from the CSV files.

Steps with Docker container:

1. Login to container via docker-compose run Shell command.
2. Download few CSV files via: /opt/scripts/py/bt_data_dl.py -p EURUSD -y 2018 -m 2 -d 2 -c.
3. Convert into HCC format via: /opt/scripts/py/conv_csv_to_mt.py -i download/dukascopy/EURUSD/2018/02/2018-02-02--02h_ticks.csv -f hcc (it's going to generate a .hcc file).
4. Read it via: /opt/scripts/py/mt_read.py -i EURUSD1.hcc -t hcc-header.

Without Docker, just use the same scripts as above from the repo.

Expected results

$ /opt/scripts/py/mt_read.py -i EURUSD1.hcc -t hcc-header | grep name
name = History
$ /opt/scripts/py/mt_modify.py -i EURUSD1.hcc -t hcc-header -m name=Foo
$ /opt/scripts/py/mt_read.py -i EURUSD1.hcc -t hcc-header | grep name
name = Foo

SRV format

Preparation

The files can be found in Docker container in ~/.wine/drive_c/Program Files/MetaTrader 4/config or /opt/conf/servers, or in conf/servers in the repo it-self.

Task

The task is to expand mt_modify.py to be able to modify various values by specifying key, in formats such as string, integer, floats and dates.

Expected results

$ mt_read -i FILE.srv -t srv | grep serverName
serverName = SomeValue
$ mt_modify -i FXCM-USDDemo01.srv -t srv -m serverName=Foo
serverName = Foo

The same should work for other strings such as serverAddress.

RAW formats such as symbols.raw, ticks.raw and symgroups.raw

The RAW file can be found at ~/.wine/drive_c/Program Files/MetaTrader 4/history/default/symbols.raw in the container, or download one of the ZIP files from 4.x Release tab (this type of file should be there).

Ignore ticks.raw and symgroups.raw, as I don't know where it find it, but probably it'll work after the above work.

SEL format

Ignore it, as I don't know where it find it. It's probably is going to work after the above work.

[kenorb]

Fixed in #156