Unable to get historical data from CEX.io

[askmike]

Since a couple of days the CEX.io API does not give back old trades anymore (using the since parameter). After contacting support they stated:

Hi Mike,

We have limited our trade_history call to 1000 trades now, We will be implementing a daily trade static file in a future update I apologize for the inconvenence and we will try to get this out ASAP.

Which means that until the static file is offered we need to redesign CEX.io implementation to:

• on gekko start, start gathering trades and save them in the database (this functionality is already there). But don't calculate EMA's or give any advice yet.
• Monitor how many trades we have stored in the database, as soon as the oldest data we have is old enough to calculate the start EMA's (config.EMA.candles * config.EMA.interval) start giving advice and trade.

[nicobrion]

looking forward for the CEX.io historical data fix. I truly congratulate you for an outstanding dev, this project is awesome!
Kudos,

Nico

[askmike]

Thanks!

I am working on a new version of Gekko that is able to store history locally, this means it is able to monitor:

• CEX.io
• BTC-e (including altcoin markets)
• Bitstamp (was broken for a long time)
• All other API's that don't offer historical data.

So far I have written the functionality that can monitor the exchange and store candles. If people could test it that would be awesome!

Test new Gekko

• Go to the dev branch and download it.
• Cd to it & npm install
• Set the config to either cexio, bitstamp or btce. (Mt. Gox not implemented yet, just as: real trading / mailer / advice). [edit: and pick a new market here]
• Run it
• History is now saved in the history folder.
• Give me feedback!

[fmjensen]

Is this okay ?
I dont know what to expect, so you just got the output here :

gekko-localDB : node gekko.js 
2013-12-24 10:48:25 (INFO): I'm gonna make you rich, Bud Fox.
2013-12-24 10:48:25 (INFO): Let me show you some Exponential Moving Averages.
2013-12-24 10:48:25 (INFO): Using normal settings to monitor the live market
2013-12-24 10:48:25 (INFO): NOT trading with real money
2013-12-24 10:48:25 (INFO): Starting to watch the market: CEX.io BTC/GHS
2013-12-24 10:48:25 (DEBUG):    Scheduling next fetch: in 1 minutes
2013-12-24 10:48:25 (DEBUG):    Fetched 1001 new trades, from 09:29:18 (UTC) to 09:48:23 (UTC)
2013-12-24 10:48:25 (DEBUG):    Creating a new daily database for day 2013-12-24
2013-12-24 10:48:25 (DEBUG):    inserting candle 569 (09:29:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 570 (09:30:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 571 (09:31:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 572 (09:32:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 573 (09:33:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 574 (09:34:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 575 (09:35:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 576 (09:36:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 577 (09:37:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 578 (09:38:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 579 (09:39:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 580 (09:40:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 581 (09:41:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 582 (09:42:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 583 (09:43:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 584 (09:44:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 585 (09:45:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 586 (09:46:00 UTC)
2013-12-24 10:48:25 (DEBUG):    inserting candle 587 (09:47:00 UTC)
2013-12-24 10:48:25 (DEBUG):    Leftovers: 588
2013-12-24 10:49:26 (DEBUG):    Scheduling next fetch: in 1 minutes
2013-12-24 10:49:26 (DEBUG):    Fetched 1001 new trades, from 09:29:52 (UTC) to 09:49:25 (UTC)
2013-12-24 10:49:26 (DEBUG):    inserting candle 588 (09:48:00 UTC)
2013-12-24 10:49:26 (DEBUG):    Leftovers: 589
2013-12-24 10:50:27 (DEBUG):    Scheduling next fetch: in 1 minutes
2013-12-24 10:50:27 (DEBUG):    Fetched 1001 new trades, from 09:29:52 (UTC) to 09:50:26 (UTC)
2013-12-24 10:50:27 (DEBUG):    inserting candle 589 (09:49:00 UTC)
2013-12-24 10:50:27 (DEBUG):    Leftovers: 590
2013-12-24 10:51:27 (DEBUG):    Scheduling next fetch: in 1 minutes

[askmike]

Thanks for testing :)

Looks great, I'll explain some things:

• Why so much logging? I am still developing it and I want all feedback I can get.
• You now have a new history folder (inside Gekko) that has files like cexio-BTC-GHS-2013-12-24.db, this is all the trade information Gekko was able to store!
• The times are off: no they are not, they're in in UTC. The reason for this is that everyone from over the world can share historical candles and the timezones doesn't mess everything up.

One small thing I am going to look into though: It shouldn't fetch once every minute (doesn't cause any trouble, but a lot of overhead).

[fmjensen]

I configured it to fetch every minute, and run in debug mode to get some activity on stdout ;-)
The time fits my timezone.
Im running on OSX btw...

[askmike]

Ah, that explains it. And the answers were also for clarifying the new logic for everyone.

---

On CEX.io [and BTC-e] you only get the last 1000 [or 150] trades, whether they happened in a minute or over the last hour. This makes it quite hard to determine when to fetch. But I quite like how the current setup deals with it: after every fetch it calculates a new fetchtime based on trades/min from the previous fetch. It appears to do well but it needs to prove itself over langer periods of time.

Here is a log where you can see this in action:

Fetched 1001 new trades, from 12:06:49 (UTC) to 12:07:19 (UTC)
inserting candle 726 (12:06:00 UTC)
Leftovers: 727
Scheduling next fetch: in 1 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:08:17 (UTC)
inserting candle 727 (12:07:00 UTC)
Leftovers: 728
Scheduling next fetch: in 2 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:09:22 (UTC)
inserting candle 728 (12:08:00 UTC)
Leftovers: 729
Scheduling next fetch: in 3 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:11:00 (UTC)
inserting candle 729 (12:09:00 UTC)
inserting candle 730 (12:10:00 UTC)
Leftovers: 731
Scheduling next fetch: in 5 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:13:52 (UTC) 
inserting candle 731 (12:11:00 UTC)
inserting candle 732 (12:12:00 UTC)
Leftovers: 733
Scheduling next fetch: in 8 minutes
Fetched 1001 new trades, from 12:06:49 (UTC) to 12:18:34 (UTC)
inserting candle 733 (12:13:00 UTC)
inserting candle 734 (12:14:00 UTC)
inserting candle 735 (12:15:00 UTC)
inserting candle 736 (12:16:00 UTC)
inserting candle 737 (12:17:00 UTC)
Leftovers: 738
Scheduling next fetch: in 13 minutes

I do want to make it more aggressive, missing trades is not acceptable.