Add multi price feed mode with weighted averages and fall-back on failed feed

[sacarlson]

// see sample feature request here: #2

Desired Behavior

Would like to have a mode that uses more than one price feed from say cctx and combine the output to some effect.

Impact

Not sure how many others would like these features. maybe just the fall-back or diff failure spec

Feature Suggestion

I was thinking it would be nice if we could use multiple price feeds with just some new config file features added to control how it effects final kelp feed. Maybe add a multi feed mode with each added feed that you already support with cctx for having some multiple weight on how it effected the final output feed to kelp. Also could add some critical point that would make the feed fail at some point. The fail point could be as simple as they all must be within some max percent different from each other or maybe just one must be within the group averge output. many possibilities at that point of failure and what can be done in that case. Also might want to add if one fails that it's output would just not be included in the outcome on that trade run. It would be cool to have some redundancy with fall-back to this on failure of that.
my present ruby bot already supports dual feeds of kraken and polonix that just makes sure of close match with settable percent limit differences that just returns failure with disabled trade on that run if match out of spec.
My other idea on this same subject is to write a ruby cctx feed that does the above actions by reading a group of cctx or other feeds and have kelp read the ruby cctx server to perform the actions discussed above. We can try this and many other more complex ideas with trail in a ruby prototype

References

Additional context

// Add any additional context here including any alternative solutions you considered (if any) and why

Specification

// If this feature does not exist anywhere else then please provide additional details here.
// This can be as long and detailed as may be necessary.

...

[nikhilsaraf]

@sacarlson
We have two options.. both of which I'd like to support in Kelp at some point. I'd recommend we go for option 1 first as that may be simpler to develop in Kelp and would be more customizable by you too. Of course if you can write your price feed in golang that would be quickest:

---

1. Support HTTP based price feeds: support a "remote" type price feed where you specify a URL that should be called out to along with it's query params (example: https://my-price-feed.com/?currency=BTCETH). This HTTP endpoint will return you a price. This way you can have a price feed written in any language (Python, Ruby, Golang, etc.) and can make it as complex as you want with data filtering, matching, etc. as long as it adhere's to the sample API below.

Here's a sample API that we could use for this:
Whatever URL you input into the DATA_FEED_A_URL will be hit with a GET request.
example: https://my-price-feed.com/?currency=BTCETH
another example: https://any-http-endpoint-here-which-always-returns-XLM-USD-price.com

Response format from your price feed server (v1.0):

price: float
precision: integer (representing precision of price in number of decimals)

example response:

{
    price: 1.23445789,
    precision: 8,               
}

---

2. Combine existing price feeds: support a "formula" type feed where your URL specifies the feeds you want to combine. You can support a lot of the features you mentioned: dropping feeds that have errors, matching feed prices within a specific range (will need to be tackled as a subsequent issue), combining different feeds based on weight, etc. Here's an example of what your config URL may look like for something like this:

DATA_FEED_A_URL="averageDropErrors((exchange;kraken/XXLM/ZUSD), ((exchange;ccxt-binance/XLM/USDT) / (crypto;https://api.coinmarketcap.com/v1/ticker/tether/)))"

In the example above, we've used the function "averageDropErrors", which would return the average of the feeds by dropping any that had an error.
Some other functions we could support are: "average", "min", "max", "first", "median", "dropError", etc.
As off today, none of these functions exist in Kelp so we would need to build out each of these "native" functions to really make this feed type usable.

---

The first option seems a lot quicker to develop for your specific use case. I'd recommend preparing your ruby price feed to serve data in the format specified in the first point. That way whenever we support HTTP price feeds (I'll create a separate github issue for that) you'll be able to use your custom ruby based price feed.

@sacarlson let me know your thoughts on the proposal.

[nikhilsaraf]

idea for a possible (theoretical) workaround (credit to @sacarlson): implement a few endpoints to mock out the CCXT REST interface (/exchanges/:exchangeName/:exchangeInstanceId/fetchTicker, /exchanges/:exchangeName/:exchangeInstanceId, etc.). fetchTicker has your price feed logic.

Host this web server at port 3000 at the appropriate URL endpoints.
Create your new ccxt type in factory.go (or overwrite an existing one). In theory, you should be able to connect to this "remote" CCXT price feed server.

[sacarlson]

I have already written a prototype ccxt-rest-averager in ruby that acts as a middle restclient api server to another real ccxt-rest server that is also running on a different port on the same system. kelp can now interface to make use of it for a weight averaged price feed. the ruby ccxt-server has modes for customizable fallback feeds and weighted averages mode of a group of exchange feeds that are obtained from the real ccxt-server or some of the other built in feeds I also wrote in ruby. This can be hooked into kelp or most any other software that supports the ccxt feed format. Remind you this is just quick crap code I write to experiment with the concept idea. I expect you all to make something more production oriented. I think a golang version of something like this might be cool. or maybe just modify a branch of ccxt or ccxt-rest to add the features at that end. At this point the ruby code is the quickest solution I can deal with to get what I need now. I've now made the initial release of the ruby code I have been speaking of at: https://github.com/sacarlson/ccxt-rest-averager