Incorporate Alpha Vantage as a source of historical equity prices

[pteetor]

Description

The Alpha Vantage web service provides real-time and historical equity data. The service is free, requiring only a free, one-time registration. The service provides daily, weekly, and monthly history for both domestic and international markets, with up to 20 years of history. For daily data, adjusted close prices are available to account for dividends and splits. The service can also provide real-time price bars at a resolution of 1 minute or higher, for up to 10 recent days. They have a clean, documented, public API that returns JSON-encoded data.

I propose incorporating Alpha Vantage into getSymbols as a source of historical data, much like Yahoo and Google.

Expected behavior

The caller would invoke the Alpha Vantage downloader through the usual mechanism; i.e.,

getSymbols("IBM", src="av")`.

The associated function, getSymbols.av, would have this signature.

getSymbols.av(Symbols, env, apikey,
                 return.class = "xts",
                 periodicity = "daily",
                 adjusted = FALSE,
                 interval = "1min",
                 outputsize = "compact",
                 ...)

• Symbols - a character vector specifying the names of the symbols to be loaded
• env - where to create objects.
• apikey - the API key issued by Alpha Vantage when you registered (character)
• return.class - class of returned object
• periodicity - one of "daily", "weekly", "monthly", or "intraday"
• adjusted - if TRUE, include a column of closing prices adjusted for dividends and splits (daily data only)
• interval - one of "1min", "5min", "15min", "30min", or "60min" (intraday data only)
• outputsize - either "compact" or "full"

You must register with Alpha Vantage in order to download their data, but the on-time registration is fast and free. A.V. will assign an API key to you, a short string of alphanumeric characters (e.g., "FOO4").

You must provide your API key every time you call getSymbols.av. Alternatively, you can set it globally using

setDefaults(getSymbols.av, apikey="yourKey")

The Alpha Vantage site provides daily, weekly, monthly, and intraday data. Use periodicity to select one.

Set adjusted=TRUE to include a column of closing prices adjusted for dividends and stock splits (available only for daily data).

The intraday data is provided as a sequence of OHLC bars. Use the interval argument to determine the "width" of the bars: 1 minute, 5 minutes, 15 minutes, etc.

By default Alpha Vantage returns the 100 most-recent data points. Set outputsize="full" to obtain the entire available history (which requires more download time, of course). Alpha Vantage says they provide up to 20 years of daily data: or the most recent 10 or 15 days of intraday data.

Issues and Limitations

The web service is free, but registration is required. Alpha Vantage provides an API key at registration, and that key is needed to download data.

The A.V. API returns its data via JSON. In my experience, pulling large, historical datasets can be slower than Google or Yahoo (but not crippling slow). At this point, I don't know if the bottleneck is the A.V. server itself or the encoding/decoding process for (inefficient) JSON.

Beyond historical data, the A.V. API can also provide sector performance data and technical indicators (e.g., moving averages). I am not planning to provide access to those data elements. The sector performance stuff is kind of orthogonal to quantmod scope; and the technical indicators are already available via the TTR package.

I believe the intraday data is available only for domestic stocks, not international; but I'm not sure.

[joshuaulrich]

Great idea!

Please use the current getSymbols "methods" as a template for getSymbols.av(). There is a getSymbols.skeleton(), but I'm not sure if it's up-to-date. Should we consider creating one (or more?) "alias" functions? Maybe getSymbols.alphavantage()?

Great suggestion to use setDefaults() to store the user's API key.

quantmod already uses jsonlite for a few things (e.g. getOptionChain()), so please take a look at that code to see how it uses jsonlite to manipulate JSON data.

[jaryan]

This is great! Thanks.

…

On Tue, Jul 4, 2017 at 13:43 Joshua Ulrich ***@***.***> wrote: Great idea! Please use the current getSymbols "methods" as a template for getSymbols.av(). There is a getSymbols.skeleton(), but I'm not sure if it's up-to-date. Should we consider creating one (or more?) "alias" functions? Maybe getSymbols.alphavantage()? Great suggestion to use setDefaults() to store the user's API key. quantmod already uses jsonlite for a few things (e.g. getOptionChain()), so please take a look at that code to see how it uses jsonlite to manipulate JSON data. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#176 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA1swNwYXghpCdGeintmIRPJhos5TGBBks5sKofWgaJpZM4ONmTL> .

[pteetor]

Thanks, Josh and Jeff, for the encouragement and suggestions.

Yes, getSymbols.skeleton is incomplete, I now know. I studied getSymbols.yahoo and incorporated the logic related to getSymbolLookup and symbol-level overrides.

I checked getOptionChain, and I believe I'm using jsonlite in a way consistent with it, including the call to requireNamespace.

Finally, I did create an alias, getSymbols.alphavantage.

You can see the updated code in the feature branch of my fork.

[anozari]

It would be great if we can have adjusted prices outside of Yahoo which is not reliable.

It would be great if we can have a function with the same features of getSymbols using a different data source.
I tried to use getSymbols.alphavantage in place of getSymbols and it did not work. Is there a way to have it fully replaceable?

Thank you

[anozari]

getSymbol.av and its expected behavior would be ideal. I do not have teh skill set to do it. Just a user! My apologies

[braverock]

@anozari , getSymbols is a wrapper which calls methods for different data sources. by default, it has historically called Yahoo.
Of course, the breadth, history, and quality of yahoo data are now much worse.
For some things, src='google' will provide a near-equivalent of the old yahoo data. For other things, no free source exists.
The alphavantage code is brand new, and probably needs more testing to make it robust. Saying 'it did not work' does not provide enough information for anyone to help you diagnose your problem.

[anozari]

Understood. Thank you.

I was not complaining. I naively used getSymbols.alphavantage instead of getSymbols and gave me an error. I probably should have not done that.

Thank you for your suggestion.

[anozari]

I used alphavantage today into an R code. I have found no issues to report. It does look very promising. Thank you.

[anozari]

Brian had a very valid point about alphavantage. I have been using it for a week. I have found serious quality issues with their data. I am sure that over time they will get better and would be to get the kinks out. But, for now their data is not robust enough.

[pteetor]

@anozari Thanks for the testing and feedback. Can you summarize what problems you found with the data? I would like to replicate the problems if possible and study them.

[anozari]

They are working very hard to fix their problems. But their data quality does not appear to be robust. For a few days they had issues for SPY. They had prices for some holidays -- including Christmas. And, they just fixed it. They seem to have problems with some mutual funds. For example, if you look at PTTRX, you will see data on holidays -- Like April 30, 2017. These are simple things that I noticed while comparing long series against each other. I noticed that the dates are out of sync. There might be other issues. I was surprised that they did not know about these things. So, their quality assurance may not be as robust. I am just guessing

I think their processes are just new and over time things will only get better. They will get the kinks out. Good folks!

[pteetor]

Thanks for the detailed description of the problems. They are pretty serious. We can keep an eye on AlphaVantage and watch for improvements.

[team-alphavantage]

Thank you @pteetor for proposing the quantmod-alphavantage integration!

We are members of the product development team at Alpha Vantage Inc., and would like to share the following updates with the community:

• All the issues mentioned above by Mr. Nozari (anozari) were resolved shortly after they surfaced, and we have confirmed with the original poster on this matter.
• Alpha Vantage now has a dedicated quality assurance algorithm that operates on a 24/7 basis.
• In addition to JSON, we have also enabled support for the CSV (comma-separated-value) format (here is an example). The performance benchmarks for both the JSON and CSV formats are now below 500 milliseconds at 25%-75% percentile.

Many thanks again - we appreciate the continued feedback and support from the community.

[coryleuck]

Excellent post, @pteetor! Apparently there are a number of libraries built on top of Alpha Vantage on GitHub. Below is a review article on Alpha Vantage (written by a Python developer) that may provide some additional context on the Alpha Vantage API.

Source: RomelTorres/alpha_vantage#13

Alpha Vantage API Review from a Python Developer

The Experience

I was one of the "yahoo refugees" who stumbled upon Alpha Vantage via Google search. My first interactions with their website could be summarized by the good-old term WYSIWYG (what-you-see-is-what-you-get). Their documentation promised three things: (1) time series data with various resolutions, (2) over 50 technical signals, (3) sector data, and they kept their promise by providing demo URLs for all the said API functions.

They promised free API keys, and they also delivered. They promised "no promotional materials to your inbox," and indeed the only email I got from them so far was the announcement email for their new CSV feature.

This being said, there are a couple areas they could optimize upon.

• Be more specific about their business model. Being "100% free" is good, but it can also be a bit scary, especially for a yahoo refugee like me.
• Add multi-symbol support so that one can query multiple stocks with a single API call.
• Their FAQ recommends "~200 API calls per minute." It would be great if they can set a hard limit on the call volume to prevent client-side abuses in the form of ultra-high-frequency requests.

The Data

Of the thousands of US-based equities I have analyzed so far, their historical data and technical indicators seem to match other reputable data sources. Their intraday data is realtime up to the current minute, which is fine for my research purposes but may not satisfy users who want to beat the market with millisecond precision. Perhaps a premium feature for this down the road?

Their JSON output is easily readable and python-parsable. For the daily time series, however, I understand that their most recent data point is the cumulative information of the current trading day (updated realtime), but why is the first timestamp in YYYY-MM-DD HH:MM:SS format while all the others are in the normal YYYY-MM-DD format typical of the EOD data?

"Meta Data": {
"1. Information": "Daily Prices (open, high, low, close) and Volumes",
"2. Symbol": "MSFT",
"3. Last Refreshed": "2017-08-18 16:00:00",
"4. Output Size": "Compact",
"5. Time Zone": "US/Eastern"
},
"Time Series (Daily)": {
"2017-08-18 16:00:00": {
"1. open": "72.2700",
"2. high": "72.8400",
"3. low": "71.9300",
"4. close": "72.4900",
"5. volume": "18215276"
},
"2017-08-17": {
"1. open": "73.5800",
"2. high": "73.8700",
"3. low": "72.4000",
"4. close": "72.4000",
"5. volume": "21834250"
},

I would love to see a consistent YYYY-MM-DD format across all the timestamps. The "last refreshed" timestamp can be specified in Meta Data instead:

"Meta Data": {
"1. Information": "Daily Prices (open, high, low, close) and Volumes",
"2. Symbol": "MSFT",
"3. Last Refreshed": "2017-08-18 16:00:00",
"4. Output Size": "Compact",
"5. Time Zone": "US/Eastern"
},
"Time Series (Daily)": {
"2017-08-18": {
"1. open": "72.2700",
"2. high": "72.8400",
"3. low": "71.9300",
"4. close": "72.4900",
"5. volume": "18215276"
},
"2017-08-17": {
"1. open": "73.5800",
"2. high": "73.8700",
"3. low": "72.4000",
"4. close": "72.4000",
"5. volume": "21834250"
},

In addition to the data presentation aspects, below are couple other data-related proposals:

• Expand CSV support to all API functions. CSV is currently enabled only for the time series APIs.
• Make error messages more informative for debugging purposes.

In Summary

It is always a pleasure to have an API service that is well documented, platform/language-agnostic, and easily integratable. The fact that we have several third-party libraries built on top of Alpha Vantage on GitHub is in a sense testament to its developer-friendly nature. While there is still room for them to become a better version of themselves, I hope they thrive and stay true to the description on their home page - "driven by rigorous research, cutting edge technology, and a disciplined focus on democratizing access to data."