getSymbols: catch errors and continue processing

[joshuaulrich]

Wrap download.file() in a try() or tryCatch() in order to catch errors and provide more useful messages to the user.

For example, Google Finance does not provide historical data for the S&P 500 index, even though it's displayed on their website. Note there's no "Export/Download to spreadsheet" link under the historical chart like there are for symbols where getSymbols.google() works (e.g. General Electric).

R> getSymbols(".INX", src = "google")
Error in download.file(paste(google.URL, "q=", Symbols.name, "&startdate=",  : 
  cannot open URL 'http://finance.google.com/finance/historical?q=.INX&startdate=Jan+01,+2007&enddate=Jan+21,+2017&output=csv'
In addition: Warning message:
In download.file(paste(google.URL, "q=", Symbols.name, "&startdate=",  :
  cannot open URL 'http://finance.google.com/finance/historical?q=.INX&startdate=Jan+01,+2007&enddate=Jan+21,+2017&output=csv': HTTP status was '404 Not Found'

It would also be nice if a download.file() error did not completely stop getSymbols() when getSymbols() is called with multiple symbols.

[joshuaulrich]

Although my initial thought was to put the try/catch in each getSymbols "method", it might be better to put into getSymbols proper, where errors from any future or custom "methods" would be caught. That might be difficult however, because getSymbols currently groups symbols by source and calls getSymbols.[source] one time for each group. So none of the symbols for a given source would have data if any one of the tickers in the getSymbols.[source] call fails.

It might be simple to just loop over the symbols in whatever order they're passed in, and call each method on the symbol. Though I'm not sure the reason they're pulled in groups by source, and therefore a bit reluctant to change that.

[helgasoft]

Was going to open a new entry, but saw this one which is (maybe) the enhancement I was thinking about.
When multiple symbols are requested, getSymbols stops on the first encountered problem, like this:
> getSymbols(c('F','delisted','GE'), from='2017-02-22')
Warning: delisted download failed; trying again.
Error: delisted download failed after two attempts. Error message:
HTTP error 404.
Wouldn't it be better if it continues with all symbols and spits out the error(s) at the end ? Companies are delisted often and we do not always know exactly when that happened. Finishing the list and issuing a message like 'X, Y, Z failed' will help us users update our records and not ask for a delisted symbol again.

[joshuaulrich]

@helgasoft Yes, this issue is to implement what you describe. I started working on it, then ran into the issue I described in my first comment.

[helgasoft]

FYI - this is working well, skipping delisted/misspelled symbols:

library(tidyquant)
mylist <- c("FB", "AMZN", "dummy", "GOOG") %>%
	tq_get(get="stock.prices", from="2017-01-01", to="2017-05-01")

Warning message:
x = 'dummy', get = 'stock.prices': Error: dummy download failed after two attempts. Error message:
HTTP error 404.
Removing dummy.

[khanh-ld]

@helgasoft you may use this to work around this in the mean time:
library(quantmod)
e <- new.env()

lapply(ticker_list, function(x){
try(
getSymbols(
x,
env=WoW),
silent=TRUE)
})

[joshuaulrich]

One consequence of this change is that getSymbols() calls for a single ticker that used to error will now only warn and possibly error somewhere after the download (e.g. when trying to read the response, return a non-existent object, etc). Those things need to be tested and addressed before this is merged.