Issues with the data reader fetching yahoo finance

[Crowbeezy]

Apologies first issue/comment on GitHub. I will review proper protocol. Please correct me if this is not the correct place to put this.

---

RemoteDataError Traceback (most recent call last)
in ()
4 end = dt.datetime(2017, 5, 8)
5
----> 6 INPX = data.DataReader(INPX ,'yahoo', start, end)
7
8 #Convert Volume from Int to Float

C:\Users\randomname\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas_datareader\data.py in DataReader(name, data_source, start, end, retry_count, pause, session)
92 adjust_price=False, chunksize=25,
93 retry_count=retry_count, pause=pause,
---> 94 session=session).read()
95
96 elif data_source == "yahoo-actions":

C:\Users\randomname\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas_datareader\yahoo\daily.py in read(self)
75 def read(self):
76 """ read one data from specified URL """
---> 77 df = super(YahooDailyReader, self).read()
78 if self.ret_index:
79 df['Ret_Index'] = _calc_return_index(df['Adj Close'])

C:\Users\randomname\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas_datareader\base.py in read(self)
176 df = self._dl_mult_symbols(self.symbols.index)
177 else:
--> 178 df = self._dl_mult_symbols(self.symbols)
179 return df
180

C:\Users\randomname\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas_datareader\base.py in _dl_mult_symbols(self, symbols)
195 if len(passed) == 0:
196 msg = "No data fetched using {0!r}"
--> 197 raise RemoteDataError(msg.format(self.class.name))
198 try:
199 if len(stocks) > 0 and len(failed) > 0 and len(passed) > 0:

RemoteDataError: No data fetched using 'YahooDailyReader'

[rgkimball]

Can you provide a sample that replicates your issue? This works for me:

start = datetime(2016, 12, 31)
end = datetime.now()
INPX = data.DataReader('INPX', 'yahoo', start, end)

[Crowbeezy]

On Thu, May 11, 2017 at 9:49 PM, Rob Kimball ***@***.***> wrote: start = datetime(2016, 12, 31) end = datetime.now() INPX = data.DataReader('INPX', 'yahoo', start, end)

I'm not sure what the problem was. Perhaps using 'end = datetime.today()'? You're code worked. Thank you for your help!

[benpillet]

From my requirements.txt:

pandas-datareader==0.4.0
pandas==0.20.1

and in python shell:

from datetime import *
import pandas_datareader.data as data
start = datetime(2016, 12, 31)
end = datetime.now()
INPX = data.DataReader('INPX', 'yahoo', start, end)

with error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "./venv3.5/lib/python3.5/site-packages/pandas_datareader/data.py", line 117, in DataReader
    session=session).read()
  File "./venv3.5/lib/python3.5/site-packages/pandas_datareader/yahoo/daily.py", line 77, in read
    df = super(YahooDailyReader, self).read()
  File "./venv3.5/lib/python3.5/site-packages/pandas_datareader/base.py", line 157, in read
    params=self._get_params(self.symbols))
  File "./venv3.5/lib/python3.5/site-packages/pandas_datareader/base.py", line 74, in _read_one_data
    out = self._read_url_as_StringIO(url, params=params)
  File "./venv3.5/lib/python3.5/site-packages/pandas_datareader/base.py", line 85, in _read_url_as_StringIO
    response = self._get_response(url, params=params)
  File "./venv3.5/lib/python3.5/site-packages/pandas_datareader/base.py", line 120, in _get_response
    raise RemoteDataError('Unable to read URL: {0}'.format(url))
pandas_datareader._utils.RemoteDataError: Unable to read URL: http://ichart.finance.yahoo.com/table.csv?f=2017&ignore=.csv&b=31&c=2016&g=d&a=11&d=4&s=INPX&e=16

I have a feeling yahoo updated their endpoint to be something else. I get a 502 when I try curl too. The link at https://finance.yahoo.com/quote/SPY/history?p=SPY points to https://query1.finance.yahoo.com/v7/finance/download/SPY?period1=1492372898&period2=1494964898&interval=1d&events=history&crumb=MLOX17FWABw

[benpillet]

Looks like there's also a cookie that needs to be sent in order to avoid a 401 Unauthorized. https://www.elitetrader.com/et/threads/yahoo-historical-data-did-they-change-the-url-recently.309554/

[rgkimball]

Not sure if the icharts failure is a temporary problem, but I submitted a WIP PR (above) to replace the request structure. Even if icharts does come back online, may be a good idea to implement a backup.

[IvanTrendafilov]

I don't have the time to fix this in the library, but, essentially, there is another API endpoint that one can use. It's https://query1.finance.yahoo.com. But it requires a matching cookie and crumb to use it. I wrote a little PhantomJS script to get it, whilst I was working on it: https://github.com/IvanTrendafilov/YahooFinanceAPITokens

It can be useful to someone who needs to create a URL that they want to query automatically.

You can also get a valid cookie / crumb combination from the Chrome dev tools in the Network tab.

[IvanTrendafilov]

They've confirmed icharts isn't coming back, so @rgkimball's patch should certainly go in.

https://forums.yahoo.net/t5/Yahoo-Finance-help/Is-Yahoo-Finance-API-broken/td-p/250503/page/3

[bkcollection]

@rgkimball @IvanTrendafilov , can the fix be pip install upgrade for the ease for beginner?

[Franlodo]

Yahoo has change the URL, and the way the use date. Now date are Unixtime.
For example to get historic cvs from AAPL :
https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1492510098&period2=1495102098&interval=1d&events=history&crumb=ydacXMYhzrn

period1 or period2 is date in (unixtime = (Human time - 25568) * 86400) but you must check your timezone, for example my place is Europe, then I have UTC+2 and I must sustract 7200 seconds. So my formula is (Human time - 25568) * 86400) -7200; where Human time is the time (d/mm/yyyy), 25568 is the number of days from 01/01/1900 till 01/01/1970 (This is because i do it in Excel and this is the minimun date), 86400 are the seconds in a day and 7200 are the number of seconds in my 2 hours difference with UTC

Interval is day, week or month
Events history is historic data prices, div|split&filter=split, for splits and div|split&filter=div for dividens
Crumb is the cookie ... I don't really know how it works, but I have with the same since monday.

I'm using this for update my data in Excel and it works and now I don't need to wait until morning to get the historical data because it's available at less an hour after the market close (I´m talking about american markets)

I apologyze for not to be fluent in english.

I hope this help

[bkcollection]

@Franlodo DO you try do download using the new link to download like 1000 stocks, will it get blocked? The old API seems has no limitation but I am curious if the new one still allow that. Hope you can try on it to validate

[Franlodo]

The link is to get the csv file in the web, it must run for 2 or for 2000; In my Excel file I have nearly 200 and run properly.

You can get 1000 of csv files and import from pandas, it will be the same made it saving files or "in the air"
I had post here because the error reported for pandas-datareader was the url

Anyway, I will try and comment.