pandas_datareader failing to get all options for '^ndx' from yahoo

[vishalapte]

The code below works for over 500 symbols but doesn't work for ^ndx.

In [1]: from pandas_datareader import data as pddata

In [2]: foo = pddata.Options('^ndx', 'yahoo')

In [3]: foo.expiry_dates
Out[3]: 
[datetime.date(2016, 3, 18),
 datetime.date(2016, 3, 24),
 datetime.date(2016, 4, 1),
 datetime.date(2016, 4, 8),
 datetime.date(2016, 4, 15),
 datetime.date(2016, 4, 22),
 datetime.date(2016, 4, 29),
 datetime.date(2016, 5, 20),
 datetime.date(2016, 6, 17),
 datetime.date(2016, 9, 16),
 datetime.date(2016, 12, 16),
 datetime.date(2017, 6, 16),
 datetime.date(2017, 12, 15),
 datetime.date(2018, 12, 21)]

In [4]: for d in foo.expiry_dates:
    print(d)
    foo.get_options_data(expiry=d)
   ...:     
2016-03-18
2016-03-24
2016-04-01
2016-04-08
2016-04-15
2016-04-22
2016-04-29
2016-05-20
2016-06-17
2016-09-16
2016-12-16

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-5-dad98abfb543> in <module>()
      1 for d in foo.expiry_dates:
      2     print d
----> 3     foo.get_options_data(expiry=d)
      4 

/Users/v/virtenvs/ns/lib/python2.7/site-packages/pandas_datareader/yahoo/options.pyc in get_options_data(self, month, year, expiry)
    146         return concat([f(month, year, expiry)
    147                        for f in (self.get_put_data,
--> 148                                  self.get_call_data)]).sortlevel()
    149 
    150     def _get_option_frames_from_yahoo(self, expiry):

/Users/v/virtenvs/ns/lib/python2.7/site-packages/pandas_datareader/yahoo/options.pyc in get_put_data(self, month, year, expiry)
    360         """
    361         expiry = self._try_parse_dates(year, month, expiry)
--> 362         return self._get_data_in_date_range(expiry, put=True, call=False)
    363 
    364     def get_near_stock_price(self, above_below=2, call=True, put=False,

/Users/v/virtenvs/ns/lib/python2.7/site-packages/pandas_datareader/yahoo/options.pyc in _get_data_in_date_range(self, dates, call, put)
    606                     frame = getattr(self, nam)
    607                 except AttributeError:
--> 608                     frame = self._get_option_data(expiry=expiry_date, name=name)
    609                 data.append(frame)
    610 

/Users/v/virtenvs/ns/lib/python2.7/site-packages/pandas_datareader/yahoo/options.pyc in _get_option_data(self, expiry, name)
    229             frames = getattr(self, frame_name)
    230         except AttributeError:
--> 231             frames = self._get_option_frames_from_yahoo(expiry)
    232 
    233         option_data = frames[name]

/Users/v/virtenvs/ns/lib/python2.7/site-packages/pandas_datareader/yahoo/options.pyc in _get_option_frames_from_yahoo(self, expiry)
    150     def _get_option_frames_from_yahoo(self, expiry):
    151         url = self._yahoo_url_from_expiry(expiry)
--> 152         option_frames = self._option_frames_from_url(url)
    153         frame_name = '_frames' + self._expiry_to_string(expiry)
    154         setattr(self, frame_name, option_frames)

/Users/v/virtenvs/ns/lib/python2.7/site-packages/pandas_datareader/yahoo/options.pyc in _option_frames_from_url(self, url)
    173 
    174         root = self._parse_url(url)
--> 175         calls = root.xpath('//*[@id="optionsCallsTable"]/div[2]/div/table')[0]
    176         puts = root.xpath('//*[@id="optionsPutsTable"]/div[2]/div/table')[0]
    177 

IndexError: list index out of range

In [9]: import pandas_datareader

In [10]: pandas_datareader.version
Out[10]: '0.2.1'

[femtotrader]

Here is my traceback (which is slightly different than yours probably as I'm using Python 3)

In [15]: for d in foo.expiry_dates:
        print(d)
        foo.get_options_data(expiry=d)
   ....:
2016-03-24
2016-04-01
2016-04-08
2016-04-15
2016-04-22
2016-04-29
2016-05-20
2016-06-17
2016-09-16
2016-12-16
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in _get_data_in_date_range(self, dates, call, put)
    610                 try:  # Try to access on the instance
--> 611                     frame = getattr(self, nam)
    612                 except AttributeError:

AttributeError: 'Options' object has no attribute 'puts161216'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in _get_option_data(self, expiry, name)
    233         try:
--> 234             frames = getattr(self, frame_name)
    235         except AttributeError:

AttributeError: 'Options' object has no attribute '_frames161216'

During handling of the above exception, another exception occurred:

IndexError                                Traceback (most recent call last)
//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in _option_frames_from_url(self, url)
    177         try:
--> 178             calls = root.xpath('//*[@id="optionsCallsTable"]/div[2]/div/table')[0]
    179             puts = root.xpath('//*[@id="optionsPutsTable"]/div[2]/div/table')[0]

IndexError: list index out of range

During handling of the above exception, another exception occurred:

RemoteDataError                           Traceback (most recent call last)
<ipython-input-15-42e2169d1e3b> in <module>()
      1 for d in foo.expiry_dates:
      2         print(d)
----> 3         foo.get_options_data(expiry=d)
      4

//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in get_options_data(self, month, year, expiry)
    148         return concat([f(month, year, expiry)
    149                        for f in (self.get_put_data,
--> 150                                  self.get_call_data)]).sortlevel()
    151
    152     def _get_option_frames_from_yahoo(self, expiry):

//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in <listcomp>(.0)
    147         """
    148         return concat([f(month, year, expiry)
--> 149                        for f in (self.get_put_data,
    150                                  self.get_call_data)]).sortlevel()
    151

//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in get_put_data(self, month, year, expiry)
    365         """
    366         expiry = self._try_parse_dates(year, month, expiry)
--> 367         return self._get_data_in_date_range(expiry, put=True, call=False)
    368
    369     def get_near_stock_price(self, above_below=2, call=True, put=False,

//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in _get_data_in_date_range(self, dates, call, put)
    611                     frame = getattr(self, nam)
    612                 except AttributeError:
--> 613                     frame = self._get_option_data(expiry=expiry_date, name=name)
    614                 data.append(frame)
    615

//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in _get_option_data(self, expiry, name)
    234             frames = getattr(self, frame_name)
    235         except AttributeError:
--> 236             frames = self._get_option_frames_from_yahoo(expiry)
    237
    238         option_data = frames[name]

//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in _get_option_frames_from_yahoo(self, expiry)
    152     def _get_option_frames_from_yahoo(self, expiry):
    153         url = self._yahoo_url_from_expiry(expiry)
--> 154         option_frames = self._option_frames_from_url(url)
    155         frame_name = '_frames' + self._expiry_to_string(expiry)
    156         setattr(self, frame_name, option_frames)

//anaconda/lib/python3.5/site-packages/pandas_datareader-0.2.1-py3.5.egg/pandas_datareader/yahoo/options.py in _option_frames_from_url(self, url)
    179             puts = root.xpath('//*[@id="optionsPutsTable"]/div[2]/div/table')[0]
    180         except IndexError:
--> 181             raise RemoteDataError('Option Table not available for url: %s' % url)
    182
    183         if not hasattr(self, 'underlying_price'):

RemoteDataError: Option Table not available for url: http://finance.yahoo.com/q/op?s=^NDX&date=1481846400

Such traceback can occur by doing

foo.get_options_data(expiry=datetime.date(2016, 12, 16))

but I' m getting

In [22]: foo.get_options_data(expiry=datetime.date(2017, 12, 15))
Out[22]:
                                             Last     Bid     Ask  Chg  \
Strike Expiry     Type Symbol
1980   2015-12-17 call NDX171215C01980000  2294.9  2369.0  2384.5    0
                  put  NDX171215P01980000    28.1    20.0    23.5    0
2000   2015-12-17 call NDX171215C02000000  2343.8  2350.0  2365.5    0
                  put  NDX171215P02000000     0.0    26.0    32.0    0
2100   2015-12-17 call NDX171215C02100000  2182.5  2256.1  2272.5    0
...                                           ...     ...     ...  ...
6950   2015-12-17 put  NDX171215P06950000     0.0  2629.4  2650.0    0
6975   2015-12-17 call NDX171215C06975000     0.0     0.0     5.0    0
                  put  NDX171215P06975000  2260.9  2543.7  2563.0    0
7000   2015-12-17 call NDX171215C07000000     0.0     0.0     5.0    0
                  put  NDX171215P07000000     0.0  2677.8  2699.0    0

                                           PctChg  Vol  Open_Int      IV Root  \
Strike Expiry     Type Symbol
1980   2015-12-17 call NDX171215C01980000       0    6        10  0.0000  NDX
                  put  NDX171215P01980000       0    0         1  0.3526  NDX
2000   2015-12-17 call NDX171215C02000000       0    1         9  0.0000  NDX
                  put  NDX171215P02000000       0    0         0  0.3710  NDX
2100   2015-12-17 call NDX171215C02100000       0    1        29  0.0000  NDX
...                                           ...  ...       ...     ...  ...
6950   2015-12-17 put  NDX171215P06950000       0    0         0  0.2837  NDX
6975   2015-12-17 call NDX171215C06975000       0    0         0  0.1551  NDX
                  put  NDX171215P06975000       0    0         1  0.0994  NDX
7000   2015-12-17 call NDX171215C07000000       0    0         0  0.1561  NDX
                  put  NDX171215P07000000       0    0         0  0.2859  NDX

                                          IsNonstandard Underlying  \
Strike Expiry     Type Symbol
1980   2015-12-17 call NDX171215C01980000          True       ^NDX
                  put  NDX171215P01980000          True       ^NDX
2000   2015-12-17 call NDX171215C02000000          True       ^NDX
                  put  NDX171215P02000000          True       ^NDX
2100   2015-12-17 call NDX171215C02100000          True       ^NDX
...                                                 ...        ...
6950   2015-12-17 put  NDX171215P06950000          True       ^NDX
6975   2015-12-17 call NDX171215C06975000          True       ^NDX
                  put  NDX171215P06975000          True       ^NDX
7000   2015-12-17 call NDX171215C07000000          True       ^NDX
                  put  NDX171215P07000000          True       ^NDX

                                           Underlying_Price  \
Strike Expiry     Type Symbol
1980   2015-12-17 call NDX171215C01980000           4410.83
                  put  NDX171215P01980000           4410.83
2000   2015-12-17 call NDX171215C02000000           4410.83
                  put  NDX171215P02000000           4410.83
2100   2015-12-17 call NDX171215C02100000           4410.83
...                                                     ...
6950   2015-12-17 put  NDX171215P06950000           4410.83
6975   2015-12-17 call NDX171215C06975000           4410.83
                  put  NDX171215P06975000           4410.83
7000   2015-12-17 call NDX171215C07000000           4410.83
                  put  NDX171215P07000000           4410.83

                                                   Quote_Time
Strike Expiry     Type Symbol
1980   2015-12-17 call NDX171215C01980000 2016-03-19 17:15:00
                  put  NDX171215P01980000 2016-03-19 17:15:00
2000   2015-12-17 call NDX171215C02000000 2016-03-19 17:15:00
                  put  NDX171215P02000000 2016-03-19 17:15:00
2100   2015-12-17 call NDX171215C02100000 2016-03-19 17:15:00
...                                                       ...
6950   2015-12-17 put  NDX171215P06950000 2016-03-19 17:15:00
6975   2015-12-17 call NDX171215C06975000 2016-03-19 17:15:00
                  put  NDX171215P06975000 2016-03-19 17:15:00
7000   2015-12-17 call NDX171215C07000000 2016-03-19 17:15:00
                  put  NDX171215P07000000 2016-03-19 17:15:00

[758 rows x 13 columns]

I notice that for 2016/12/16
http://finance.yahoo.com/q/op?s=^NDX&date=1481846400
doesn't display any data

but for 2017/12/15
http://finance.yahoo.com/q/op?s=^NDX&date=1513296000
I'm getting data

maybe in such a case (of empty page) we might return an empty DataFrame like

pd.DataFrame(columns=['Last', 'Bid', 'Ask', 'Chg', 'PctChg', 'Vol', 'Open_Int', 'IV', 'Root',
   ....:        'IsNonstandard', 'Underlying', 'Underlying_Price', 'Quote_Time'])

an other approach (on your side) might be to use a try/except like:

from pandas_datareader.yahoo.options import RemoteDataError

for d in foo.expiry_dates:
    print(d)
    try:
        foo.get_options_data(expiry=d)
    except RemoteDataError:
        print("Can't get options data for %s" % d)

[davidastephens]

This looks to work now after #244. Please re-open if still broken.