Broken options class for pandas.io.data - Did Yahoo just change their options site?

[dscheste]

Hello,
Yahoo seems to have changed their site design as I no longer can pull any options data with panda.

Does anyone else experience this?

Here's what's called:

#!/usr/bin/python
import sys, getopt
import pandas as pd
import numpy as np
from pandas.io.data import DataFrame
from pandas.io.data import Options
import h5py as h5
from pandas_datareader import data, wb
import datetime
import time
ticker = str(sys.argv[1])
new_data=pd.DataFrame()
option = Options(ticker, 'yahoo')
data = option.get_all_data()
new_data = new_data.append(data)
new_data.to_csv('cache/'+ticker)

And here's what I am getting:

Traceback (most recent call last):
File "db1.py", line 18, in 
data = option.get_all_data()
File "/usr/local/lib/python2.7/dist-packages/pandas/io/data.py", line 1134, in get_all_data
expiry_dates = self.expiry_dates
File "/usr/local/lib/python2.7/dist-packages/pandas/io/data.py", line 1165, in expiry_dates
expiry_dates, _ = self._get_expiry_dates_and_links()
File "/usr/local/lib/python2.7/dist-packages/pandas/io/data.py", line 1191, in _get_expiry_dates_and_links
raise RemoteDataError('Data not available')
pandas.io.data.RemoteDataError: Data not available

output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-24-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_CA.UTF-8

pandas: 0.18.1
nose: 1.3.7
pip: 8.1.1
setuptools: 20.7.0
Cython: 0.23.4
numpy: 1.11.0
scipy: 0.17.0
statsmodels: 0.6.1
xarray: None
IPython: None
sphinx: None
patsy: 0.4.1
dateutil: 2.4.2
pytz: 2014.10
blosc: None
bottleneck: None
tables: 3.2.2
numexpr: 2.4.3
matplotlib: 1.5.1
openpyxl: 2.3.0
xlrd: 0.9.4
xlwt: 0.7.5
xlsxwriter: None
lxml: 3.5.0
bs4: 4.4.1
html5lib: 0.999
httplib2: 0.9.1
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
boto: None
pandas_datareader: 0.2.0

[Bodisatva]

Same issue. Did yahoo proceeded to a complete overhaul of there site ? I hope not

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pandas_datareader/yahoo/options.py", line 619, in expiry_dates
    expiry_dates = self._expiry_dates
AttributeError: 'Options' object has no attribute '_expiry_dates'

OS X10.11.5
Python v3.5.0:374f501f4567, Sep 12 2015

[dscheste]

Bump.

[nborggren]

ended up here as my codes have broken as well (see error below), I'll fork it and try to fix it, but I presume someone else can fix it better/faster...

from pandas.io.data import Options
Options('AAPL','yahoo')
<pandas.io.data.Options object at 0x1051c7c90>
opt.get_all_data()
Traceback (most recent call last):
File "", line 1, in
File "/Users/nborggren/anaconda/lib/python2.7/site-packages/pandas/io/data.py", line 1115, in get_all_data
expiry_dates = self.expiry_dates
File "/Users/nborggren/anaconda/lib/python2.7/site-packages/pandas/io/data.py", line 1146, in expiry_dates
expiry_dates, _ = self._get_expiry_dates_and_links()
File "/Users/nborggren/anaconda/lib/python2.7/site-packages/pandas/io/data.py", line 1172, in _get_expiry_dates_and_links
raise RemoteDataError('Data not available')
pandas.io.data.RemoteDataError: Data not available

[Slimdog588]

how does this get fixed? by whom? and in how long? Any info is appreciated.

ALL my shit is busted..lol

[emican86]

I was hoping to contribute to the fix, but yahoo is not returning a neatly formatted json object ( from what I can see) The http response is different and I have not made sense of the changes yet.

[nborggren]

I noticed the yahoo page is a mess too (from a scrapers perspective), I moved over to google for a quick fix. Below returns a rather clean json file that might be a good starting point, but I haven't put it into a pandas data frame yet. A lot of the symbols I was looking up in yahoo can't be found in google though.

import urllib2 as ul
dat = ul.urlopen('https://www.google.com/finance/option_chain?q=AAPL&output=json').read()

[Slimdog588]

@nborggren Do you have any code that will parse this? Thanks

[MattRijk]

Same issue. All my options data scripts return Data not available. They were all working when i last checked 7-6-2016

[nborggren]

I'm having a little trouble parsing this, but here is what I have so far. I needed to reference

https://github.com/femtotrader/pandas_datareaders_unofficial/blob/master/pandas_datareaders_unofficial/datareaders/google_finance_options.py

and

http://stackoverflow.com/questions/21104592/json-to-pandas-dataframe

I have a pandas dataframe (two actually one for calls and one for puts) but it will take some massaging to put it in the form of yahoos data before this catastrophe. Perhaps someone else can try too. Below is what I have so far:

from google_fix import *
calls, puts = Options('AAPL')
puts.head()

shows a dataframe that looks like the image.

below is the google_fix code

import token, tokenize
from six.moves import cStringIO as StringIO
import json
from pandas.io.json import json_normalize
import urllib2 as ul

def fix_lazy_json(in_text):
    """
    Handle lazy JSON - to fix expecting property name
    this function fixes the json output from google
    http://stackoverflow.com/questions/4033633/handling-lazy-json-in-python-expecting-property-name
    """
    tokengen = tokenize.generate_tokens(StringIO(in_text).readline)
    result = []
    for tokid, tokval, _, _, _ in tokengen:
        # fix unquoted strings
        if (tokid == token.NAME):
            if tokval not in ['true', 'false', 'null', '-Infinity', 'Infinity', 'NaN']:
                tokid = token.STRING
                tokval = u'"%s"' % tokval

        # fix single-quoted strings
        elif (tokid == token.STRING):
            if tokval.startswith ("'"):
                tokval = u'"%s"' % tokval[1:-1].replace ('"', '\\"')

        # remove invalid commas
        elif (tokid == token.OP) and ((tokval == '}') or (tokval == ']')):
            if (len(result) > 0) and (result[-1][1] == ','):
                result.pop()

        # fix single-quoted strings
        elif (tokid == token.STRING):
            if tokval.startswith ("'"):
                tokval = u'"%s"' % tokval[1:-1].replace ('"', '\\"')

        result.append((tokid, tokval))

    return tokenize.untokenize(result)

def Options(symbol):
    dat = ul.urlopen('https://www.google.com/finance/option_chain?q='+symbol+'&output=json').read()
    dat = json.loads(fix_lazy_json(dat))
    puts = json_normalize(dat['puts'])
    calls = json_normalize(dat['calls'])
    return calls, puts

[Slimdog588]

@nborggren Does it work for you? I tried it and it gives me this:

line 556, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found

The only thing I added was Options('AAPL') at the end of the code in order to run it. I'm relatively new to python (and coding), if it works for you, any insight would be appreciated. Thanks

[nborggren]

@Slimdog588 It looks like I introduced a typo when github wrapped the urlopen line. Make sure in the file you copied that the ?q='+symbol+... part in the Options function appears right next to the option_chain part. See if that helps.

[Slimdog588]

@nborggren thank you sir!

[femtotrader]

Thanks @nborggren for citing my code.

You might use requests http://docs.python-requests.org/ because it's currently use by pandas-datareader and so it will be easier to add your code here (and have Python 2 / 3 compatibility)

import requests
url = "https://www.google.com/finance/option_chain"
r = requests.get(url, params={"q": "AAPL", "output": "json"})
content_json = response.text
...