Regression: pandas 0.10 fails to read an excel file that pandas 0.9 can read

[lexual]

I have an excel file that reads fine in pandas 0.9, fails to read in pandas 0.10

  fname = sys.argv[1]                                                            
  xlsx = pd.ExcelFile(fname)                                                     
  sheet = xlsx.sheet_names[0]                                                    

  # this works in pandas 0.9, fails in pandas 0.10.0                             
 df = xlsx.parse(sheet, index_col=[1, 2])                                                                                                                                                                                                                                                                              

Triggers this error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/home/vagrant/.virtualenvs/foo/local/lib/python2.7/site-packages/IPython/utils/py3compat.pyc in execfile(fname, *where)
    176             else:
    177                 filename = fname
--> 178             __builtin__.execfile(filename, *where)

/tmp/X/pandas_error_test.py in <module>()
     15 
     16 if __name__ == '__main__':
---> 17     main()

/tmp/X/pandas_error_test.py in main(files)
     11 
     12     # this works in pandas 0.9, fails in pandas 0.10.0
---> 13     df = xlsx.parse(sheet, index_col=[1, 2])
     14 
     15 

/home/vagrant/.virtualenvs/foo/local/lib/python2.7/site-packages/pandas/io/parsers.pyc in parse(self, sheetname, header, skiprows, skip_footer, index_col, parse_cols, parse_dates, date_parser, na_values, thousands, chunksize, **kwds)
   1866                                      thousands=thousands,
   1867                                      chunksize=chunksize,
-> 1868                                      skip_footer=skip_footer)
   1869 
   1870     def _should_parse(self, i, parse_cols):

/home/vagrant/.virtualenvs/foo/local/lib/python2.7/site-packages/pandas/io/parsers.pyc in _parse_xlsx(self, sheetname, header, skiprows, skip_footer, index_col, has_index_names, parse_cols, parse_dates, date_parser, na_values, thousands, chunksize)
   1934                             chunksize=chunksize)
   1935 
-> 1936         return parser.read()
   1937 
   1938     def _parse_xls(self, sheetname, header=0, skiprows=None,

/home/vagrant/.virtualenvs/foo/local/lib/python2.7/site-packages/pandas/io/parsers.pyc in read(self, nrows)
    622             #     self._engine.set_error_bad_lines(False)
    623 
--> 624         ret = self._engine.read(nrows)
    625 
    626         if self.options.get('as_recarray'):

/home/vagrant/.virtualenvs/foo/local/lib/python2.7/site-packages/pandas/io/parsers.pyc in read(self, rows)
   1245             content = content[1:]
   1246 
-> 1247         alldata = self._rows_to_cols(content)
   1248         data = self._exclude_implicit_index(alldata)
   1249 

/home/vagrant/.virtualenvs/foo/local/lib/python2.7/site-packages/pandas/io/parsers.pyc in _rows_to_cols(self, content)
   1461             msg = ('Expected %d fields in line %d, saw %d' %
   1462                    (col_len, row_num + 1, zip_len))
-> 1463             raise ValueError(msg)
   1464 
   1465         return zipped_content

ValueError: Expected 10 fields in line 2, saw 11

I suspect a similar error happened when building the docs on the pandas site. See:

http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#v0-10-0-december-17-2012

And search for "expected", and you'll see a similar error is being triggered. See screenshot for details.

I haven't attached the file, as it contains a client's data.

[ghost]

If you can provide an excel file that repros the problem, I'll take a look.
Maybe you can replace the data with random data, or perhaps narrow it
down to a couple of offending lines?

[wesm]

Pushing this issue to 0.10.2 until a reproduction is found. Docs issue was just a bug in the code

[lexual]

Is it possible to provide a file that reproduces to one of the devs directly offline? The file I have does have a client's data, which while isn't particularly sensitive, would be best kept out of public domain.

[wesm]

No problem. send it to my e-mail address (found in my profile)

[wesm]

That it parsed before was basically luck and not by design. I can't fix this in time for 0.10.1, if it even can be fixed

[lexual]

FYI, this looks to be specific to using index_col.

i.e.

# This works
df = xlsx.parse(sheet)          
# this fals
df = xlsx.parse(sheet, index_col=[1, 2])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                

[lexual]

Calling without index_col is allowing me to parse the file now. Thanks again for looking into this one for Wes.