Pandas read_sas error: 'ascii' codec can't decode byte 0xd8 in position 0: ordinal not in range(128)

[randomgambit]

Hello everybody,

I am using Pandas 0.18 to open a sas7bdat dataset

I simply use:

df=pd.read_sas('P:/myfile.sas7bdat')

and I get the following error

    buf[0:text_block_size].rstrip(b"\x00 ").decode())

UnicodeDecodeError: 'ascii' codec can't decode byte 0xd8 in position 0: ordinal not in range(128)

If I use

import sys
reload(sys)
sys.setdefaultencoding("utf-8")

I get

UnicodeDecodeError: 'utf8' codec can't decode byte 0xd8 in position 0: invalid continuation byte

Other sas7bdat files in my folder are handled just fine by Pandas.

When I open the file in SAS I see that the column names are very long and span several lines, but otherwise the files look just fine.

There are not so many possible options in read_sas... what should I do? Is this a bug in read_sas?

Many thanks!

[TomAugspurger]

Are you able to share that file, or a similar file with non-senstitive data that raises the same error?

[randomgambit]

well this is the problem.. I cant. but I can do my best to run tests on my side, or do stuff in sas, or whatever you need to sort out the problem

[TomAugspurger]

You said the lines were long and span several lines. Can you make a dummy file with long names (just random strings like AAAAA.... might work) and a bit of fake data? (I don't have a copy of SAS).

Actually, this might be a dupe of #12659 Can you try reading the file linked there and see if the same error is raised?

[randomgambit]

when I try to read that file, I get TypeError: read() takes at most 1 argument (2 given)

[TomAugspurger]

Just making sure, did you click on the raw link to download?

https://github.com/pydata/pandas/raw/8857008178eb90b59ca67874507ef860b3faf88f/pandas/io/tests/sas/data/test17.sas7bdat

[randomgambit]

yes, I dowloaded the file test17.sas7bdat. This is not the error expected?

[randomgambit]

I checked the details of the file in SAS. Apparently it is encoded in latin 1 western.
So I tried read_sas('myfile.sas7bdat', encoding='latin-1') but I get the same error

-- ascii codec cant decode byte etc..

[TomAugspurger]

Sorry, I was mistaken about the error message. Looks like this is a different issue.

[randomgambit]

something strange is that even if I specify some encoding, I still get some error relative to the ascii codec. Can that be a cause of the error?

[randomgambit]

the encoding of my sas file is more precisely latin1 western ISO. Created in linux. (but I use pandas on windows)

[TomAugspurger]

Can you drop into the debugger after it raises the error? %debug if your in IPython. Then you can see what's going on.

The docstring says encoding is just for decoding string columns (the actual values), so perhaps it isn't being applied to decoding the column names.

[randomgambit]

aha! ok lemme try the debugger

[randomgambit]

> c:\users\me\appdata\local\continuum\anaconda2\lib\site-packages\pandas\io\sas\sas7bdat.py(529)_process_columntext_subheader()
    527         buf = self._read_bytes(offset, text_block_size)
    528         self.column_names_strings.append(
--> 529             buf[0:text_block_size].rstrip(b"\x00 ").decode())
    530 
    531         if len(self.column_names_strings) == 1:

[randomgambit]

this is what I get. then the debugger seems to wait for instructions