Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modest performance, address #12647 #12656

Closed
wants to merge 19 commits into from
Closed

Modest performance, address #12647 #12656

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
a7df841
Modest performance, address #12647
kshedden Mar 17, 2016
7d91d51
Add test data set from raderaj
kshedden Mar 19, 2016
ea2339f
Use encoding when reading column headers
kshedden Mar 19, 2016
bdc9a06
More cython for performance, refactored constants
kshedden Mar 19, 2016
3bd1b35
Move more code to cython
kshedden Mar 19, 2016
23bdf7a
Decouple data decoding and decoding e.g. of column names
kshedden Mar 19, 2016
dc330c5
Add two missing alignment constants
kshedden Mar 19, 2016
7e156b7
Working on cython issues
kshedden Apr 10, 2016
3ef626e
Working on cython issues
kshedden Apr 10, 2016
11c2f31
Further cythonization
kshedden Apr 10, 2016
873a877
Added option to not decode header text
kshedden Apr 10, 2016
c26d22b
added to whatsnew
kshedden Apr 11, 2016
1af73b3
Further encoding work
kshedden Apr 11, 2016
8b4b96d
pep8 cleanup
kshedden Apr 11, 2016
ea87a7f
Fix encoding handling bug for py2
kshedden Apr 21, 2016
b7de358
flake8 fixes
kshedden Apr 21, 2016
fe4731b
Integrate jreback's cython improvements
kshedden Apr 22, 2016
af085f7
Add one more type
kshedden Apr 22, 2016
b3024ed
Add missing test data files
kshedden Apr 22, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use encoding when reading column headers
  • Loading branch information
@kshedden
kshedden committed Apr 21, 2016
commit ea2339fc59ca770636ec0a367a82f82024beaa5c
5 changes: 3 additions & 2 deletions pandas/io/sas/sas7bdat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,8 @@

_compression_literals = [_rle_compression, _rdc_compression]

# Incomplete list of encodings
# Incomplete list of encodings, using SAS nomenclature:
# http://support.sas.com/documentation/cdl/en/nlsref/61893/HTML/default/viewer.htm#a002607278.htm
_encoding_names = {29: "latin1", 20: "utf-8", 33: "cyrillic", 60: "wlatin2",
61: "wcyrillic", 62: "wlatin1", 90: "ebcdic870"}

Expand Down Expand Up @@ -526,7 +527,7 @@ def _process_columntext_subheader(self, offset, length):

buf = self._read_bytes(offset, text_block_size)
self.column_names_strings.append(
buf[0:text_block_size].rstrip(b"\x00 ").decode())
buf[0:text_block_size].rstrip(b"\x00 ").decode(self.encoding))

if len(self.column_names_strings) == 1:
column_name = self.column_names_strings[0]
Expand Down
37 changes: 37 additions & 0 deletions pandas/io/tests/sas/data/test_12659.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
yearmonth,useGpCo,useGpVi,useSpec,useUrge,caseGpCo,caseGpVi,caseSpec,caseUrge,expendGpCo,expendGpVi,expendSpec,expendUrge
201401,11,12,13,14,15,16,17,18,19,20,21,22
201402,11,12,13,14,15,16,17,18,19,20,21,22
201403,11,12,13,14,15,16,17,18,19,20,21,22
201404,11,12,13,14,15,16,17,18,19,20,21,22
201405,11,12,13,14,15,16,17,18,19,20,21,22
201406,11,12,13,14,15,16,17,18,19,20,21,22
201407,11,12,13,14,15,16,17,18,19,20,21,22
201408,11,12,13,14,15,16,17,18,19,20,21,22
201409,11,12,13,14,15,16,17,18,19,20,21,22
201410,11,12,13,14,15,16,17,18,19,20,21,22
201411,11,12,13,14,15,16,17,18,19,20,21,22
201412,11,12,13,14,15,16,17,18,19,20,21,22
201501,11,12,13,14,15,16,17,18,19,20,21,22
201502,11,12,13,14,15,16,17,18,19,20,21,22
201503,11,12,13,14,15,16,17,18,19,20,21,22
201504,11,12,13,14,15,16,17,18,19,20,21,22
201505,11,12,13,14,15,16,17,18,19,20,21,22
201506,11,12,13,14,15,16,17,18,19,20,21,22
201507,11,12,13,14,15,16,17,18,19,20,21,22
201508,11,12,13,14,15,16,17,18,19,20,21,22
201509,11,12,13,14,15,16,17,18,19,20,21,22
201510,11,12,13,14,15,16,17,18,19,20,21,22
201511,11,12,13,14,15,16,17,18,19,20,21,22
201512,11,12,13,14,15,16,17,18,19,20,21,22
201601,11,12,13,14,15,16,17,18,19,20,21,22
201602,11,12,13,14,15,16,17,18,19,20,21,22
201603,11,12,13,14,15,16,17,18,19,20,21,22
201604,11,12,13,14,15,16,17,18,19,20,21,22
201605,11,12,13,14,15,16,17,18,19,20,21,22
201606,11,12,13,14,15,16,17,18,19,20,21,22
201607,11,12,13,14,15,16,17,18,19,20,21,22
201608,11,12,13,14,15,16,17,18,19,20,21,22
201609,11,12,13,14,15,16,17,18,19,20,21,22
201610,11,12,13,14,15,16,17,18,19,20,21,22
201611,11,12,13,14,15,16,17,18,19,20,21,22
201612,11,12,13,14,15,16,17,18,19,20,21,22
Binary file added pandas/io/tests/sas/data/test_12659.sas7bdat
View file
Open in desktop
Binary file not shown.
10 changes: 10 additions & 0 deletions pandas/io/tests/sas/test_sas7bdat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,13 @@ def test_productsales():
vn = ["ACTUAL", "PREDICT", "QUARTER", "YEAR", "MONTH"]
df0[vn] = df0[vn].astype(np.float64)
tm.assert_frame_equal(df, df0)


def test_12659():
dirpath = tm.get_data_path()
fname = os.path.join(dirpath, "test_12659.sas7bdat")
df = pd.read_sas(fname, encoding='latin1')
fname = os.path.join(dirpath, "test_12659.csv")
df0 = pd.read_csv(fname)
df0 = df0.astype(np.float64)
tm.assert_frame_equal(df, df0)