BUG: read_csv throws UnicodeDecodeError with unicode aliases #13571
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1469,3 +1469,23 @@ def test_memory_map(self): | |
|
|
||
| out = self.read_csv(mmap_file, memory_map=True) | ||
| tm.assert_frame_equal(out, expected) | ||
|
|
||
| def test_read_csv_utf_aliases(self): | ||
| # see gh issue 13549 | ||
| path = 'test.csv' | ||
|
There was a problem hiding this comment. use the context manager remove |
||
| expected = DataFrame({'A': [0, 1], 'B': [2, 3], | ||
|
There was a problem hiding this comment.
There was a problem hiding this comment. I suppose we could do one row as expected = pd.DataFrame({'mb_num': [4.8], 'multibyte': ['test']}) I used BytesIO because I don't think StringIO can support different encodings (I tried and wasn't able to get StringIO to work). |
||
| 'multibyte_test': ['testing123', 'bananabis'], | ||
| 'mb_nums': [154.868, 457.8798]}) | ||
|
|
||
| for byte in [8, 16]: | ||
| expected.to_csv(path, encoding='utf-' + str(byte), index=False) | ||
|
There was a problem hiding this comment. You don't need this anymore. But much cleaner! There was a problem hiding this comment. Ah yeah, whoops. I'm testing the test in a separate file because I haven't figured out how to run the nosetests in common.py yet. |
||
| for fmt in ['utf-{0}', 'utf_{0}', 'UTF-{0}', 'UTF_{0}']: | ||
| encoding = fmt.format(byte) | ||
| for engine in ['c', 'python', None]: | ||
|
There was a problem hiding this comment. you don't iterate thru the engines, this is what There was a problem hiding this comment. So should I just write it like for byte in [8, 16]:
expected.to_csv(path, encoding='utf-' + str(byte), index=False)
for fmt in ['utf-{0}', 'utf_{0}', 'UTF-{0}', 'UTF_{0}']:
encoding = fmt.format(byte)
result = self.read_csv(
path,
encoding=encoding)
tm.assert_frame_equal(result, expected)There was a problem hiding this comment. Yes, that's correct (I didn't see this before because it was hidden away), see below. |
||
| out = self.read_csv( | ||
|
There was a problem hiding this comment. Do you mean change variable 'out' to 'result'? There are at least 8 other test functions in that file that use 'out' and 'expected' and do tm.assert_numpy_array_equal(out, expected) There was a problem hiding this comment. oh i c. we almost always use we NEVER use an There was a problem hiding this comment. the prior function is explicity comparing for rec-arrays There was a problem hiding this comment. Ok, I'll change the 'out's to 'result's in the file for consistency. Is that ok to include in this pull? There was a problem hiding this comment. Surprised no one responded to this, but as that change is NOT directly related to your main fix, it would have been preferable to put in a separate PR (or at least in an individual commit SEPARATE from anything else). So don't undo what you've done, but just for future reference. |
||
| path, | ||
| engine=engine, | ||
| encoding=encoding) | ||
| tm.assert_frame_equal(out, expected) | ||
|
|
||
| os.remove("test.csv") | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
put in 0.18.2
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
double backticks around
UnicodeDecodeErrorThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pd.read_csv()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like 0.18.2 was moved to 0.19.0 this weekend