Skip to content

IO: Fix parquet read from s3 directory #33632

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Apr 26, 2020
Merged

IO: Fix parquet read from s3 directory #33632

Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
aa94fe7
parquet init
alimcmaster1 Apr 18, 2020
a30c71a
Doc Str
alimcmaster1 Apr 18, 2020
b2747eb
Simplify read
alimcmaster1 Apr 18, 2020
a51757a
Fix writer with partition
alimcmaster1 Apr 18, 2020
968f3b6
Test case
alimcmaster1 Apr 18, 2020
789f4ca
Clean up test case
alimcmaster1 Apr 18, 2020
040763e
Add whatsnew
alimcmaster1 Apr 18, 2020
40f5889
Clean ups
alimcmaster1 Apr 18, 2020
753d647
Clean ups
alimcmaster1 Apr 18, 2020
e4dcdc3
Update whatsnew
alimcmaster1 Apr 18, 2020
bb21431
Add skip if no
alimcmaster1 Apr 18, 2020
fb38932
Fix import
alimcmaster1 Apr 18, 2020
c29befd
Removed fixed xfail
alimcmaster1 Apr 18, 2020
4f78fc5
remove import
alimcmaster1 Apr 18, 2020
4b2828b
Merge master
alimcmaster1 Apr 21, 2020
463c2ea
Merge remote-tracking branch 'upstream/master' into mcmali-parquet
alimcmaster1 Apr 21, 2020
dabfe58
Add further test case
alimcmaster1 Apr 21, 2020
dea95f3
Update parquet.py
alimcmaster1 Apr 25, 2020
ae76e42
Update parquet.py
alimcmaster1 Apr 25, 2020
4b48326
Add whatsnew 2
alimcmaster1 Apr 26, 2020
211c36e
Rename var
alimcmaster1 Apr 26, 2020
4897a32
Improve get_fs_for_path docstring
alimcmaster1 Apr 26, 2020
bba4040
Merge remote-tracking branch 'origin/mcmali-parquet' into mcmali-parquet
alimcmaster1 Apr 26, 2020
5bc6327
Add doc example
alimcmaster1 Apr 26, 2020
ca89c21
Make whatsnew clearer
alimcmaster1 Apr 26, 2020
0df818e
Merge remote-tracking branch 'upstream/master' into mcmali-parquet
alimcmaster1 Apr 26, 2020
2a1a85c
Merge remote-tracking branch 'upstream/master' into mcmali-parquet
alimcmaster1 Apr 26, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update parquet.py
  • Loading branch information
@alimcmaster1
alimcmaster1 authored Apr 25, 2020
commit ae76e42a277b743950f1d3fc6768ddfc10988ec6
2 changes: 1 addition & 1 deletion pandas/io/parquet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ def write(
from_pandas_kwargs["preserve_index"] = index

table = self.api.Table.from_pandas(df, **from_pandas_kwargs)
# write_to_dataset does not support a file-like object when
# write_to_dataset does not support a file-like object when
# a dircetory path is used, so just pass the path string.
if partition_cols is not None:
self.api.parquet.write_to_dataset(
Expand Down