Skip to content

Infer filesystem from path when writing a partitioned DataFrame to remote file systems using pyarrow #34842

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 12 commits into from
Closed

Infer filesystem from path when writing a partitioned DataFrame to remote file systems using pyarrow #34842

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
fdb3d7c
Pass filesystem to parquet.write_table and parquet.write_to_dataset
Jun 17, 2020
a39e900
Fixed style issues
Jun 17, 2020
b30aebb
Simple refactor
Jun 17, 2020
eacd613
Style fix
Jun 17, 2020
38a50fc
Handle user providing filesystem in kwargs
Jun 17, 2020
7bd1ef7
Update whatsnew
Jun 17, 2020
2400420
Remove filesystem from kwargs if it exists
Jun 17, 2020
5ce4ffd
Add tests without the filesystem in the kwargs
Jun 17, 2020
649ce2b
Remove whitespace
Jun 17, 2020
ff444f4
Really remove the whitespace
Jun 17, 2020
720b6c0
Apply black
Jun 17, 2020
a918c6e
Remove comma for PEP8
Jun 17, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Pass filesystem to parquet.write_table and parquet.write_to_dataset
  • Loading branch information
Yuan Chuan Kee committed Jun 18, 2020
commit fdb3d7c8cfe8ebb5f7cdd61c0246b40d08d7977a
7 changes: 6 additions & 1 deletion pandas/io/parquet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,12 +111,17 @@ def write(
table,
path,
compression=compression,
filesystem=get_fs_for_path(path),
partition_cols=partition_cols,
**kwargs,
)
else:
self.api.parquet.write_table(
table, file_obj_or_path, compression=compression, **kwargs
table,
file_obj_or_path,
compression=compression,
filesystem=get_fs_for_path(path),
**kwargs
)
if should_close:
file_obj_or_path.close()
Expand Down