Skip to content

ENH: add support for reading .tar archives #44787

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 38 commits into from
May 7, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
c1823ef
Add reproduction test for .tar.gz archives
Skn0tt Dec 6, 2021
9a85cba
add support for .tar archives
Skn0tt Dec 6, 2021
e673061
update doc comments
Skn0tt Dec 6, 2021
a0d6386
fix: pep8 errors
Skn0tt Dec 6, 2021
6a8edef
refactor: flip _compression_to_extension around to support multiple e…
Skn0tt Dec 7, 2021
d4e40c9
refactor: detect tar files using existing extension mapping
Skn0tt Dec 7, 2021
5f22df7
feat: add support for writing tar files
Skn0tt Dec 7, 2021
c6573ef
feat: assure it respects .gz endings
Skn0tt Dec 15, 2021
f3b6ed5
Merge branch 'master' into read-tar-archives
Skn0tt Dec 15, 2021
a4ac382
feat: add "tar" entry to compressionoptions
Skn0tt Dec 15, 2021
e66826b
chore: add whatsnew entry
Skn0tt Dec 15, 2021
941be37
fix: test_compression_size_fh
Skn0tt Dec 15, 2021
e3369aa
Merge branch 'master' into read-tar-archives
Skn0tt Jan 4, 2022
0468e5f
add tarfile to shared compression docs
Skn0tt Jan 4, 2022
2531ee0
fix formatting
Skn0tt Jan 4, 2022
57eba0a
pass through "mode" via compression args
Skn0tt Jan 4, 2022
38f7d54
fix pickle test
Skn0tt Jan 4, 2022
887fd10
add class comment
Skn0tt Jan 4, 2022
fc2e7f0
Merge remote-tracking branch 'origin/main' into read-tar-archives
Skn0tt Apr 9, 2022
669d942
sort imports
Skn0tt Apr 9, 2022
7d7d3c6
add _compression_to_extension back for backwards compatibility
Skn0tt Apr 9, 2022
8b8b8ac
fix some type warnings
Skn0tt Apr 9, 2022
dd356f6
fix: formatting
Skn0tt Apr 9, 2022
514014a
fix: mypy complaints
Skn0tt Apr 9, 2022
38971c7
fix: more tests
Skn0tt Apr 9, 2022
e35d361
fix: some error with xml
Skn0tt Apr 9, 2022
c5088fc
fix: interpreted text role
Skn0tt Apr 9, 2022
f6c5173
move to v1.5 whatsnw
Skn0tt Apr 9, 2022
9a4fa07
add versionadded note
Skn0tt Apr 11, 2022
0c31aa8
don't leave blank lines
Skn0tt Apr 11, 2022
086c598
add tests for zero files / multiple files
Skn0tt Apr 13, 2022
861faf0
move _compression_to_extension to tests
Skn0tt Apr 13, 2022
9458ecb
revert added "mode" argument
Skn0tt Apr 13, 2022
d20f315
add test to ensure that `compression.mode` works
Skn0tt Apr 13, 2022
1066f1b
Merge branch 'main' into read-tar-archives
Skn0tt Apr 25, 2022
6b0e1e6
Merge branch 'main' into read-tar-archives
Skn0tt May 5, 2022
0d9ed18
compare strings, not bytes
Skn0tt May 5, 2022
37370c2
replace carriage returns
Skn0tt May 6, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat: assure it respects .gz endings
  • Loading branch information
Skn0tt authored Dec 15, 2021
commit c6573efdabb07ab5074a6b3b06ab63d9ac29a04e
20 changes: 18 additions & 2 deletions pandas/io/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -875,9 +875,25 @@ def __init__(
super().__init__(name=name, mode=mode, fileobj=fileobj, **kwargs)

@classmethod
def open(cls, mode="r", **kwargs):
def open(cls, name=None, mode="r", **kwargs):
mode = mode.replace("b", "")
return super().open(mode=mode, **kwargs)
return super().open(name=name, mode=cls.extend_mode(name, mode), **kwargs)

@classmethod
def extend_mode(
cls, name: FilePath | ReadBuffer[bytes] | WriteBuffer[bytes], mode: str
) -> str:
if mode != "w":
return mode
if isinstance(name, (os.PathLike, str)):
filename = Path(name)
if filename.suffix == ".gz":
return mode + ":gz"
elif filename.suffix == ".xz":
return mode + ":xz"
elif filename.suffix == ".bz2":
return mode + ":bz2"
return mode

def infer_filename(self):
"""
Expand Down
24 changes: 24 additions & 0 deletions pandas/tests/io/parser/test_compression.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@

import os
from pathlib import Path
import tarfile
import zipfile

import pytest
Expand Down Expand Up @@ -180,3 +181,26 @@ def test_ignore_compression_extension(all_parsers):
Path(path_zip).write_text(Path(path_csv).read_text())

tm.assert_frame_equal(parser.read_csv(path_zip, compression=None), df)


@skip_pyarrow
def test_writes_tar_gz(all_parsers):
parser = all_parsers
data = DataFrame(
{
"Country": ["Venezuela", "Venezuela"],
"Twitter": ["Hugo Chávez Frías", "Henrique Capriles R."],
}
)
with tm.ensure_clean("test.tar.gz") as tar_path:
data.to_csv(tar_path, index=False)

# test that read_csv infers .tar.gz to gzip:
tm.assert_frame_equal(parser.read_csv(tar_path), data)

# test that file is indeed gzipped:
with tarfile.open(tar_path, "r:gz") as tar:
result = parser.read_csv(
tar.extractfile(tar.getnames()[0]), compression="infer"
)
tm.assert_frame_equal(result, data)