Skip to content

Default to_* methods to compression='infer' #22011

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 40 commits into from
Aug 1, 2018
Merged
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
8689167
Default to_csv & to_json to compression='infer'
dhimmel Jul 21, 2018
3ccfb00
to_json compression=infer in pandas/core/generic.py
dhimmel Jul 21, 2018
648bf4d
Simplify CSVFormatter.save
dhimmel Jul 21, 2018
be724fa
Exploratory commit of what CSVFormatter.save should look like
dhimmel Jul 21, 2018
9fe27c9
fixup! Simplify CSVFormatter.save
dhimmel Jul 23, 2018
65f0689
"Revert changes not related to compression default
dhimmel Jul 23, 2018
868e671
TST: test to_csv infers compression by default
dhimmel Jul 23, 2018
c3b76ee
Debugging print statements
dhimmel Jul 23, 2018
cebc0d9
Debugging: use logging rather than print
dhimmel Jul 23, 2018
8411eb2
_infer_compression in CSVFormatter
dhimmel Jul 26, 2018
c098c8f
CSVFormatter: process encoding in init for consistency
dhimmel Jul 26, 2018
2f6601d
TST + DOC: test_compression_warning docstring
dhimmel Jul 26, 2018
eb7f9b5
fixup! CSVFormatter: process encoding in init for consistency
dhimmel Jul 26, 2018
d4a5c90
Tests passing: remove debugging
dhimmel Jul 26, 2018
abd19e3
Parametrized test for compression='infer' is default
dhimmel Jul 26, 2018
2f670fe
Default compression='infer' in series.to_csv
dhimmel Jul 26, 2018
aa9ce13
What's New Entry for v0.24.0
dhimmel Jul 26, 2018
a6aabad
Remove unused tmpdir fixture argument
dhimmel Jul 26, 2018
8a0c97e
Update to_json docstring
dhimmel Jul 26, 2018
6be808d
Change test docstrings to comments
dhimmel Jul 26, 2018
63e6591
Consolidate testing to a single parametrized test
dhimmel Jul 26, 2018
fadb943
Split test_compression_defaults_to_infer into Series & DataFrame tests
dhimmel Jul 26, 2018
0edffc7
Parametrize write_kwargs
dhimmel Jul 26, 2018
97f5de5
Fix kwargs in test_series_compression_defaults_to_infer
dhimmel Jul 26, 2018
83bc0a8
Attempt to fix CSV series roundtrip
dhimmel Jul 26, 2018
874a4bf
Fix test failure
dhimmel Jul 26, 2018
14c3945
Python 2 flake8 error
dhimmel Jul 26, 2018
9a4dc41
Reduce / remove comments
dhimmel Jul 27, 2018
25bdb4c
Merge master: fix zip-docs conflict
dhimmel Jul 29, 2018
1ba8f3a
DOC: versionchanged & tweaks
dhimmel Jul 30, 2018
24e051e
Update doc/source/io.rst as needed
dhimmel Jul 30, 2018
387d1d2
Move tests from tests/test_common.py to tests/io/test_common.py
dhimmel Jul 30, 2018
12f14e2
Organize / simplify pandas/tests/test_common.py imports
dhimmel Jul 30, 2018
6db23d9
Ignore flake error needed for test
dhimmel Jul 30, 2018
e3a0f56
fixup! Organize / simplify pandas/tests/test_common.py imports
dhimmel Jul 30, 2018
af8c137
change import: cmn to icom
dhimmel Jul 31, 2018
f8829a6
Blank lines after versionchanged
dhimmel Jul 31, 2018
918c0f8
Move compression tests to new file tests/io/test_compression.py
dhimmel Jul 31, 2018
eadf68e
blank lines before .. versionchanged
dhimmel Jul 31, 2018
cf5b62e
Remove comments and space after GH
dhimmel Aug 1, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Change test docstrings to comments
  • Loading branch information
dhimmel committed Jul 26, 2018
commit 6be808d87f8470b733534969397613784eb35a1c
25 changes: 10 additions & 15 deletions pandas/tests/test_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -220,10 +220,8 @@ def test_standardize_mapping():
Series(100 * [0.123456, 0.234567, 0.567567], name='X')])
@pytest.mark.parametrize('method', ['to_pickle', 'to_json', 'to_csv'])
def test_compression_size(obj, method, compression_only):
"""
Tests that compression is occurring by comparing to the bytes on disk of
the uncompressed file.
"""
# Tests that compression is occurring by comparing to the bytes on disk of
# the uncompressed file.
extension = _compression_to_extension[compression_only]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would prefer to keep this test unchanged and ensure that the test above is responsible for infer behavior

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the paremetrization of this test is ideal. Currently this tests compression X to_method X dataframe_or_series. Luckily this exhaustive parametrization caught that I hadn't switched the default for Series.to_csv. Also note that the functional changes to this test are minimal (just adding getting the size of an inferred file). The rest of the changes help readability.

Were we to leave test_compression_size unmodified, I'd want to repeat the parametrization. Is there a dataframe_or_series fixture?

to_method = getattr(obj, method)
with tm.ensure_clean('no-compression') as path:
Expand Down Expand Up @@ -264,17 +262,14 @@ def test_compression_size_fh(obj, method, compression_only):


def test_compression_warning(compression_only):
"""
Assert that passing a file object to to_csv while explicitly specifying a
compression protocol triggers a RuntimeWarning, as per
https://github.com/pandas-dev/pandas/issues/21227.

Note that pytest has an issue that causes assert_produces_warning to fail
in Python 2 if the warning has occurred in previous tests
(see https://git.io/fNEBm & https://git.io/fNEBC). Hence, should this test
fail in just Python 2 builds, it likely indicates that other tests are
producing RuntimeWarnings, thereby triggering the pytest bug.
"""
# Assert that passing a file object to to_csv while explicitly specifying a
# compression protocol triggers a RuntimeWarning, as per
# https://github.com/pandas-dev/pandas/issues/21227.
# Note that pytest has an issue that causes assert_produces_warning to fail
# in Python 2 if the warning has occurred in previous tests
# (see https://git.io/fNEBm & https://git.io/fNEBC). Hence, should this
# test fail in just Python 2 builds, it likely indicates that other tests
# are producing RuntimeWarnings, thereby triggering the pytest bug.
df = DataFrame(100 * [[0.123456, 0.234567, 0.567567],
[12.32112, 123123.2, 321321.2]],
columns=['X', 'Y', 'Z'])
Expand Down