Skip to content

ENH: .read_pickle(...) from zip containing hidden OS X/macOS metadata files/folders #37101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 17 commits into from
Closed
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'master' into ml-evs/fix_#37098
  • Loading branch information
ml-evs authored Nov 19, 2020
commit 85a5cffa3e6cd83c56ec1a973cf9d71936e4d8f7
107 changes: 107 additions & 0 deletions doc/source/whatsnew/v1.2.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -252,8 +252,115 @@ Other enhancements
- :func:`read_csv` supports memory-mapping for compressed files (:issue:`37621`)
- Improve error reporting for :meth:`DataFrame.merge()` when invalid merge column definitions were given (:issue:`16228`)
- Improve numerical stability for :meth:`Rolling.skew()`, :meth:`Rolling.kurt()`, :meth:`Expanding.skew()` and :meth:`Expanding.kurt()` through implementation of Kahan summation (:issue:`6929`)
- Improved error reporting for subsetting columns of a :class:`DataFrameGroupBy` with ``axis=1`` (:issue:`37725`)
- :func:`pandas.load_pickle` can now load from ``.zip`` files created by OS X/macOS that contain ``__MACOSX/`` or ``.DS_STORE`` hidden folders/files (:issue:`37098`).


.. ---------------------------------------------------------------------------

.. _whatsnew_120.notable_bug_fixes:

Notable bug fixes
~~~~~~~~~~~~~~~~~

These are bug fixes that might have notable behavior changes.

Consistency of DataFrame Reductions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
:meth:`DataFrame.any` and :meth:`DataFrame.all` with ``bool_only=True`` now
determines whether to exclude object-dtype columns on a column-by-column basis,
instead of checking if *all* object-dtype columns can be considered boolean.

This prevents pathological behavior where applying the reduction on a subset
of columns could result in a larger :class:`Series` result. See (:issue:`37799`).

.. ipython:: python

df = pd.DataFrame({"A": ["foo", "bar"], "B": [True, False]}, dtype=object)
df["C"] = pd.Series([True, True])


*Previous behavior*:

.. code-block:: ipython

In [5]: df.all(bool_only=True)
Out[5]:
C True
dtype: bool

In [6]: df[["B", "C"]].all(bool_only=True)
Out[6]:
B False
C True
dtype: bool

*New behavior*:

.. ipython:: python

In [5]: df.all(bool_only=True)

In [6]: df[["B", "C"]].all(bool_only=True)


Other :class:`DataFrame` reductions with ``numeric_only=None`` will also avoid
this pathological behavior (:issue:`37827`):

.. ipython:: python

df = pd.DataFrame({"A": [0, 1, 2], "B": ["a", "b", "c"]}, dtype=object)


*Previous behavior*:

.. code-block:: ipython

In [3]: df.mean()
Out[3]: Series([], dtype: float64)

In [4]: df[["A"]].mean()
Out[4]:
A 1.0
dtype: float64

*New behavior*:

.. ipython:: python

df.mean()

df[["A"]].mean()

Moreover, :class:`DataFrame` reductions with ``numeric_only=None`` will now be
consistent with their :class:`Series` counterparts. In particular, for
reductions where the :class:`Series` method raises ``TypeError``, the
:class:`DataFrame` reduction will now consider that column non-numeric
instead of casting to NumPy which may have different semantics (:issue:`36076`,
:issue:`28949`, :issue:`21020`).

.. ipython:: python

ser = pd.Series([0, 1], dtype="category", name="A")
df = ser.to_frame()


*Previous behavior*:

.. code-block:: ipython

In [5]: df.any()
Out[5]:
A True
dtype: bool

*New behavior*:

.. ipython:: python

df.any()


.. _whatsnew_120.api_breaking.python:

Increased minimum version for Python
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.