Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions docs/source/python/api/formats.rst
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,14 @@ CSV Files

.. _api.feather:

Feather Files
-------------
Feather Compatibility APIs
--------------------------

.. currentmodule:: pyarrow.feather

Feather V2 files use the Arrow IPC file format. For new code, prefer the
high-level file APIs in :mod:`pyarrow.ipc`.

.. autosummary::
:toctree: ../generated/

Expand Down
2 changes: 2 additions & 0 deletions docs/source/python/api/ipc.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ Inter-Process Communication
.. autosummary::
:toctree: ../generated/

ipc.write_file
ipc.read_file
ipc.new_file
ipc.open_file
ipc.new_stream
Expand Down
12 changes: 6 additions & 6 deletions docs/source/python/dataset.rst
Original file line number Diff line number Diff line change
Expand Up @@ -142,20 +142,20 @@ provides a consistent interface across multiple file formats and filesystems.
Currently, Parquet, ORC, Feather / Arrow IPC, and CSV file formats are
supported; more formats are planned in the future.

If we save the table as Feather files instead of Parquet files:
If we save the table as an Arrow IPC file instead of a Parquet file:

.. code-block:: python

>>> import pyarrow.feather as feather
>>> import pyarrow.ipc as ipc
>>>
>>> feather.write_feather(table, base / "data.feather")
>>> ipc.write_file(table, base / "data.arrow")

…then we can read the Feather file using the same functions, but with specifying
``format="feather"``:
…then we can read the Arrow IPC file using the same functions, but with
``format="ipc"``:

.. code-block:: python

>>> dataset = ds.dataset(base / "data.feather", format="feather")
>>> dataset = ds.dataset(base / "data.arrow", format="ipc")
>>> dataset.to_table().to_pandas().head()
a b c
0 0 1.764052 1
Expand Down
50 changes: 18 additions & 32 deletions docs/source/python/feather.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@

.. _feather:

Feather File Format
===================
Feather Compatibility API and File Format
=========================================

Feather is a portable file format for storing Arrow tables or data frames (from
languages like Python or R) that utilizes the :ref:`Arrow IPC format <ipc>`
Expand All @@ -35,11 +35,12 @@ R. There are two file format versions for Feather:
* Version 1 (V1), a legacy version available starting in 2016, replaced by
V2. V1 files are distinct from Arrow IPC files and lack many features, such
as the ability to store all Arrow data types. V1 files also lack compression
support. Reading and writing V1 files is deprecated as of 25.0.0 and will
support. Reading and writing V1 files are deprecated as of 25.0.0 and will
be removed in a future version.

The ``pyarrow.feather`` module contains the read and write functions for the
format. :func:`~pyarrow.feather.write_feather` accepts either a
For new Arrow IPC files, prefer the high-level functions in
:mod:`pyarrow.ipc`. The ``pyarrow.feather`` module provides compatibility with
both Feather versions. :func:`~pyarrow.feather.write_feather` accepts either a
:class:`~pyarrow.Table` or ``pandas.DataFrame`` object:

.. code-block:: python
Expand All @@ -48,10 +49,8 @@ format. :func:`~pyarrow.feather.write_feather` accepts either a
feather.write_feather(df, '/path/to/file')

:func:`~pyarrow.feather.read_feather` reads a Feather file as a
``pandas.DataFrame``. :func:`~pyarrow.feather.read_table` reads a Feather file
as a :class:`~pyarrow.Table`. Internally, :func:`~pyarrow.feather.read_feather`
simply calls :func:`~pyarrow.feather.read_table` and the result is converted to
pandas:
``pandas.DataFrame``, while :func:`~pyarrow.feather.read_table` reads it as a
:class:`~pyarrow.Table`:

.. code-block:: python

Expand Down Expand Up @@ -112,17 +111,11 @@ Writing Version 1 (V1) Files
For compatibility with libraries without support for Version 2 files, you can
write the version 1 format by passing ``version=1`` to ``write_feather``.

Migration to IPC
----------------
Using the Arrow IPC API Directly
--------------------------------

.. note::

``pyarrow.feather.write_feather`` and ``pyarrow.feather.read_table``
equivalents will be provided in :mod:`pyarrow.ipc` before the
``pyarrow.feather`` module is deprecated.

Since Feather V2 is the Arrow IPC file format, you can use the
:mod:`pyarrow.ipc` module as a direct replacement:
Since Feather V2 is the Arrow IPC file format, new code can use the high-level
functions in :mod:`pyarrow.ipc` directly:

.. code-block:: python

Expand All @@ -131,21 +124,14 @@ Since Feather V2 is the Arrow IPC file format, you can use the

table = pa.table({"col1": [1, 2, 3], "col2": ["a", "b", "c"]})

# Writing (replaces feather.write_feather)
options = pa.ipc.IpcWriteOptions(compression='lz4')
with pa.ipc.new_file("data.arrow", table.schema, options=options) as writer:
writer.write_table(table)

# Reading (replaces feather.read_table)
with pa.ipc.open_file("data.arrow") as reader:
result = reader.read_all()
# Writing (replaces feather.write_feather for V2 files)
pa.ipc.write_file(table, "data.arrow")

.. note::
# Reading (replaces feather.read_table for V2 files)
result = pa.ipc.read_file("data.arrow")

``feather.write_feather`` defaults to LZ4 compression, while
``ipc.new_file`` does not compress by default. To preserve the same
behavior, pass ``compression='lz4'`` via
:class:`~pyarrow.ipc.IpcWriteOptions` as shown above.
# Convert the result to pandas (replaces feather.read_feather)
dataframe = pa.ipc.read_file("data.arrow").to_pandas()

For reading multiple files, use the :mod:`pyarrow.dataset` module with
``format='ipc'`` instead of :class:`~pyarrow.feather.FeatherDataset`.
25 changes: 25 additions & 0 deletions docs/source/python/ipc.rst
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,31 @@ An important point is that if the input source supports zero-copy reads
(e.g. like a memory map, or ``pyarrow.BufferReader``), then the returned
batches are also zero-copy and do not allocate any new memory on read.

Writing and Reading Complete Tables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Use :func:`~pyarrow.ipc.write_file` and :func:`~pyarrow.ipc.read_file` when
writing or reading a complete :class:`~pyarrow.Table` or
``pandas.DataFrame``:

.. code-block:: python

table = pa.table({"col1": [1, 2, 3], "col2": ["a", "b", "c"]})

pa.ipc.write_file(table, "data.arrow")
result = pa.ipc.read_file("data.arrow")
dataframe = pa.ipc.read_file("data.arrow").to_pandas()

:func:`~pyarrow.ipc.write_file` writes uncompressed files by default and
supports explicit compression and record batch chunking options.
:func:`~pyarrow.ipc.read_file` can select columns by name or index. These
functions read and write Arrow IPC files; use :mod:`pyarrow.feather` only when
legacy Feather V1 compatibility is required. For multiple files, use the
:mod:`pyarrow.dataset` module with ``format='ipc'``.

For incremental writes or random access to individual record batches, use the
lower-level reader and writer APIs described below.

Writing and Reading Random Access Files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down
Loading
Loading