Export to Pandas

[MarcoGorelli]

Say I have an object df which implements __arrow_c_stream__ (say, a Polars dataframe)

Currently, I can convert that to pandas by doing:

import pyarrow as pa

pa.table(df).to_pandas()

In this case, I would only be using PyArrow as a container, without needing all of its compute

Does arro3 provide a way to convert from Polars to pandas without having to go via PyArrow?

[kylebarron]

Short answer: close, but not stable and tested enough to put in prod. And no top-level to_pandas API (yet).

There are a couple ways of converting Arrow data to Numpy/Python scalars, but beyond that, there's specifics around pandas semantics that I don't know well enough. At this point, it might be easiest to just look at how pyarrow implements it and port that, because there will be specifics around null handling I assume. But I'd want to see how many LOC the pyarrow implementation is.

Scalar.as_py

This is the most well-supported way right now. Only interval, decimal, and REE arrays are not currently supported.

So you could call this for every scalar and pass that to a pd.DataFrame constructor, but that would be pretty slow.

Array.to_numpy/ChunkedArray.to_numpy

This is implemented for primitive numeric types:

arro3/pyo3-arrow/src/interop/numpy/to_numpy.rs

Lines 27 to 37 in c14dd05

	DataType::Float16 => impl_primitive!(Float16Type),
	DataType::Float32 => impl_primitive!(Float32Type),
	DataType::Float64 => impl_primitive!(Float64Type),
	DataType::UInt8 => impl_primitive!(UInt8Type),
	DataType::UInt16 => impl_primitive!(UInt16Type),
	DataType::UInt32 => impl_primitive!(UInt32Type),
	DataType::UInt64 => impl_primitive!(UInt64Type),
	DataType::Int8 => impl_primitive!(Int8Type),
	DataType::Int16 => impl_primitive!(Int16Type),
	DataType::Int32 => impl_primitive!(Int32Type),
	DataType::Int64 => impl_primitive!(Int64Type),

That function could for non-numeric types just fall back to creating a list of Python scalars and passing that to the numpy constructor.

[yohplala]

Hi, trying to use arro3 here, ... with pandas dataframe ...
I am managing this kind of data:

tests/test_store/test_ordered_parquet_dataset.py opdmd data
        ordered_on_mins     ordered_on_maxs  n_rows  file_ids
0   2021-01-01 00:00:00 2021-01-01 00:00:00       1         0
1   2021-01-01 00:01:00 2021-01-01 00:01:00       1         1

I think I can print it successfully to parquet file.

from arro3.io import write_parquet
from pandas import DataFrame, Timestamp
from numpy import uint16, uint32

tmp_path = "/home/yoh/Documents/code/data/oups/test_arro3.parquet"

df = DataFrame({
"ordered_on_mins": [Timestamp("2021-01-01 00:00:00")],
"n_rows":  [1],
"file_ids": [0]
}).astype({"n_rows": uint32, "file_ids": uint16})

write_parquet(df, tmp_path)

This works.
But it seems reading back timestamp is not yet supported.

from arro3.io import read_parquet

table = read_parquet(tmp_path).read_all()
row_group_stats = DataFrame(table.to_struct_array().to_numpy())

This raises:

NotImplementedError:
Unsupported type in to_numpy Struct([
Field {
 name: "ordered_on_mins",
 data_type: Timestamp(Nanosecond, None),
 nullable: true,
 dict_id: 0,
 dict_is_ordered: false,
 metadata: {}
 },
 Field {
 name: "ordered_on_maxs",
 data_type: Timestamp(Nanosecond, None),
 nullable: true,
 dict_id: 0,
 dict_is_ordered: false,
 metadata: {}
 },
 Field {
 name: "n_rows",
 data_type: UInt32,
 nullable: true,
 dict_id: 0,
 dict_is_ordered: false,
 metadata: {}
 },
 Field {
 name: "file_ids",
 data_type: UInt16,
 nullable: true,
 dict_id: 0,
 dict_is_ordered:
 false,
 metadata: {}
 }
])

Please, do you think you could add support for Timestamp / datetime type in to_numpy()?
Thanks a lot for your help.

[kylebarron]

I think I can print it successfully to parquet file.

from arro3.io import write_parquet
from pandas import DataFrame, Timestamp
from numpy import uint16, uint32

tmp_path = "/home/yoh/Documents/code/data/oups/test_arro3.parquet"

df = DataFrame({
"ordered_on_mins": [Timestamp("2021-01-01 00:00:00")],
"n_rows":  [1],
"file_ids": [0]
}).astype({"n_rows": uint32, "file_ids": uint16})

write_parquet(df, tmp_path)

This works. But it seems reading back timestamp is not yet supported.

This works because you're using the __arrow_c_stream__ method on pandas.DataFrame, which uses pyarrow to convert the data into Arrow.

from arro3.io import read_parquet

table = read_parquet(tmp_path).read_all()
row_group_stats = DataFrame(table.to_struct_array().to_numpy())

The ideal way to solve this would be pandas-dev/pandas#59696. If you already have pyarrow in your environment and you're happy with using that, you can use that to convert from Arrow to Pandas.

I would be happy to accept a PR to convert Arrow datetimes to numpy arrays, but I don't expect to implement that soon myself.