[Python] Add Python protocol for the Arrow C (Data/Stream) Interface

[jorisvandenbossche]

Context: we want that Arrow can be used as the format to share data between (Python) libraries/applications, ideally in a generic way that doesn't need to hardcode for specific libraries.
We already have __arrow_array__ for objects that know how to convert itself to a pyarrow.Array or ChunkedArray. But this protocol is for actual pyarrow objects (so a better name might have been __pyarrow_array__ ..), thus tied to the pyarrow library (and also only for arrays, not for tables/batches). For projects that have an (optional) dependency on pyarrow, that is fine, but we want to avoid that this is required (e.g. nanoarrow). However, we also have the Arrow C Data Interface as a more generic way to share Arrow data in-memory focusing on the actual Arrow spec without relying on a specific library implementation.

Right now, the way to use the C Interface are the _export_to_c and _import_from_c methods.
But those methods are 1) private, advanced APIs (although we can of course decide to make them "official", since many projects are already using them, and document them that way), and 2) again specific to pyarrow (I don't think other projects have adopted the same names).
So other projects (polars, datafusion, duckdb, etc) use those to convert from pyarrow to their own representation. But those projects don't have a similar API to use the C Data Interface to share their data with another (eg to pyarrow, or polars to duckdb, ...).
If we would have a standard Python protocol (dunder) method for this, libraries could implement support for consuming (and producing) objects that expose their data through the Arrow C Interface without having to hard code for specific implementations (such as those libraries currently do for pyarrow).

The most generic protocol would be one supporting the Stream interface, and that could look something like this:

class MyArrowCompatibleObject:

    def __arrow_c_stream__(self) -> PyCapsule:
        """
        Returning a PyCapsule wrapping an ArrowArrayStream struct
        """
        ...

And in addition we could have variants that do the same for the other structs, such __arrow_c_data__ or __arrow_c_array__, __arrow_c_schema__, ..

Some design questions:

• For the mechanics of the method, I would propose to use PyCapsules instead of raw pointers as described here: [Python] Use PyCapsule for communicating C Data Interface pointers at the Python level #34031
• Which set of protocol methods do we need? Is only a stream version sufficient (since a single array can always be put in a stream of one array)? Or would it be useful (and simpler for some applications) to also have an Array version?
  • But what would an array version return exactly? (since it needs to return both the ArrowArray as the ArrowSchema)
• With the ongoing discussion about generalizing the C Interface to other devices (GH-34971: [Format] Add non-CPU version of C Data Interface #34972), should we focus here on the current interfaces, or should we directly use the Device versions?
• Do we want to distinguish between an array and a tabular version? From the C Interface point of view, that's all the same, it's just a ArrowArray. But for example, we currently define _export_to_c on a RecordBatch and RecordBatchReader, where you know this will always return a StructArray representation of one batch, vs the same method on Array where it can return an array of any type. It could be nice to distinguish those use cases for consumers.

[pitrou]

• But what would an array version return exactly? (since it needs to return both the ArrowArray as the ArrowSchema)

It doesn't need to. You can have DataType.__arrow_c_schema__ and Array.__arrow_c_array__ (also for Schema and RecordBatch, respectively).

It could be nice to distinguish those use cases for consumers.

I'm not sure that's useful. @lidavidm Thoughts?

[pitrou]

Also, this proposal doesn't dwell on the consumer side. Would there be higher-level APIs to construct Array and RecordBatch from those capsules?

[jorisvandenbossche]

Also, this proposal doesn't dwell on the consumer side. Would there be higher-level APIs to construct Array and RecordBatch from those capsules?

Yes, indeed I currently didn't touch on that aspect. I think that could certainly be useful, but thought to start with the producer side of things. And some consumers might already have an entry point that could be reused for this (for example, duckdb already implicitly reads from any object that is a pandas DataFrale, pyarrow Table, RecordBatch, Dataset/Scanner, RecordBatchReader, polars DataFrame, ...., and they could just extend this to any object implementing this protocol).
Making the parallel with DLPack again, they recommend that libraries implement a from_dlpack function as the consumer interface. So we could here also have such a recommendation (for example from_arrow, although that might need to differentiate between stream/array/schema), but that's maybe less essential initially? (that's more about user facing API)

[pitrou]

Yes, I don't think we have to recommend a consumer API. But we'll have to choose one for ourselves ;-)

[lidavidm]

It could be nice to distinguish those use cases for consumers.

I'm not sure that's useful. @lidavidm Thoughts?

I'm also not sure it's useful, but it seems we could define __arrow_c_array__ after the fact if we find a use case.

[paleolimbot]

Just a note that I think __arrow_c_array__ and __arrow_c_schema__ are rather essential (I'd build nanoarrow's Python support on top of them). I think it's fairly uncontroversial that their behaviour should align with __arrow_c_array_stream__. A concrete example of somewhere that might implement __arrow_c_schema__ is a GeoArrow type representation...currently they're stored as something more like an integer type ID because it's faster. Substrait types could also implement it or maybe pandas dtypes. It would be rather useful if numpy/pandas.Series implemented __arrow_c_array__, no?

I don't know it if it was mentioned in the discussion, but I think it's fairly important that the PyCapsule have a finalizer that calls the release() callback (if non-null), and to have that documented. I assume that's the point of using the PyCapsule but I haven't discussed that with anybody except maybe in passing with Joris.

Do we want to distinguish between an array and a tabular version?

Most ways that I know about to create an ArrowArray (pa.array(), pa.record_batch(), arrow::as_arrow_array(), etc.) also accept a type or schema. Above the level of "array or table", there are certainly objects whose "one true Arrow type" is ambiguous. You could do __arrow_c_array__(self, schema=None) and __arrow_c_array_stream__(self, schema=None). That gets a little hard because then either the producer or the consumer has to do some sort of equality check or validation.

Did you envision that __arrow_c_stream__() could return things that are not tables? They certainly can and do outside pyarrow (I beleive Rust2 supports it...nanoarrow in R does too). It's a fairly useful representation of a ChunkedArray since there's no other officially ABIified way to do that.

[jorisvandenbossche]

Yes, I don't think we have to recommend a consumer API. But we'll have to choose one for ourselves ;-)

Indeed. And for pyarrow, it could also be something like RecordBatchReader.from_arrow_stream (or from_arrow_c_stream, or other name), and similarly for other objects, to keep it consistent with existing from_ methods.

[Do we want to distinguish between an array and a tabular version? ...] It could be nice to distinguish those use cases for consumers.

I'm not sure that's useful. @lidavidm Thoughts?

I'm also not sure it's useful, but it seems we could define __arrow_c_array__ after the fact if we find a use case.

To clarify this part a bit, and assume we are talking about the ArrowArray version to keep it simple (not the stream). Currently, a pyarrow.Array can be exported to an ArrowArray, and a pyarrow.RecordBatch as well (but in the second case, you know you always have a struct type).
The C Interface itself doesn't distinguish between both (and that's fine), but in practice the interface is used for both "types" of data (array vs tabular). And for a consumer, I can imagine it would be useful to distinguish. For example, assume that pandas has a function to construct a pandas.DataFrame from any object that supports this protocol. In that case, pandas might only be interested in data that logically represents tabular data, and not an array (because then you don't have column names, might have top-level nulls, etc). In case there is only a single __arrow_c_array__, pandas could of course check if the data it received matches the requirements for tabular data (i.e. is a struct array and has no validity bitmap). But if there would be two protocol methods (eg __arrow_c_array__ and __arrow_c_batch__), it could only check for objects that define the second method (and declare themselves as tabular data)

Did you envision that __arrow_c_stream__() could return things that are not tables? They certainly can and do outside pyarrow (I beleive Rust2 supports it...nanoarrow in R does too). It's a fairly useful representation of a ChunkedArray since there's no other officially ABIified way to do that.

Yes, it currently essentially returns an array, not a table. We just mostly use for tables in practice.

As a concrete example: in the arrow-rs implementation, the RecordBatch conversion to/from pyarrow actually iterates over each field to convert field by field using the C interface on each array, instead of using a single C interface call using a struct array for the full RecordBatch (https://github.com/apache/arrow-rs/blob/3adca539ad9e1b27892a5ef38ac2780aff4c0bff/arrow/src/pyarrow.rs#L167-L204)
(EDIT: this example is for the array interface, not the stream interface, though. It might be true that in practice the stream interface is only being used for tabular data)

[jorisvandenbossche]

I don't know it if it was mentioned in the discussion, but I think it's fairly important that the PyCapsule have a finalizer that calls the release() callback (if non-null), and to have that documented.

Yes, that's certainly the idea, and discussed in #34031

Most ways that I know about to create an ArrowArray (pa.array(), pa.record_batch(), arrow::as_arrow_array(), etc.) also accept a type or schema. ..

That's a good question. Other protocols like __array__ or __arrow_array__ (which are library specificy, numpy and pyarrow, respectively), also support this. But more similar protocols like __array_interface__ or __dlpack__ (which both also share pointers to buffers, library-independent) don't do this.

I think one aspect here is that this second set of methods assume you "just" give access to the actual, underlying data, and not do any conversion (and so are always zero-copy), or otherwise would raise an error if the type is not supported through the protocol (and so there is never a question of what the "correct" type would be for the C interface).
In the DataFrame world, there is more variation in data types, though, and so this might be less straightforward. You will much more easily end up in a situation where the DataFrame object has a column of data that is not exactly / natively in arrow memory, and in those cases there might indeed be some ambiguity in which type should be used. Or, whether such conversion should be supported or rather an error should be raised.

[pitrou]

But if there would be two protocol methods (eg __arrow_c_array__ and __arrow_c_batch__), it could only check for objects that define the second method (and declare themselves as tabular data)

That doesn't sound particularly useful? Especially as Pandas will probably have to check for other things (such as the datatypes it supports).

[paleolimbot]

Or, whether such conversion should be supported or rather an error should be raised.

I think you could do both via schema=None as the default? That's the case most of the time anyway ("just get me this array").

class Integerish:

    def __arrow_c_array__(self, schema=None):
        if schema is not None and :
            raise ValueError("Only default export supported")

        return make_array_capsule(self), make_int_schema_capsule()

Of course, if schema happens to be the correct type this will still error. In R/nanoarrow I also have infer_nanoarrow_schema() (the type the array would be if it were requested) so a consumer that, say, only supports signed integer types can choose the appropriate signed type if ___arrow_c_schema__() returns an unsigned one. A producer like pyarrow can cast no problem (more minimal producers would probably just error).

[wjones127]

I like Dewey's suggestion for having an API to request casting types you don't know how to use, given there is also an API to get the zero-copy schema. This seems especially useful for Arrow "types" that are just encodings of the same logical type, such as Utf8 / LargeUtf8 (and soon StringView).