[R/Python] PyCapsule layer may need unpacking

[eddelbuettel]

In the past, one could generating a nanoarrow Array object and pass it from R via reticulate to Python to, say, instantiate a polars object.

That no longer works as an intermediate PyCapsule layer is added, and this may need an extra layer of unpacking

> pp <- palmerpenguins::penguins
> na <- nanoarrow::as_nanoarrow_array(pp)
> class(na)
[1] "nanoarrow_array"
> pdf <- polars::as_polars_lf(na)   # conversion on the R side, R package `polars` know `nanoarrow`
> pdf
<polars_lazy_frame>
> 
> po <- reticulate::import("polars")
> df <- po$from_arrow(na)     # conversion on the Python side via `reticulate`
Error in py_call_impl(callable, call_args$unnamed, call_args$named) : 
  TypeError: expected PyArrow Table, Array, or one or more RecordBatches; got 'PyCapsule'
Run `reticulate::py_last_error()` for details.
> 

[eddelbuettel]

I fiddled a little with this last evening, based on off-line comments first gotten from @paleolimbot and @t-kalinowski. I may have something that relies only reticulate and uses a combination of nanoarrow::nanoarrow_pointer_addr_chr() and reticulate::r_to_py to pass the pointers "Python-style". But as so often, I am stuck on a tiny bit: how do I (in R only) get the schema external pointer out of the tag of the array external pointer that is the nanoarrow_array? I helped myself with an ad-hoc C/C++ accessor but I wondering if there is a shorter path I overlooked?

[eddelbuettel]

A slightly refined and cleaned up version, which needs a small extension to the archived xptr CRAN package (which I'll likely resurrect and re-send to CRAN, author/former maintainer is on board) to create '%lld' output which we can pass to r_to_py() so that int() can take them:

capsule_prep <- function(na) {
    ## pointer addresses for array and schema in 'long long int' format
    arrchr <- xptr::xptr_address(na, FALSE)
    schchr <- xptr::xptr_address(xptr::xptr_tag(na), FALSE)
    ## Import with convert = FALSE so that `_import_from_c` returns a Python object
    py <- reticulate::import_builtins(convert = FALSE)
    pa <- reticulate::import("pyarrow", convert = FALSE)
    out <- pa$Array$`_import_from_c`(py$int(reticulate::r_to_py(arrchr)),
                                     py$int(reticulate::r_to_py(schchr)))
    out
}

If I wrap that around my nanoarrow-created RecordBatch then I can send this to Python as in

    pa <- import("pyarrow")

    na2 <- capsule_prep(narb)   # narb is the nanoarrow-create record batch object

    ## create a record batch
    df <- pa$record_batch(na2)
    ## which we can turn into a table
    ## which can be exported to pandas
    print(pa$table(df)$to_pandas())

The question (mostly to @paleolimbot) now is: can something like this (possibly without the xptr dependence, this is easy to replicate here / already exists) be injected here in an elegant way / possibly with S3 magic to be 'automagically' invoked? I am a little unclear on how a nanoarrow object would know it is become a Python object ...

[paleolimbot]

Thank you for this!

We probably don't need xptr (I beleive infer_nanoarrow_schema() will get you the external pointer you're looking for from an array).

Perhaps the nanoarrow Python package can help here?

library(nanoarrow)
library(reticulate)

na <- reticulate::import("nanoarrow.c_array", convert = FALSE)
dst <- na$allocate_c_array()
dst_addr <- reticulate::py_str(dst[["_addr"]]())
dst_schema_addr <- reticulate::py_str(dst$schema[["_addr"]]())

arr <- as_nanoarrow_array(data.frame(x = 1:10))
nanoarrow_pointer_export(arr, dst_addr)
nanoarrow_pointer_export(infer_nanoarrow_schema(arr), dst_schema_addr)

dst
#> <nanoarrow.c_array.CArray struct<x: int32>>
#> - length: 10
#> - offset: 0
#> - null_count: 0
#> - buffers: (0,)
#> - dictionary: NULL
#> - children[1]:
#>   'x': <nanoarrow.c_array.CArray int32>
#>     - length: 10
#>     - offset: 0
#>     - null_count: 0
#>     - buffers: (0, 4684873592)
#>     - dictionary: NULL
#>     - children[0]:

pl <- reticulate::import("polars")
pl$from_arrow(dst)
#> shape: (10, 1)
#> ┌─────┐
#> │ x   │
#> │ --- │
#> │ i32 │
#> ╞═════╡
#> │ 1   │
#> │ 2   │
#> │ 3   │
#> │ 4   │
#> │ 5   │
#> │ 6   │
#> │ 7   │
#> │ 8   │
#> │ 9   │
#> │ 10  │
#> └─────┘

^{Created on 2025-09-26 with reprex v2.1.1}

[eddelbuettel]

Slick and minimal and great existence proof. One downside, though, is that in order to make this happen you allocated the objects differently.

In my ideal world, I would keep the same universality of Arrow object created by nanoarrow at C(++) level and pass them whereever I want. And if Python needs special treatment, would it then be possibky to shimmy a conversion layer in "somewhere" as per the earlier email discussion? Can you see a way to tuck the above away somewhere in R package layer?

[paleolimbot]

Yes, we can use that to implement r_to_py/py_to_r 🙂

The allocation is quite literally a C++ "move" (i.e., no data is moving, just the top-level structure). nanoarrow_pointer_export() takes an extra step to ensure that the original R object arr remains valid...you can use nanoarrow_pointer_move() to skip that step if you know that arr will never be accessible (e.g., inside a function you own).

[eddelbuettel]

Good point regardingt the ownership. I had not passed the simple example past, say, valgrind and was wondering aboout that too because my minimal example above just copies the pointer address around without clearing ownership as nanoarrow generally. But hey that's why I brought it here to have you eyeball it too 😉 Will be nice to have this.

[eddelbuettel]

I didn't get back to this for a few days but felt, as you had also hinted, that we might not need the xptr helpers. Indeed -- nanoarrow can 'print pointers as character' in the %lld form, and as you reminded me, infer_nanoarrow_schema() picks the pointer from the tag (if one is there). [ If there was a vote, I would vote for aliasing this to a more evocative second name ... ] So the helper becomes

capsule_prep <- function(na) {
    ## pointer addresses for array and schema in 'long long int' format
    arrchr <- nanoarrow::nanoarrow_pointer_addr_chr(na)
    schchr <- nanoarrow::nanoarrow_pointer_addr_chr(nanoarrow::infer_nanoarrow_schema(na))
    
    ## Import with convert = FALSE so that `_import_from_c` returns a Python object
    py <- reticulate::import_builtins(convert = FALSE)
    pa <- reticulate::import("pyarrow", convert = FALSE)
    out <- pa$Array$`_import_from_c`(py$int(reticulate::r_to_py(arrchr)),
                                     py$int(reticulate::r_to_py(schchr)))
    out
}

and with that we can pass everything around as hoped for:

> narb <- nanoarrow::as_nanoarrow_array( palmerpenguins::penguins )
> na2 <- capsule_prep(narb)
> df <- po$from_dataframe(na2)
> df
shape: (344, 8)
┌───────────┬───────────┬────────────────┬───────────────┬───────────────┬─────────────┬────────┬──────┐
│ species   ┆ island    ┆ bill_length_mm ┆ bill_depth_mm ┆ flipper_lengt ┆ body_mass_g ┆ sex    ┆ year │
│ ---       ┆ ---       ┆ ---            ┆ ---           ┆ h_mm          ┆ ---         ┆ ---    ┆ ---  │
│ cat       ┆ cat       ┆ f64            ┆ f64           ┆ ---           ┆ i32         ┆ cat    ┆ i32  │
│           ┆           ┆                ┆               ┆ i32           ┆             ┆        ┆      │
╞═══════════╪═══════════╪════════════════╪═══════════════╪═══════════════╪═════════════╪════════╪══════╡
│ Adelie    ┆ Torgersen ┆ 39.1           ┆ 18.7          ┆ 181           ┆ 3750        ┆ male   ┆ 2007 │
│ Adelie    ┆ Torgersen ┆ 39.5           ┆ 17.4          ┆ 186           ┆ 3800        ┆ female ┆ 2007 │
│ Adelie    ┆ Torgersen ┆ 40.3           ┆ 18.0          ┆ 195           ┆ 3250        ┆ female ┆ 2007 │
│ Adelie    ┆ Torgersen ┆ null           ┆ null          ┆ null          ┆ null        ┆ null   ┆ 2007 │
│ Adelie    ┆ Torgersen ┆ 36.7           ┆ 19.3          ┆ 193           ┆ 3450        ┆ female ┆ 2007 │
│ …         ┆ …         ┆ …              ┆ …             ┆ …             ┆ …           ┆ …      ┆ …    │
│ Chinstrap ┆ Dream     ┆ 55.8           ┆ 19.8          ┆ 207           ┆ 4000        ┆ male   ┆ 2009 │
│ Chinstrap ┆ Dream     ┆ 43.5           ┆ 18.1          ┆ 202           ┆ 3400        ┆ female ┆ 2009 │
│ Chinstrap ┆ Dream     ┆ 49.6           ┆ 18.2          ┆ 193           ┆ 3775        ┆ male   ┆ 2009 │
│ Chinstrap ┆ Dream     ┆ 50.8           ┆ 19.0          ┆ 210           ┆ 4100        ┆ male   ┆ 2009 │
│ Chinstrap ┆ Dream     ┆ 50.2           ┆ 18.7          ┆ 198           ┆ 3775        ┆ female ┆ 2009 │
└───────────┴───────────┴────────────────┴───────────────┴───────────────┴─────────────┴────────┴──────┘
> 

[eitsupi]

Given that reticulate already supports things like NumPy and Pandas, and Apache Arrow's importance in this space, I wondered if it would be worthwhile for reticulate to have nanoarrow as a soft dependency and support interconversion between R and Python via the Arrow C Stream interface.

[paleolimbot]

Hmm...like a blanket as_nanoarrow_array_stream() implementation for Python objects that just calls __arrow_c_stream__()? It sounds like a good idea (but also potentially a lot of work!)

[eitsupi]

Sorry for the lack of words. I'm imagining S3 methods like:
https://github.com/apache/arrow/blob/16ceade8f8b6a2720acf028ca3b6352000c60cdb/r/R/python.R#L56-L92

It seems quite excessive to install R arrow and pyarrow just to transfer data between R and Python, but it would be great if we could just need nanoarrow on the R side. (I think it is also acceptable for nanoarrow to be added to the Python side)

[paleolimbot]

Ah! We can add those methods in nanoarrow an lazily register with s3_register().

[eddelbuettel]

It seems quite excessive to install R arrow and pyarrow just to transfer data between R and Python

💯 -- and both rpy2-arrow and ryp seem to do that, and would be a lot lighterweight if they could get by with nanoarrow.

[eitsupi]

It is certainly possible to add this functionality to the nanoarrow package, like the R arrow package, but given the importance of the arrow ecosystem, I think it is worth implementing it on the reticulate side.
For example, the conversion from Python Polars to R here (rstudio/reticulate#1568) goes through pyarrow and Pandas in between, but I suspect that in reality it would only need to go through nanoarrow on the R side.