Return PyArrow Scalars instead of Python objects

[samgd]

We have a use case where there are large (TiB) tables on disk. These tables are loaded via a memory map to avoid reading them all into memory.

Individual elements in certain columns in these table can be non-trivially large (MiB-GiB++). e.g. long lists of values. Typically only small slices of these lists are actually needed at runtime. Currently fletcher forces fetched elements to Python objects thus the entire memory-mapped list is fetched into memory and then returned to the user for slicing.

Is it possible for the ExtensionArray implementation to instead return the PyArrow [List]Scalar object thereby enabling efficient slicing without forcing the entire object into memory?

[xhochy]

This project has been archived as development has ceased around 2021.
With the support of Apache Arrow-backed extension arrays in pandas, the major goal of this project has been fulfilled.