[Format] Feather V2 based on Arrow IPC file format, with compression support #21958
Description
The initial Feather file format is a minimal subset of the Arrow IPC format. It has a number of limitations (see https://wesmckinney.com/blog/feather-arrow-future/).
We want to retain "feather" as the name of the on-disk representation of Arrow memory, so in order to support everything that Arrow supports, we need a "feather 2.0" format.
IIUC, defining the file format is "done" (dump the memory to disk). Remaining issues include upgrading "feather" readers and writers in all languages to support both feather 1.0 and feather 2.0. (e.g. https://issues.apache.org/jira/browse/ARROW-5501)
Reporter: Neal Richardson / @nealrichardson
Assignee: Wes McKinney / @wesm
Related issues:
- [C++] Feather: slow writing of NullArray (is a parent of)
- [C++/Python] Feather format cannot accommodate string columns containing more than a total of 2GB of data (is related to)
PRs and other links:
Note: This issue was originally created as ARROW-5510. Please see the migration documentation for further details.