[C++/Python] Feather format cannot accommodate string columns containing more than a total of 2GB of data

[asfimport]

This can be fixed by migrating Feather to using the Arrow file/stream format internally. As soon as we have R bindings for Arrow, we can do this

See wesm/feather#325

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

Related issues:

• [Format] Feather V2 based on Arrow IPC file format, with compression support (relates to)

_{Note: This issue was originally created as ARROW-1907. Please see the migration documentation for further details.}

[asfimport]

Sritanu Chakraborty:
What's the current state of this issue? I am trying to convert a pandas dataframe of 10 Gb(approx) to feather. I am getting the same error:

TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Array

 

[asfimport]

Wes McKinney / @wesm:
Now that the R library is available on CRAN, tackling this issue should be a little bit easier

[asfimport]

Neal Richardson / @nealrichardson:
Do ARROW-3308 and ARROW-3317 cover this, or is there more involved?

[asfimport]

Wes McKinney / @wesm:
This will be handled by the Feather V2 project (since the IPC protocol will be used to represent the data on disk)