[C++] Feather: slow writing of NullArray

[asfimport]

From https://stackoverflow.com/questions/57877017/pandas-feather-format-is-slow-when-writing-a-column-of-none

Smaller example with just using pyarrow, it seems that writing an array of nulls takes much longer than an array of for example ints, which seems a bit strange:

In [93]: arr = pa.array([None]*1000, type='int64')

In [94]: %%timeit 
    ...: w = pyarrow.feather.FeatherWriter('__test.feather') 
    ...: w.writer.write_array('x', arr) 
    ...: w.writer.close() 

31.4 µs ± 464 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [95]: arr = pa.array([None]*1000)  

In [96]: arr    
Out[96]: 
<pyarrow.lib.NullArray object at 0x7fa47a23ca40>
1000 nulls

In [97]: %%timeit 
    ...: w = pyarrow.feather.FeatherWriter('__test.feather') 
    ...: w.writer.write_array('x', arr) 
    ...: w.writer.close() 

3.75 ms ± 64.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

So writing the same length NullArray takes ca 100x more time compared to an array of nulls but with Integer type.

Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Wes McKinney / @wesm

Related issues:

• [Format] Feather V2 based on Arrow IPC file format, with compression support (is a child of)

_{Note: This issue was originally created as ARROW-6529. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Confirmed here. @wesm  any thoughts?

[asfimport]

Wes McKinney / @wesm:
That's very weird. Let me have a look

[asfimport]

Wes McKinney / @wesm:
The reason is that the Feather writer is "faking" a NULL type by writing a UTF8 column with all nulls

https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/feather.cc#L615

In both R and Python this results in an all-null array when read back.

Probably the easiest quick way to fix this would be to write "user_metadata" for the Column to signal to C++ that the output should be NullArray. Longer term we need to deprecate the details in feather.fbs and make Feather a wrapper around the standard IPC protocol which is much better supported and featureful

I don't think this is urgent for 0.15.0 so I'm removing from the milestone

[asfimport]

Wes McKinney / @wesm:
This will be addressed by ARROW-5510 which replaces the custom Feather data layout with the Arrow IPC protocol

[asfimport]

Wes McKinney / @wesm:
This is a non-issue in Feather V2 in ARROW-5510 (which uses exactly the Arrow IPC file format) so we won't spend any more energy improving performance in Feather V1