Add pd.read_bin() to read data of binary file to dataframe.

[halleygithub]

Create pd.read_bin(binary_file, record_fmt) to wrap Python struct.unpack(), facilitate the data exchange with C binary data file.

[cpcloud]

You can do this easily with record arrays and numpy.fromfile. For example, let's say you have a file that's made up of an array of C structs that look like this

typedef struct Data
{
    int8_t *name;
    int32_t data1;
    double data2;
} Data;

(Your file is essentially an fwrite of something that looks like Data struct_array[20];, for example)

from numpy import fromfile, dtype
from pandas import DataFrame
dt = dtype([('name', 'S'), ('data1', 'int32'), ('data2', 'float64')])
records = fromfile('the_binary_file.dat', dt)
df = DataFrame(records)

This is darn fast, and there's no need to use the struct module here.

The implementation would probably look something like this

def read_bin(filename, rec_dtype):
    return DataFrame(fromfile(filename, rec_dtype))

IMHO there's no point in supporting this if you can already do it in a single concise line of code.

[halleygithub]

Thanks for the tip. Even though it is a simple function, but including the read_bin() in Pandas will unify the data reading format, make- up the I/O capability to talk with binary format file which Pandas lacks before.That will be a good handy function than using numpy fromfile and create a user function.

Since Pandas has read_csv, read_Excel.., it is nature to expect read_bin exist there.

[jreback]

@halleygithub how about a cookbook entry?

[cpcloud]

Just to clarify one thing: For whatever reason (probably my own stupidity), I can't get this to work using any 32bit type (float or int32_t), but 64 bit types work ... i would guess this is because of padding

[halleygithub]

@jreback, when an amateur programmer like me want to read data from bin-file, normally they will expect Pandas to provide read_bin() already, just like read_csv() or read_stata() etc. If not, they will resort to numpy.fromfile or struct.unpack sooner or later, but that will cause them some efforts to search or learn. IMHO, not seeing read_bin there besides read_csv, read_excel always makes me feel pandas miss something.

[jreback]

@halleygithub

it is one thing to simple write the frames values to a file as possibly a record array
but quite another to properly support this - meaning to provide a round trippable serialization format

I think u would be better off IMHO using HDFStore or msgpack. (available in 0.13)

proper serialization to a binary format is hard when done in a portable reproducible way

[halleygithub]

Understood. Thanks for your information.

[cpcloud]

I'll add an example to the cookbook

[cpcloud]

Turns out that the 32 bit type issue was padding after all.

[rspadim]

that's not a good solution, maybe we could improve this, i'm using cython...

    dadosnp=np.fromfile(FileName, dtype=dt)

95.3 µs ± 3.07 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [11]:

    dadospd=pd.DataFrame(np.fromfile(FileName, dtype=dt))

5.17 ms ± 197 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [ ]:

numpy magic is here: https://github.com/numpy/numpy/blob/5f01e54b20e38d483e8bab31bf5f953a860fe8d3/numpy/core/records.py#L786