DataFrame - Add chunksize to from_records()

[achapkowski]

Code Sample, a copy-pastable example if possible

Currently the from_records does not take a chunksize.

import pandas as pd
df = pd.DataFrame.from_records(data, index=None, exclude=None, 
                                     columns=None, 
                                     coerce_float=False, 
                                     nrows=None)

Enhancement

I would like to see the chunksize options like in read_csv()

import pandas as pd
dfs = pd.DataFrame.from_records(data, index=None, exclude=None, 
                                     columns=None, 
                                     coerce_float=False, 
                                     nrows=None, chunksize=10000)
res = []
for df in dfs:
    df['col'] = 'blah'
   res.append(df)

df = pd.concat(res)

output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.10.final.0
python-bits: 32
OS: Windows
OS-release: 8
machine: AMD64
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.16.1
nose: 1.3.7
Cython: None
numpy: 1.9.2
scipy: 0.15.1
statsmodels: None
IPython: 4.2.0
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2016.4
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.4.3
openpyxl: None
xlrd: 0.9.3
xlwt: 1.0.0
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: 0.9.2
apiclient: 1.5.1
sqlalchemy: None
pymysql: None
psycopg2: 2.6.2 (dt dec pq3 ext)

[TomAugspurger]

Can you say a bit more about why this would be useful to you? Typically chunksize is used to manage larger-than-memory datasets that are on disk. The data argument to from_records is presumably already in memory.

[achapkowski]

@TomAugspurger - I can do my best. The idea of using a cursor like object with a __next__() implemented is that it allows you to go line by line through a dataset table/file, whatever. If you have a large dataset, let's say bigger than memory, then you need to limit the number of rows a user works with at one time. This means you maybe able to work row by row, but you won't get the advantage of DataFrame object, where as if we can chunk it, it will load better.

read_csv has the chunksize, but the from_records does not. It should because sometimes you need to limit the number rows that can be processed at once.

Thanks

[TomAugspurger]

Thanks, that's perfectly reasonable. The other option is to chunk your dataset before passing it to from_records, using e.g. toolz.partition_all

stream = ...
dfs = (pd.DataFrame(chunk, ...) for chunk in toolz.partition_all(chunksize, stream))

Is adding the parameter worth the extra complexity / maintenance?

[achapkowski]

@TomAugspurger - I think it would be worth it in the long run. All the supported datasets should probably allow for chunking not just a couple.

[jreback]

what is the actual use case for this? if you have a rec-array by-definition its in memory, so not sure how this helps at all.

[achapkowski]

There are other 3rd party objects out there that implement iterator and generator objects.

For example, spatial data, there is an arcpy.da.SearchCursor. This object is your standard iterator object over a table, but I may not want to load the whole table into memory at once to do my work, but rather, I want to piece it together much like with csv files.

The concept here is to process large generator/iterators into smaller pieces to be more efficient when it comes to memory consumption.

[jreback]

and u can easily do that

why should pandas add this very narrow case

[achapkowski]

@jreback because iterators and generators are common ways to get data and it allows a more generic method to load data into dataframes.

Can you provide guidance on how to implement it if it is very easy?

[jreback]

@achapkowski this would be extremely inefficient, but I suppose a usecase exists.

I said its easy to do externally.

pd.concat(list(generator))

[achapkowski]

@jreback but aren't you just loading the whole dataset into memory then?

[jreback]

of course. a Dataframe is a fixed size. Expanding it requires reallocattion and copying, which is quite expensive.

[jreback]

you can see #5902 if you want to see the discussion

[achapkowski]

But the chunksize, like in the csv file returns an iterator of dataframe of size x, why couldn't that be done for and iterator object?