PERF: Load data (create Series, Dataframe) in a more functional way.

[tinproject]

I've been trying to find out the process of creating DataFrames in order to try to solve #2305 with minimal memory use. I've made some tests that put in a IPython notebook: http://nbviewer.ipython.org/gist/tinproject/7d0e0de9475b16910fcf

Currently when reads from a generator pandas internally allocate the whole generator in memory within a list, using at least twice the memory needed. I suspect sometimes could be even more if there is a later type conversion needed.

Also, as it can be viewed in the notebook, the input data collection it's completely readed many times before data it's loaded in the final data structure.

To read from an iterator, without put the whole it in memory, it's needed to process the values that yield one by one, or at least in a one pass form.

One it could read data from a iterator in one pass, it could read from every iterable collection only needing the count of elements to read. It includes collections with implicit lenght: sequences(list, tuple, str), sets, mappings, etc. and collections with unknown length: generators, iterators, etc. which size must be explicitly given.

I think this could end in a great improve on performance and enhance the input data types accepted (In my tests I saw that the tuple collection it's not a valid type for DataFrame!)

Relates to #2305, #2193, #5898

I want to help to solve this, but it's a bis task, and I'm lost when comes to the Blocks part. Is there anywhere documented what structure have and how Blocks works?

[jreback]

can you give an actual use case of this?

are you trying to create really large frames?

and if you are, then why would you not write a disk-based storage, say HDF and then read chunks as needed?

[tinproject]

Load data efficiently from a remote endpoint, a database query for example, you have a iterator (cursor, pointer to data) and how many values the query have. The same with an HDF file. Iterator to elemets and count of elements (cursor tpye) could be consider as an universal endpoint to any kind of external library

The problem originally arise when I try to load some data on my 4GB machine. I read a big file with a generator and try to load it directly on pandas, I can't.
I solve this creating an empty numpy.ndarray and loading the data of my generator directly on the array. Then pass the numpy.ndarray to pandas.

Currently we have better machines but it'a a nosense not to use it efficiently.

[jreback]

ok, so you want to read say a 1GB of data (say that's what the frame ends up being). But right now you say it say takes 2GB to create. ok prob true. What happens if you magically create it for only 1GB then say perform an operation with it (saydf+1), which creates another object of 1GB? You have exactly the same problem.

The way to do this is to operate out-of-core. I didn't mean iterator over HDF per se, but rather

In chunks, create an intermediate store from your generator (doesn't have to be HDF but generally the most efficient). Then operate on those chunks.

Doesn't matter how much memory you have; when you have a dataset that is a large fraction of your memory and you want to do actual operations your are forced to do it this way.

I think you can attack the iterator problem in a very similar way.

Process a chunk of data, form to a frame, repeat, then concat these at the end. I think a way to do this in pandas would be nice; I also think that doing this in the constructor is not intuitive (well you could do it internally I suppose, but you maybe need a more sophisticated way of doing this, including additional arguments, not only the count of the generator, but a place for intermediate storage, etc.)

so you could do something like this:

df = DataFrame(data=GeneratorObject(src, addtitional arg))...

where the GeneratorObject is a pandas object that could handle the construction via chunks

[jreback]

see #3202 which is a template for doing this (a GeneratorObject could be a form of this actually)

[tinproject]

I'm not thinking on any particular data, the goal it's to solve the bug and improve pandas.

You are rigth, it's always needed space for operations but I'm only talking about loading data. When load chunked data that it's not in memory you always need twice the memory, half for the chunks dataframes, half for the final dataframe.
Currently pandas only load data from memory to memory (previously allocated, externally or internally). Which I try to express it's load data from x directly to memory, x being disk, network, memory or whatever.

This approach may help to load the data from the pipeline without the need to store the whole tank first.

The way of loading data in pandas internally it's column base, if wants to load data from an element/row base (iterator) efficiently it might change the way of doing it.

I'll start with the simplest case, a Series without index, it loads 0-dimensional data (elements) from a 1-dimensional collection.
Elements can come in one type of data and pandas store in another type, that could be explicitly expressed (dtype) or infered.

The data collection could be of one type:

• Iterable of known length:
  • Sequence (list, tuple, str..), numeric index access
  • Sets, unordered access (but one arbitrary order for the iterator)
  • Mappings (dicts..), key access
• Iterable of unknown length (must set explicitely):
  • Generator, Iterator..
• Non iterable
  • It's there any collection object in python?

To load the data could do the following:

1. Take an iterator to the data
2. How many elements are?

When read data from a unknown length collection it must be explicitely given. If not expressed takes it with len(), if collection is sized. Otherwise there it's no other solution than read the whole data to an intermediary list.
3. Which type the elements are? It will be stored in some other type?

This could be explictely indicated with dtype property. If not it have to infer it. It could be done by reading the first element of the collection. As it trying to do this in one pass, it assume that the type of the first element is going to be compatible with the rest elements.
4. Allocate the memory needed for the array and store the first element.

This array it's at first empty (np.empty for example), or na filled.
5. Store the rest of colection in data structure.

Element type could be tracked without store the whole collection in memory, if someone its from a different type it could raise a exception, treat it like a na value, or reallocate the data in a new array with the new type before move to the next element.
6. Fit data structure.

If the iterator exhausted before reach the number of elements of count, resize the array to the correct size

This structure can be applied to whatever kind of data collection. The question it's to do the checks and data loading in a flip way, by elements (rows) rather than columns.

This example can be consider the easiest case, the harder case it's when a DataFrame reads from a mapping iterator.

[jreback]

you are welcome to try to implement a solution

you can't change the API and must pass all current tests

lmk if u have questions

[tinproject]

@jreback Could you point me to any kind of doc/explanations of how internals/Block API works?

[jreback]

No docs. You just have to step thru creation code. Here's a mini-version.

BlockManagers hold blocks that are divided by dtype. A block is a container that holds items which label the 0'th dimension of a block; ref_items are reference (e.g. by take) back to where these livie in the BlockManager. A block generally has 2 or more dimensions (even in the case of a Series); a SparseBlock is the exception. The 0th dimension in block is the info_axis (e.g. columns for a Frame, they are 'backwards'). Panels have 3-d blocks. Again the 0th is are the items (the info_axis).

Block are created then merged/consolidated. Very rarely are they actually modified in-place. When you see an in-place operation it atually is creating a new block manager (possibly with the same blocks) and replace in the top-level object. (thats the ._data attribute).

Their are some non-trivial things going on, esp when you have duplicate items which are the whole _ref_locs bizness.

You just have to spend some time with it tracing flow.