df.info() incorrectly reports memory use for string data?

[kay1793]

Doubling the string size does not change the reported memory use:

In [64]: d=pd.DataFrame(["%06d" %x for x in range(1000000)])
    ...: d.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000000 entries, 0 to 999999
Data columns (total 1 columns):
0    1000000 non-null object
dtypes: object(1)
memory usage: 15.3 MB

In [65]: d=pd.DataFrame(["%012d" %x for x in range(1000000)])
    ...: d.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000000 entries, 0 to 999999
Data columns (total 1 columns):
0    1000000 non-null object
dtypes: object(1)
memory usage: 15.3 MB

[jreback]

nope, this is how it is. It is just a pointer. which is always 8 bytes. I believe that python interns the actual strings for smaller ones, and allocates from the heap for bigger. Though not really sure.

[kay1793]

That is very misleading. It must be possible to calculate the memory footprint of native python objects.

cc @jorisvandenbossche

[jreback]

see discussion in #7619
if u can find a reliable cross version way of doing this without sacrificing perf I am all ears

[kay1793]

ok:

@jreback:

The reason is the 'real' storage for strings is NOT the object pointer, but rather 
a fixed-width storage block of memory depending on the length of strings.

This is a teeny bit wrong for actual objects

"a teeny bit wrong" ;)
it may be complicated to estimate precise memory but even an estimate would be more "reliable".

@shoyer:

 For now, I would just warn that memory usage will be an underestimate for object dtype.

Maybe its much work to do better but don't you think the warning is good idea?

why does performance of info() matter, memory report could be off by default if there are object dtypes. is it better to give a wrong answer quickly?

[jreback]

@kay1793 you are welcome to provide a soln if you would like

performance of info is extremely crucial
if I have 1m rows and every time I display this then it's perceptibly slow that is a problem

[kay1793]

@jreback how about this solution:

if object dtype is present, the memory usage reported gets an asterisk and warning text added at bottom.
if a better memory estimate is possible but slow it'll be off by default but user can request it with keyword to info().

Would that be ok?

[shoyer]

Let's see just how slow it is :).

But I don't think there is any option other than Cython here.

Pure Python will be impossibly slow for iterating over large enough arrays for memory usage to be relevant, but the numpy solution will double memory usage -- which makes the command almost useless, since it is there particularly for debugging and understanding in low memory situations.

On Saturday, Oct 18, 2014 at 11:16 PM, kay1793 notifications@github.com, wrote:

@jreback how about this solution:

if object dtype is present, the memory usage reported gets an asterisk and warning text added at bottom.

if a better memory estimate is possible but slow it'll be off by default but user can request it with keyword to info().

Would that be ok?

—
Reply to this email directly or view it on GitHub.

[kay1793]

Here you go @jreback, it was much easier than you said.

courtesy of blosc/bcolz 's @esc from:
http://nbviewer.ipython.org/github/Blosc/movielens-bench/blob/master/querying-ep14.ipynb

In [48]: d=pd.DataFrame(["%06d" %x for x in range(1000000)])
    ...: print("%d MB" % (sum(sys.getsizeof(o) for o in d[0].values.flat)/ 2**20))
    ...: d=pd.DataFrame(["%12d" %x for x in range(1000000)])
    ...: print("%d MB" % (sum(sys.getsizeof(o) for o in d[0].values.flat)/ 2**20))
    ...: d=pd.DataFrame([u"%06d" %x for x in range(1000000)])
    ...: print("%d MB" % (sum(sys.getsizeof(o) for o in d[0].values.flat)/ 2**20))
41 MB
46 MB
72 MB

# compare info() which gives the same wrong size for all of them
In [51]: d.info()
<...>
memory usage: 15.3 MB

# for 1M objects
In [49]: %timeit sum(sys.getsizeof(o) for o in d[0].values.flat)/ 2**20
1 loops, best of 3: 290 ms per loop

[jreback]

well that is nice for a SINGLE column. What if you have 100? (and assume we only do the strings, you cannot do df.values as that turns everything to object dtype).

Don't get me wrong I think this would be nice,
but you cannot wait 20s for a result on an interactive display

and this can possibly way overstimate if things are interned.

You are welcome to put together something to do this.

[jreback]

@kay1793 ok, will reopen to see if anything can be done for this. But not sure of an easy way to surmount the perf issue here (though cytonizing may be enough).

sizeof is a c-operator, so this may be pretty easy at the cython level.