Reduce amount of data for DataFrames by sampling

[uprokevin]

On Side Panel, variable visualizer,
When clicking on large dataframe or large dictionnary, Panel and Spyder freezed.

Suggest the workaround : 
    nmax= 50000
    On Click Visualize(   dfbig.sample( n = min( len(dfbig, max_df))    , replace=False ) 

Suppose len(dfbig) = 1 million ...
It will sample the dataframe with nmax= 50000 values. and Spyder does not crash...

Same for list 
    On Click Visualize (    listbig[:nmax. ) 

Reference:
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.sample.html

Thanks !

[ccordoba12]

Hey @pprokevin, thanks for reporting. Could you post a video or animated gif that shows Spyder freezing after opening a big dataframe or dictionary?

I just tested with a one million row/single column dataframe, and Spyder didn't freeze for me.

[uprokevin]

Does it handle visualization of
10 million rows with 560 columns in string ?

believe sub-sampling is simple and efficient way to reduce load in visualization ...

[ccordoba12]

Does it handle visualization of
10 million rows with 560 columns in string ?

That depends on the amount of memory available in your computer, not on Spyder. That's because we need to make a copy of the dataframe in the IPython console kernel to send and display it in Spyder (which runs in a different process).

believe sub-sampling is simple and efficient way to reduce load in visualization ...

Sure, this is a good idea too. Thanks for the suggestion, I didn't know about it. We'll try to implement it in Spyder 6.

[uprokevin]

Thanks for considering it.
Think visualizing 1 million rows table does not make much sense for human...
At max 100,000 rows would handl most use visualization use cases ( ie find pattern, wrong columns)
and reduce memory footprint a lot.