You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
On Side Panel, variable visualizer,
When clicking on large dataframe or large dictionnary, Panel and Spyder freezed.
Suggest the workaround :
nmax= 50000
On Click Visualize( dfbig.sample( n = min( len(dfbig, max_df)) , replace=False )
Suppose len(dfbig) = 1 million ...
It will sample the dataframe with nmax= 50000 values. and Spyder does not crash...
Same for list
On Click Visualize ( listbig[:nmax. )
Does it handle visualization of
10 million rows with 560 columns in string ?
That depends on the amount of memory available in your computer, not on Spyder. That's because we need to make a copy of the dataframe in the IPython console kernel to send and display it in Spyder (which runs in a different process).
believe sub-sampling is simple and efficient way to reduce load in visualization ...
Sure, this is a good idea too. Thanks for the suggestion, I didn't know about it. We'll try to implement it in Spyder 6.
ccordoba12
changed the title
Feature: Variable Explorer : reduce amount of data visualized for DataFrame by sampling
Reduce amount of data for DataFrames by sampling
Jun 13, 2023
Thanks for considering it.
Think visualizing 1 million rows table does not make much sense for human...
At max 100,000 rows would handl most use visualization use cases ( ie find pattern, wrong columns)
and reduce memory footprint a lot.
On Side Panel, variable visualizer,
When clicking on large dataframe or large dictionnary, Panel and Spyder freezed.
Reference:
https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.sample.html
Thanks !
The text was updated successfully, but these errors were encountered: