Storing large data frames

[sylvaincom]

Issue

From the notebook, suppose that I load a large dataframe of shape (1M, 1k) let's say. I try to store it: it takes a lot of time.

Expectation

I would like a warning to the user that his file is too big, or maybe just store a subset of the big Dataframe (and warning that the display of the dashboard is truncated). Indeed, no one needs to visualize a dataframe of shape (1M, 1k) in the dashboard.

[tuscland]

Thank you for this interesting feedback.

For a warning, I am not sure what the right behavior would be.

So I suggest picking sane defaults:

• In put and put_item, offer control to limit dataframe size using the keyword argument dataframe_max_rows. None means no limit, default is 1000.
• The same idea could be implemented for numpy arrays (suggested keyword argument numpy_array_max_length wit default value 1000 as well).

Then an enhancement that must examined in a wider context:

• Implement metadata recording for tables as per ENH: Save context of objects in backend and leverage in UX #120 (to be defined).

[tuscland]

As suggested by @rouk1, we could think about using Perspective as a visualization tool unlocking many interesting use-cases, including the ability to store large amounts of data in a skore database..

[tuscland]

Discussed with @fcharras that suggested to have a look at DeltaLake, and parquet files.

[tuscland]

It would be nice to allow storing large dataframes so that users can share data and benefit from skore's ability to store successive versions. Data over a certain size could be stored as parquet files.