[Feature]: Add support for Data frame state being restored

[schloerke]

Component

UI (ui.*)

Severity

P2 - Medium (workaround exists)

Shiny Version

1.4.0

Minimal Reproducible Example

Related: #1952

If Shiny inputs components are serialized in any of the cells, they will not be properly serialized for bookmarking as their state will not be captured.

Behavior

Currently the data frame is not restored at all. This is due to timing issues in when the state can be set.

In the Data Frame situation, the CellValue is the original Tags. In the ui.Chat, the messages have already been serialized.

In both cases, they will lose server-side html-dependencies that do not already exist.

---

For the updated cell patch map... If the ui.output_data_frame was updated to include restore context input values (similar to ui.Chat PR), the state could be restored within Javascript.

When trying to restore the selected cells, currently viewed rows, and other data frame information, this should be handled within ui.output_data_frame setting a JS state.

[vnijs]

Question: @schloerke Does this currently not work? I'm trying to figure out how to save state to server with a dataframe involved but I'm not yet seeing how to do this (see Discord post below).

https://discord.com/channels/1109483223987277844/1361208509886238872/1361803222980755586

On a related note, my ideal scenario for saving app state would be to combine all information into one dictionary and pickle (dill?) that so a user can restore any particular session they want in the future. Do you see that as a (future) option?

[schloerke]

Does this currently not work?

Data frames currently do not restore any column filter, column sort, row selection, or edited cells state. (... nothing is currently being restored 😞 )

On a related note, my ideal scenario for saving app state would be to combine all information into one dictionary and pickle (dill?) that so a user can restore any particular session they want in the future. Do you see that as a (future) option?

We strongly recommend that you use JSON as the intermediate storage format as pickle files are not python version safe. Meaning, when you update from 3.12 to 3.12, all of your files will not work. JSON will be able to read regardless of the python version.

I'm trying to figure out how to save state to server with a dataframe involved but I'm not yet seeing how to do this (see Discord post below).

When running your app, I get the error:

Error bookmarking state: Type is not JSON serializable: DataFrame

Please make the data into shiny.types.Jsonifiable data structures, not original pandas objects.

Updated bookmarking code within server that saves the data frames to dictionaries oriented as records and restoring from that shape:

    @session.bookmark.on_bookmark
    def _(state: BookmarkState) -> None:
        state.values["datasets"] = {
            key: value.to_dict(orient="records") for key, value in datasets().items()
        }
        state.values["available_datasets"] = available_datasets()

    @session.bookmark.on_restore
    def _(restore_state: RestoreState) -> None:
        if "available_datasets" in restore_state.values:
            available_datasets.set(restore_state.values["available_datasets"])
            tmp = available_datasets.get()
            ui.update_select("select_data", choices=tmp, selected=tmp[0])

        if "datasets" in restore_state.values:
            print(f"Restoring datasets: {restore_state.values['datasets']}")
            print(pd.DataFrame.from_records(restore_state.values["datasets"]["A"]))
            datasets.set(
                {
                    key: pd.DataFrame.from_records(value)
                    for key, value in restore_state.values["datasets"].items()
                }
            )

but I'm not yet seeing how to do [bookmark the state]

The crux of the problem...

If we add

    @session.bookmark.on_restored
    async def _(restore_state: RestoreState) -> None:
        if "selected_data_column_sort" in restore_state.input:
            selected_data_column_sort = restore_state.input["selected_data_column_sort"]
            await selected_data.update_sort(selected_data_column_sort)

... to try to delay the setting of the sort state as long as possible, a silent error is thrown.

This is hard to debug and annoying to find. It greatly helps to add a raise e here:

py-shiny/shiny/bookmark/_bookmark.py

Line 446 in 4d7a3f5

This is caused by the data frame wanting to set reactive values according to the input values received from the browser. Timing is wrong and they're attempting to work before they should.

Unfortunately, I do not know of a reasonable way to restore the data frame's input values until we sit down to take the time to do it. 😞. I still believe the best approach to restore simple data frame input values is via JS state within ui.output_data_frame().