Regression in Tabulator.selected_dataframe for panel>0.12.1

[grelston]

See my Tabulator-regression repo for the MRE, conda environments, and animated gif screenshots.

ALL software version info

OS:

• Windows 10
• version 1909
• build 18363.1679

conda environment:

• bokeh=2.4.1
• ipython=7.29.0
• jupyter_client=7.0.6
• jupyter_core=4.9.1
• notebook=6.4.5
• openpyxl=3.0.9
• pandas=1.3.4
• panel=0.12.4 (or 0.12.3 or 0.12.2)
• param=1.12.0
• python=3.9.7

name: panel-0.12.4
channels:
  - pyviz
  - conda-forge
  - defaults
dependencies:
  - argon2-cffi=21.1.0
  - async_generator=1.10
  - attrs=21.2.0
  - backcall=0.2.0
  - backports=1.0
  - backports.functools_lru_cache=1.6.4
  - bleach=4.1.0
  - bokeh=2.4.1
  - brotlipy=0.7.0
  - ca-certificates=2021.10.8
  - certifi=2021.10.8
  - cffi=1.15.0
  - chardet=4.0.0
  - colorama=0.4.4
  - cryptography=35.0.0
  - debugpy=1.5.1
  - decorator=5.1.0
  - defusedxml=0.7.1
  - entrypoints=0.3
  - et_xmlfile=1.0.1
  - freetype=2.10.4
  - idna=2.10
  - importlib-metadata=4.8.1
  - importlib_resources=5.4.0
  - intel-openmp=2021.4.0
  - ipykernel=6.4.2
  - ipython=7.29.0
  - ipython_genutils=0.2.0
  - jbig=2.1
  - jedi=0.18.0
  - jinja2=3.0.2
  - jpeg=9d
  - jsonschema=4.2.1
  - jupyter_client=7.0.6
  - jupyter_core=4.9.1
  - jupyterlab_pygments=0.1.2
  - lcms2=2.12
  - lerc=3.0
  - libblas=3.9.0
  - libcblas=3.9.0
  - libdeflate=1.8
  - liblapack=3.9.0
  - libpng=1.6.37
  - libsodium=1.0.18
  - libtiff=4.3.0
  - libzlib=1.2.11
  - lz4-c=1.9.3
  - markdown=3.3.4
  - markupsafe=2.0.1
  - matplotlib-inline=0.1.3
  - mistune=0.8.4
  - mkl=2021.4.0
  - nbclient=0.5.4
  - nbconvert=6.2.0
  - nbformat=5.1.3
  - nest-asyncio=1.5.1
  - notebook=6.4.5
  - numpy=1.21.4
  - olefile=0.46
  - openjpeg=2.4.0
  - openpyxl=3.0.9
  - openssl=1.1.1l
  - packaging=21.0
  - pandas=1.3.4
  - pandoc=2.16.1
  - pandocfilters=1.5.0
  - panel=0.12.4
  - param=1.12.0
  - parso=0.8.2
  - pickleshare=0.7.5
  - pillow=8.3.2
  - pip=21.3.1
  - prometheus_client=0.12.0
  - prompt-toolkit=3.0.22
  - pycparser=2.21
  - pyct=0.4.8
  - pyct-core=0.4.8
  - pygments=2.10.0
  - pyopenssl=21.0.0
  - pyparsing=3.0.5
  - pyrsistent=0.18.0
  - pysocks=1.7.1
  - python=3.9.7
  - python-dateutil=2.8.2
  - python_abi=3.9
  - pytz=2021.3
  - pyviz_comms=2.1.0
  - pywin32=302
  - pywinpty=1.1.5
  - pyyaml=6.0
  - pyzmq=22.3.0
  - requests=2.25.1
  - send2trash=1.8.0
  - setuptools=58.5.3
  - six=1.16.0
  - sqlite=3.36.0
  - tbb=2021.4.0
  - terminado=0.12.1
  - testpath=0.5.0
  - tk=8.6.11
  - tornado=6.1
  - tqdm=4.62.3
  - traitlets=5.1.1
  - typing_extensions=3.10.0.2
  - tzdata=2021e
  - ucrt=10.0.20348.0
  - urllib3=1.26.7
  - vc=14.2
  - vs2015_runtime=14.29.30037
  - wcwidth=0.2.5
  - webencodings=0.5.1
  - wheel=0.37.0
  - win_inet_pton=1.1.0
  - winpty=0.4.3
  - xz=5.2.5
  - yaml=0.2.5
  - zeromq=4.3.4
  - zipp=3.6.0
  - zlib=1.2.11
  - zstd=1.5.0
prefix: C:\Users\User\.conda\envs\panel-0.12.4

Description of expected behavior and the observed behavior

Expected: When filtering a table and selecting rows from the filtered subset, the Tabulator.selected_dataframe returns those selected rows for panel=0.12.1.

Observed: For panel>=0.12.2 it throws an IndexError.

Complete, minimal, self-contained example code that reproduces the issue

See the Tabulator-regression.ipynb notebook (nbviewer or GitHub), which is based on the example in the Function based filtering section of the Tabulator page of the Reference Gallery.

Stack traceback and/or browser JavaScript console output

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
~\.conda\envs\panel-0.12.4\lib\site-packages\pandas\core\indexing.py in _get_list_axis(self, key, axis)
   1529         try:
-> 1530             return self.obj._take_with_is_copy(key, axis=axis)
   1531         except IndexError as err:

~\.conda\envs\panel-0.12.4\lib\site-packages\pandas\core\generic.py in _take_with_is_copy(self, indices, axis)
   3627         """
-> 3628         result = self.take(indices=indices, axis=axis)
   3629         # Maybe set copy if we didn't actually change the index.

~\.conda\envs\panel-0.12.4\lib\site-packages\pandas\core\generic.py in take(self, indices, axis, is_copy, **kwargs)
   3614 
-> 3615         new_data = self._mgr.take(
   3616             indices, axis=self._get_block_manager_axis(axis), verify=True

~\.conda\envs\panel-0.12.4\lib\site-packages\pandas\core\internals\managers.py in take(self, indexer, axis, verify)
    861         n = self.shape[axis]
--> 862         indexer = maybe_convert_indices(indexer, n, verify=verify)
    863 

~\.conda\envs\panel-0.12.4\lib\site-packages\pandas\core\indexers.py in maybe_convert_indices(indices, n, verify)
    291         if mask.any():
--> 292             raise IndexError("indices are out-of-bounds")
    293     return indices

IndexError: indices are out-of-bounds

The above exception was the direct cause of the following exception:

IndexError                                Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_172632/1318812073.py in <module>
      1 # This fails for panel>0.12.1:
      2 if demo:
----> 3     fill_tbl2(True)

~\AppData\Local\Temp/ipykernel_172632/1200927601.py in fill_tbl2(event)
      4     df_new = (
      5         df2.append(
----> 6             other=tbl.selected_dataframe,
      7             ignore_index=True
      8         )

~\.conda\envs\panel-0.12.4\lib\site-packages\panel\widgets\tables.py in selected_dataframe(self)
    546         if not self.selection:
    547             return self._processed
--> 548         return self._processed.iloc[self.selection]
    549 
    550 

~\.conda\envs\panel-0.12.4\lib\site-packages\pandas\core\indexing.py in __getitem__(self, key)
    929 
    930             maybe_callable = com.apply_if_callable(key, self.obj)
--> 931             return self._getitem_axis(maybe_callable, axis=axis)
    932 
    933     def _is_scalar_access(self, key: tuple):

~\.conda\envs\panel-0.12.4\lib\site-packages\pandas\core\indexing.py in _getitem_axis(self, key, axis)
   1555         # a list of integers
   1556         elif is_list_like_indexer(key):
-> 1557             return self._get_list_axis(key, axis=axis)
   1558 
   1559         # a single integer

~\.conda\envs\panel-0.12.4\lib\site-packages\pandas\core\indexing.py in _get_list_axis(self, key, axis)
   1531         except IndexError as err:
   1532             # re-raise with different error message
-> 1533             raise IndexError("positional indexers are out-of-bounds") from err
   1534 
   1535     def _getitem_axis(self, key, axis: int):

IndexError: positional indexers are out-of-bounds

Screenshots or screencasts of the bug in action

Succeeds for panel=0.12.1

Fails for panel>=0.12.2

[grelston]

Selecting and copying filtered rows succeeds for panel=0.12.1

[grelston]

Selecting and copying filtered rows fails for panel>=0.12.2

[grelston]

Issue #2505 reports the same IndexError but gives no further details.

[maximlt]

Hi @grelston! Thanks for the effort you put in reporting that issue, that's fantastic.

I've been able to reproduce the bug with the last dev release of panel. Basically:

1. Select some rows
2. Filter the data so that some selected rows are no longer displayed => At this stage .selection still contains the indices of the selected rows
3. Call .selected_dataframe which is going to do an .iloc with .selection on ._processed, except that ._processed contains the filter dataframe

[grelston]

Thanks for looking at this @maximlt and @philippjfr.

I just uploaded the Tabulator-regression notebook into JupyterLab on binder with panel master (panel=0.13.0a20+g137c8733-dirty) and I see the same IndexErrors:

[snip]
IndexError: indices are out-of-bounds

The above exception was the direct cause of the following exception:

[snip]
IndexError: positional indexers are out-of-bounds

Do we reopen this issue? Or shall I open a new issue and refer to this one?

[maximlt]

I could reproduce this bug but the important factor apparently is that pagination has to be set to 'remote'. Thanks @grelston, reopening the issue then.

[maximlt]

It's not exactly the same issue, to reproduce:

1. Filter the data
2. Select a row
3. Access the selected_dataframe property.

[grelston]

Thanks for reopening this, @maximlt.

I've just tested the pagination='local' parameter (with panel=0.13.0a31) and I still get the same stack trace. I've pushed the rerun notebook to my Tabulator-regression repo.

[maximlt]

There's some strange behavior going on. When I run all the cells of your notebook (Run all cells command) with either 'local' or 'pagination' for remote I also get the IndexError. However, if I run the cells manually, step by step, I don't.

If, instead of running the whole notebook, I manually try to reproduce the bug (as you did in the videos above), I only get the IndexError with pagination='remote'.

I can't explain the strange behavior I observed in the notebook without any further investigation I can only say that I expect this kind of approach, i.e. setting programmatically filters and selection, to be pretty rare. Also the notebook is using a private method _filter_dataframe().

[grelston]

I've looked at this again in more detail, comparing 2 states of 3 variables (experimental, not code):

variable	value 1	value 2
panel=	0.12.1	0.13.0a31
pagination=	'remote'	'local'
UI driver	Cell > Run All	GUI / manual / mouse

There are in fact 2 bugs / behaviour changes that are showing up under different circumstances:

1. GUI interaction problem
2. Code to replicate the GUI interaction problem

I used the private _filter_dataframe() method because there was no current_view property in panel=0.12.1 and I wanted to achieve in code the same "select-all-visible" behaviour shown when selecting the header checkbox in the GUI.

1 GUI interaction problem

First, the GUI interaction problem as reported and illustrated when I first raised this issue.

1.1 panel=0.12.1

With panel=0.12.1, all 4 combinations of pagination and "UI driver" generate the desired outcome (5 rows in tbl2, with code=['00', '01', '03', '07', '08']). However, there are differences in the visible selection state of the first table in the GUI (tbl).

With pagination='remote', the GUI shows that all 3 filtered rows ('03', '07', '08') are selected; checking tbl.selection also shows [3, 7, 8] (see updated notebook and this screenshot in the Tabulator-regression repo). This is true for both methods of driving the UI ("notebook > Cell > Run All" and GUI/manual/mouse).

With pagination='local', the GUI shows that none of the 3 filtered rows are selected after driving the UI with the "notebook > Cell > Run All" (see this screenshot). Checking tbl.selection after GUI/manual/mouse filter-select shows [0, 1, 2] (see this screenshot).

1.2 panel=0.13.0a31

I'm going to have to come back to this later, sorry.

[maximlt]

@grelston I'm working on a fix to this issue. There are a few inconsistencies I'd like to fix:

• selection should always return the indices (numbers) of the original dataframe
• current_view should always return the state of the data currently "seen" by the user, so possibly sorted and filtered. "seen" because when remote is set it would not just return one page (I don't think that would be very useful)

When this is fixed I think we can try to run your notebook again and see if there are remaining issues.