BUG: The quantile method throws exceptions in some scenarios

[DiDeoxy]

Modin version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest released version of Modin.

• I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow this guide.)

Reproducible Example

from itertools import product
from random import choices
from string import ascii_letters, digits
from typing import Sequence, cast

from modin.pandas import DataFrame, concat  # type: ignore
from numpy.random import rand
from pandas import show_versions

# Replacing modin.pandas with pandas will make the code run without errors
# from pandas import DataFrame, concat

print(show_versions())

n_rows = 10
n_fcols = 10
n_mcols = 5


def str_generator(size: int = 5, chars: str = ascii_letters + digits) -> str:
    """Create a `str` made of random letters and numbers."""
    return "".join(choices(chars, k=size))


df1 = DataFrame(rand(n_rows, n_fcols), columns=[f"feat_{i}" for i in range(10)])
df2 = DataFrame({str_generator(): [str_generator() for _ in range(n_rows)] for _ in range(n_mcols)})
df3 = cast(DataFrame, concat([df2, df1], axis=1))

q_sets = [0.25, (0.25,), [0.25], (0.25, 0.75), [0.25, 0.75]]
col_sets = [cast(Sequence[str], df1.columns.to_list()), None]  # type: ignore

for qs, cols in product(q_sets, col_sets):
    try:
        quants = (df3 if cols is None else df3[cols]).quantile(qs, numeric_only=cols is None)  # type: ignore
        print(quants)
    except Exception:
        print(type(quants))
        print(qs, cols)


# From above the following scenarios fail:
print(1, df3.quantile(0.25, numeric_only=True))  # type: ignore
print(2, df3.quantile((0.25,), numeric_only=True))  # type: ignore
print(3, df3.quantile((0.25, 0.75), numeric_only=True))  # type: ignore
print(4, df3[df1.columns].quantile((0.25, 0.75)))  # type: ignore

# Surprisingly these do not fail:
print(5, df3[df1.columns].quantile(0.25))  # type: ignore
print(6, df3[df1.columns].quantile((0.25,)))  # type: ignore

Issue Description

Several scenarios cause various errors with the quantile method. The errors are primarily caused by the kind of values supplied to the q parameter but the numeric_only parameter also plays a role. The specific scenarios leading to errors are highlighted in the reproducible code example above.

Expected Behavior

I expect when a sequence of floats are provided to the q parameter of the quantile method for a DataFrame type object to be returned. Instead a broken Series object can be returned in certain scenarios (see above), by broken I mean trying to use it cause an Exception to occur, which one depends on how you try to use it.

Additionally when a single float value is supplied and the numeric_only parameter is set to True and non-numeric columns are present the expected Series object is returned but it is still broken.

Error Logs

1 Traceback (most recent call last):
  File "/Users/hargreaw/Documents/GitRepos/cell-paint/dfutil/tests/temp.py", line 40, in <module>
    print(1, df3.quantile(0.25, numeric_only=True))  # type: ignore
  File "/Users/hargreaw/Documents/GitRepos/cell-paint/dfutil/.venv/lib/python3.10/site-packages/ray/experimental/tqdm_ray.py", line 49, in safe_print
    _print(*args, **kwargs)
  File "/Users/hargreaw/Documents/GitRepos/cell-paint/dfutil/.venv/lib/python3.10/site-packages/modin/logging/logger_decorator.py", line 125, in run_and_log
    return obj(*args, **kwargs)
  File "/Users/hargreaw/Documents/GitRepos/cell-paint/dfutil/.venv/lib/python3.10/site-packages/modin/pandas/base.py", line 4150, in __str__
    return repr(self)
  File "/Users/hargreaw/Documents/GitRepos/cell-paint/dfutil/.venv/lib/python3.10/site-packages/modin/logging/logger_decorator.py", line 125, in run_and_log
    return obj(*args, **kwargs)
  File "/Users/hargreaw/Documents/GitRepos/cell-paint/dfutil/.venv/lib/python3.10/site-packages/modin/pandas/series.py", line 406, in __repr__
    temp_df = self._build_repr_df(num_rows, num_cols)
  File "/Users/hargreaw/Documents/GitRepos/cell-paint/dfutil/.venv/lib/python3.10/site-packages/modin/logging/logger_decorator.py", line 125, in run_and_log
    return obj(*args, **kwargs)
  File "/Users/hargreaw/Documents/GitRepos/cell-paint/dfutil/.venv/lib/python3.10/site-packages/modin/pandas/base.py", line 266, in _build_repr_df
    return self.iloc[indexer]._query_compiler.to_pandas()
  File "/Users/hargreaw/Documents/GitRepos/cell-paint/dfutil/.venv/lib/python3.10/site-packages/modin/logging/logger_decorator.py", line 125, in run_and_log
    return obj(*args, **kwargs)
  File "/Users/hargreaw/Documents/GitRepos/cell-paint/dfutil/.venv/lib/python3.10/site-packages/modin/core/storage_formats/pandas/query_compiler.py", line 293, in to_pandas
    return self._modin_frame.to_pandas()
  File "/Users/hargreaw/Documents/GitRepos/cell-paint/dfutil/.venv/lib/python3.10/site-packages/modin/logging/logger_decorator.py", line 125, in run_and_log
    return obj(*args, **kwargs)
  File "/Users/hargreaw/Documents/GitRepos/cell-paint/dfutil/.venv/lib/python3.10/site-packages/modin/core/dataframe/pandas/dataframe/utils.py", line 753, in run_f_on_minimally_updated_metadata
    result = f(self, *args, **kwargs)
  File "/Users/hargreaw/Documents/GitRepos/cell-paint/dfutil/.venv/lib/python3.10/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 4522, in to_pandas
    ErrorMessage.catch_bugs_and_request_email(
  File "/Users/hargreaw/Documents/GitRepos/cell-paint/dfutil/.venv/lib/python3.10/site-packages/modin/error_message.py", line 81, in catch_bugs_and_request_email
    raise Exception(
Exception: Internal Error. Please visit https://github.com/modin-project/modin/issues to file an issue with the traceback and the command that caused this error. If you can't file a GitHub issue, please email bug_reports@modin.org.
Internal and external indices on axis 0 do not match.
(_remote_exec_multi_chain pid=49545) Length mismatch: Expected axis has 2 elements, new values have 1 elements. fn=<function PandasDataframe._propagate_index_objs.<locals>.apply_idx_objs at 0x11fe37f40>, obj=Empty DataFrame
(_remote_exec_multi_chain pid=49545) Columns: [0.25, 0.75]
(_remote_exec_multi_chain pid=49545) Index: [], args=[], kwargs={'cols': Index([(0.25, 0.75)], dtype='object')}
(_remote_exec_multi_chain pid=49545) Length mismatch: Expected axis has 2 elements, new values have 1 elements. args=(Empty DataFrame
(_remote_exec_multi_chain pid=49545) Columns: []
(_remote_exec_multi_chain pid=49545) Index: [0.25, 0.75], <function PandasDataframe.transpose.<locals>.<lambda> at 0x11fe37eb0>, 0, 0, 0, 1, <function PandasDataframe._propagate_index_objs.<locals>.apply_idx_objs at 0x11fe37f40>, 0, 1, 'cols', Index([(0.25, 0.75)], dtype='object'), 0, 1), chain=[0, 1]
(_remote_exec_multi_chain pid=49544) Length mismatch: Expected axis has 2 elements, new values have 1 elements. fn=<function PandasDataframe._propagate_index_objs.<locals>.apply_idx_objs at 0x10f49c670>, obj=            0.25      0.75
(_remote_exec_multi_chain pid=49544)  [repeated 10x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)
(_remote_exec_multi_chain pid=49544) feat_9  0.202922  0.783791, args=[], kwargs={'cols': Index([(0.25, 0.75)], dtype='object')}
(_remote_exec_multi_chain pid=49544) Length mismatch: Expected axis has 2 elements, new values have 1 elements. args=(        feat_0    feat_1    feat_2  ...    feat_7    feat_8    feat_9
(_remote_exec_multi_chain pid=49544) 0.75  0.862124  0.947818  0.704255  ...  0.763747  0.850075  0.783791 [repeated 2x across cluster]
(_remote_exec_multi_chain pid=49544) [2 rows x 10 columns], <function PandasDataframe.transpose.<locals>.<lambda> at 0x10f49d870>, 0, 0, 0, 1, <function PandasDataframe._propagate_index_objs.<locals>.apply_idx_objs at 0x10f49c670>, 0, 1, 'cols', Index([(0.25, 0.75)], dtype='object'), 0, 1), chain=[0, 1]

Installed Versions

INSTALLED VERSIONS

commit : bdc79c146c2e32f2cab629be240f01658cfb6cc2
python : 3.10.13.final.0
python-bits : 64
OS : Darwin
OS-release : 22.6.0
Version : Darwin Kernel Version 22.6.0: Wed Jul 5 22:22:05 PDT 2023; root:xnu-8796.141.3~6/RELEASE_ARM64_T6000
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.1
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0.post0
setuptools : 69.2.0
pip : None
Cython : None
pytest : 8.1.1
hypothesis : None
sphinx : 7.2.6
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : None
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.3.1
gcsfs : None
matplotlib : None
numba : 0.59.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 15.0.2
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.13.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

[anmyachev]

Thanks @DiDeoxy! I can reproduce your errors.

[anmyachev]

@DiDeoxy a fix for this bug will be available in a new release scheduled for next week