BUG: apply on Series of dtype category produces dtype category when float was expected

[mkp-gebensleben]

I'm trying to get the left edge from a Series of Intervals. Unexpectedly, the result is a Series of dtype category. I'm also getting a FutureWarning.

>>> import pandas as pd
>>> s = pd.Series(pd.cut([1, 2, 3], 2))
>>> s
0    (0.998, 2.0]
1    (0.998, 2.0]
2      (2.0, 3.0]
dtype: category
Categories (2, interval[float64]): [(0.998, 2.0] < (2.0, 3.0]]
>>> s.apply(lambda x: x.left)
C:\Users\My.Name\.virtualenvs\MyRepo-nagZr6A2\lib\site-packages\pandas\io\formats\format.py:1403: FutureWarning: Index.ravel returning ndarray is deprecated; in a future version this wi
ll return a view on self.
  for val, m in zip(values.ravel(), mask.ravel())
0    0.998
1    0.998
2    2.000
dtype: category
Categories (2, float64): [0.998 < 2.000]

Interestingly, the individual cells have objects with the correct dtype:

>>> result = s.apply(lambda x: x.left)
>>> type(result[0])
<class 'numpy.float64'>

However, operations you would expect from a Series of floats to work do not work:

>>> result + 1
[...]
TypeError: unsupported operand type(s) for +: 'Categorical' and 'int'

pd.show_versions()

INSTALLED VERSIONS
------------------
commit : 9d598a5
python : 3.8.7.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : AMD64 Family 23 Model 96 Stepping 1, AuthenticAMD
byteorder : little
LC_ALL : None
LANG : None
LOCALE : de_DE.cp1252
pandas : 1.2.1
numpy : 1.20.0
pytz : 2021.1
dateutil : 2.8.1
pip : 20.3.3
setuptools : 51.1.2
Cython : None
pytest : 6.2.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.20.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : 3.0.6
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.6.0
sqlalchemy : 1.3.23
tables : None
tabulate : 0.8.7
xarray : None
xlrd : None
xlwt : None
numba : None

[sahahn]

So cut returns a Categorical Series, which while the individual items might be float64 I think it is expected behavior that you can't add a value. I.e., what if the categories were 'cat' and 'dog', you should not be able to add 1 to 'dog'. I think you could either set labels = False when calling cut or cast the result s to type float if you want.

The FutureWarning though seems like its from:

def format_with_na_rep(values: ArrayLike, formatter: Callable, na_rep: str):
      mask = isna(values)
      formatted = np.array(
          [
              formatter(val) if not m else na_rep
              for val, m in zip(values.ravel(), mask.ravel())
          ]
      ).reshape(values.shape)
      return formatted

in https://github.com/pandas-dev/pandas/blob/c322b2436cdbb447a430bb9eb1aff965d07e7e4e/pandas/io/formats/format.py

It seems like this might still be fine even after ravel returns a view on self, so could maybe be muted? But I'm not sure

[mkp-gebensleben]

It seems wildly inconsistent to me that Series of dtype category are the ONLY Series where apply(foo) does not care about the types of the values returned by foo. Instead, it always produces a Series of dtype category, no matter what you apply. No other dtype behaves like this.

[attack68]

It seems wildly inconsistent to me that Series of dtype category are the ONLY Series where apply(foo) does not care about the types of the values returned by foo. Instead, it always produces a Series of dtype category, no matter what you apply. No other dtype behaves like this.

Are you sure no other dtype behaves likes this? Tracing the function identifies a check performed on the series:

is_extension_array_dtype(s)

It seems that extension arrays are classified as:

    Notes
    -----
    This checks whether an object implements the pandas extension
    array interface. In pandas, this includes:

    * Categorical
    * Sparse
    * Interval
    * Period
    * DatetimeArray
    * TimedeltaArray

In the case that it is an extension array the function is applied and no conversion takes place.

In the case that it is not one of the above, there is some dtype conversion taking place. [see core/apply.py lines 855-860]

[mkp-gebensleben]

Thank you, that's good to know. If "never convert" is really intended for these dtypes, maybe it should be mentioned in the Series.apply documentation?

For example:

convert_dtype : bool, default True

    Try to find better dtype for elementwise function results. If False, leave as dtype=object.
    For the dtypes X, Y and Z the original dtype is kept.

[attack68]

Yes indeed, a good opportunity for contributions perhaps :)

[sukriti1]

take