BUG: When applying Series.clip to ordered categorical data, the outcome is different from what is expected

[vitalizzare]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
s = pd.Series(['zero','one','two','three','four','five','six'], dtype='category')
s = s.cat.reorder_categories(['zero','one','two','three','four','five','six'], ordered=True)
df = pd.concat([s, s.clip(lower='two'), s.clip(upper='four'), s.clip(lower='two', upper='four')], axis=1)
print(df)

Issue Description

Let's say we have a series of ordered categorical data (see a series s in the code above). If I apply to it a method clip with only one of lower= or upper= parameters, I get the expected output, see columns 1, 2 in the below output example. But if both of them are passed then the output is different from the expected ['two','two','two','three','four','four',four'] series, see the last column:

       0      1      2     3
0   zero    two   zero  four
1    one    two    one  four
2    two    two    two  four
3  three  three  three   two
4   four   four   four   two
5   five   five   four   two
6    six    six   four   two

Also I noticed that swapping of the values for lower=..., upper=... doesn't change the result, i.e. s.clip(lower='two',upper='four') == s.clip(lower='four',upper='two').

The issue may be caused by this part (lines 8147-8153) in pandas.core.generic:

class NDFrame(PandasObject, indexing.IndexingMixin):
    ...
    def clip(
        self: NDFrameT,
        lower=None,
        upper=None,
        ...
        if (
            lower is not None
            and upper is not None
            and is_scalar(lower)
            and is_scalar(upper)
        ):
            lower, upper = min(lower, upper), max(lower, upper)
        ...

Expected Behavior

I expect this code:

import pandas as pd
s = pd.Series(['zero','one','two','three','four','five','six'], dtype='category')
s = s.cat.reorder_categories(['zero','one','two','three','four','five','six'], ordered=True)
print(s.clip(lower='two', upper='four'))

to produce this output:

0      two
1      two
2      two
3    three
4     four
5     four
6     four
dtype: category
Categories (7, object): ['zero' < 'one' < 'two' < 'three' < 'four' < 'five' < 'six']

Installed Versions

INSTALLED VERSIONS

commit : 91111fd
python : 3.10.7.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19044
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 9, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : Russian_Ukraine.1251

pandas : 1.5.1
numpy : 1.23.4
pytz : 2022.2.1
dateutil : 2.8.2
setuptools : 63.2.0
pip : 22.3
Cython : None
pytest : 7.1.3
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.1
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.5.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.6.1
numba : None
numexpr : None
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.9.1
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None

[vamsi-verma-s]

Hi @vitalizzare
just sharing my opinion here -

The docs for .clip say lower and upper need to be float or array-like don't think this would work for categorical.
I feel this is made for numerical data. But, It may work for values that could be compared independently like float.

This would work:
s.where(s>='two', 'two').where(s<='four', 'four')
or

could workaround the issue like this :
s.clip(lower='two').clip(upper='four')

pandas/pandas/core/generic.py

Lines 8025 to 8048 in 890d097

	Trim values at input threshold(s).
	Assigns values outside boundary to boundary values. Thresholds
	can be singular values or array like, and in the latter case
	the clipping is performed element-wise in the specified axis.
	Parameters
	----------
	lower : float or array-like, default None
	Minimum threshold value. All values below this
	threshold will be set to it. A missing
	threshold (e.g `NA`) will not clip the value.
	upper : float or array-like, default None
	Maximum threshold value. All values above this
	threshold will be set to it. A missing
	threshold (e.g `NA`) will not clip the value.
	axis : {{0 or 'index', 1 or 'columns', None}}, default None
	Align object with lower and upper along the given axis.
	For `Series` this parameter is unused and defaults to `None`.
	inplace : bool, default False
	Whether to perform the operation in place on the data.
	*args, **kwargs
	Additional keywords have no effect but might be accepted
	for compatibility with numpy.

[topper-123]

A late follow-up:

If you look at the code for 'NDFrame.clip', it can be seen that the method is intended for non-numeric data also and I can see clipping ordered categorical data has a use case. So IMO the doc string is not accurate @vamsi-verma-s.

I agree this is a bug. A PR would be welcome.

[mohammedouahman]

This is interesting, here is a big one:
you can use the alternative approaches mentioned in the suggested solution. Here are the two suggested workarounds:

• Use .where() method:

s.where(s >= 'two', 'two').where(s <= 'four', 'four')

• Chain two .clip() calls:

s.clip(lower='two').clip(upper='four')