BUG: Series.str.isdigit with pyarrow dtype doesn't honor unicode superscripts

[GarrettWu]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
s = pd.Series(['23', '³', '⅕', ''], dtype=pd.StringDtype(storage="pyarrow"))
s.str.isdigit()


	0
0	True
1	False
2	False
3	False

dtype: boolean

Issue Description

Series.str.isdigit() with pyarrow string dtype doesn't honor unicode superscript/subscript. Which diverges with the public doc. https://pandas.pydata.org/docs/reference/api/pandas.Series.str.isdigit.html#pandas.Series.str.isdigit

The bug only happens in Pyarrow string dtype, Python string dtype behavior is correct.

Expected Behavior

import pandas as pd
s = pd.Series(['23', '³', '⅕', ''], dtype=pd.StringDtype(storage="pyarrow"))
s.str.isdigit()

	0
0	True
1	True
2	False
3	False

dtype: boolean

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.11.12
python-bits : 64
OS : Linux
OS-release : 6.1.123+
Version : #1 SMP PREEMPT_DYNAMIC Sun Mar 30 16:01:29 UTC 2025
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.3
numpy : 2.0.2
pytz : 2025.2
dateutil : 2.9.0.post0
pip : 24.1.2
Cython : 3.0.12
sphinx : 8.2.3
IPython : 7.34.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.13.4
blosc : None
bottleneck : 1.4.2
dataframe-api-compat : None
fastparquet : None
fsspec : 2025.3.2
html5lib : 1.1
hypothesis : None
gcsfs : 2025.3.2
jinja2 : 3.1.6
lxml.etree : 5.4.0
matplotlib : 3.10.0
numba : 0.60.0
numexpr : 2.10.2
odfpy : None
openpyxl : 3.1.5
pandas_gbq : 0.28.1
psycopg2 : 2.9.10
pymysql : None
pyarrow : 18.1.0
pyreadstat : None
pytest : 8.3.5
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.15.3
sqlalchemy : 2.0.40
tables : 3.10.2
tabulate : 0.9.0
xarray : 2025.3.1
xlrd : 2.0.1
xlsxwriter : None
zstandard : 0.23.0
tzdata : 2025.2
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report, confirmed on main. Further investigations and PRs to fix are welcome!

[iabhi4]

@rhshadrach The issue stems from pyarrow.compute.utf8_is_digit not recognizing non-ASCII Unicode digits (e.g., '³'). To align with str.isdigit()'s behavior and pandas docs, I propose replacing the Arrow compute call in _str_isdigit() with

def _str_isdigit(self):
        values = self.to_numpy(na_value=None)
        data = []
        mask = []

        for val in values:
            if val is None:
                data.append(False)
                mask.append(True)
            else:
                data.append(val.isdigit())
                mask.append(False)

        from pandas.core.arrays.boolean import BooleanArray
        return BooleanArray(np.array(data, dtype=bool), np.array(mask, dtype=bool))

While this isn’t vectorized, it correctly honors all Unicode digit categories, which aligns with user expectations. Let me know if this workaround is acceptable for now, or if you’d prefer keeping the current Arrow-based behavior and instead clarifying the limitation in the documentation.

Related upstream issue: I’ve confirmed that this is a pyarrow limitation and have raised an enhancement request in the Arrow repo to bring utf8_is_digit in line with str.isdigit().

Optionally, we could also explore reimplementing this in Cython using PyUnicode_READ and Py_UNICODE_ISDIGIT for performance while maintaining Unicode correctness.

Let me know what direction you'd prefer, happy to work on a patch either way

[rhshadrach]

Looks like this is getting fixed upstream (thanks!). Assuming that to be the case, my preference would be to leave pandas as-is.

cc @WillAyd @jorisvandenbossche for any thoughts.

[WillAyd]

Yes I agree - let's keep it as an upstream fix. Thanks for the thorough investigation and solution @iabhi4

[GarrettWu]

Thanks @iabhi4 for the upstream fix apache/arrow#46589. It solves the superscripts issue, but introduces another discrepancy:

// '¾' (vulgar fraction) is treated as a digit by utf8proc 'No'

Any chance we can fix it too? Otherwise str.isdigit is still different on python string and pyarrow string types.

[jorisvandenbossche]

Looks like this is getting fixed upstream (thanks!). Assuming that to be the case, my preference would be to leave pandas as-is.

If this comes in the next pyarrow version, I think we could still add a fallback based on the version. I.e. use pyarrow for recent versions, otherwise still fallback to the python implemenation.
Potentially, assuming the cases that behave differently are all in unicode, we could also first do a check if all elements are ascii, and if so always use the pyarrow version (I think the overhead is worth it)