BUG: df.replace() with list of regex sometimes fails with column of null objects

[zaneselvans]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
# FAILS
df1 = pd.DataFrame({
    "a": pd.Series([1], dtype="int64"),
    "b": pd.Series([None], dtype="object"),
})

# PASSES
df2 = pd.DataFrame({
    "a": [1],
    "b": [None],
}, dtype="object")

# PASSES
df3 = pd.DataFrame({
    "a": pd.Series([1], dtype="object"),
    "b": pd.Series([None], dtype="object"),
})

# PASSES
df4 = pd.DataFrame({
    "a": pd.Series([1], dtype="int64"),
    "b": pd.Series([None], dtype="object"),
    "c": pd.Series(["bye"], dtype="str"),
})

# FAILS
df5 = pd.DataFrame({
    "a": pd.Series([None], dtype="object"),
    "b": pd.Series([None], dtype="object"),
})

# FAILS
df6 = pd.DataFrame({
    "a": pd.Series([None]),
    "b": pd.Series([None]),
}, dtype="object")

# FAILS
df7 = pd.DataFrame({
    "a": pd.Series([1], dtype="int64"),
    "b": pd.Series([pd.NA], dtype="object"),
})

# PASSES
df8 = pd.DataFrame({
    "a": pd.Series([1], dtype="int64"),
    "b": pd.Series([pd.NA], dtype="string"),
})

regex_string = r"(^a$|\d{4})"
regex_list = [r"^a$", r"\d{4}"]

dfs = {
    "df1": df1,
    "df2": df2,
    "df3": df3,
    "df4": df4,
    "df5": df5,
    "df6": df6,
    "df7": df7,
    "df8": df8,
}

for label, df in dfs.items():
    try:
        df.replace(regex=regex_string, value="dude")
    except ValueError as exc:
        print(f"{label}: String regex FAILED!")
        print(exc)
    
    try:
        df.replace(regex=regex_list, value="dude")
    except ValueError as exc:
        print(f"{label}: List of regexes FAILED!")
        print(exc)

Issue Description

Under some circumstances, when using DataFrame.replace() with a list of regular expressions on a dataframe that contains a column where dtype="object" and all values in that column are some kind of null (including None, np.nan and pd.NA) the operation will fail with:

ValueError: cannot call `vectorize` on size 0 inputs unless `otypes` is set

With a single regular expression rather than a list, there's no problem, and the behavior seems to depend on how the dataframe is constructed, how the dtypes are specified, and what the dtypes of the other columns in the dataframe are.

This behavior was not present in pandas 1.5.x

Expected Behavior

I expected all of the examples above to succeed, regardless of whether the dataframes contained null values or object types, and regardless of whether I used a list of regular expressions or a single regular expression to specify the pattern to be replaced.

Installed Versions

INSTALLED VERSIONS
------------------
commit           : 0f437949513225922d851e9581723d82120684a6
python           : 3.11.4.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 22.5.0
Version          : Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:20 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T6000
machine          : arm64
processor        : arm
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.UTF-8

pandas           : 2.0.3
numpy            : 1.24.4
pytz             : 2023.3
dateutil         : 2.8.2
setuptools       : 68.0.0
pip              : 23.2.1
Cython           : None
pytest           : 7.4.0
hypothesis       : None
sphinx           : 7.1.2
blosc            : None
feather          : None
xlsxwriter       : 3.1.2
lxml.etree       : 4.9.3
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 3.1.2
IPython          : 8.14.0
pandas_datareader: None
bs4              : 4.12.2
bottleneck       : None
brotli           : 
fastparquet      : None
fsspec           : 2023.6.0
gcsfs            : 2023.6.0
matplotlib       : 3.7.2
numba            : None
numexpr          : None
odfpy            : None
openpyxl         : 3.1.2
pandas_gbq       : 0.19.2
pyarrow          : 12.0.1
pyreadstat       : None
pyxlsb           : 1.0.10
s3fs             : None
scipy            : 1.11.1
snappy           : 
sqlalchemy       : 1.4.49
tables           : None
tabulate         : 0.9.0
xarray           : None
xlrd             : 2.0.1
zstandard        : 0.21.0
tzdata           : 2023.3
qtpy             : None
pyqt5            : None

[Xemptuous]

I'm also having this issue, but also with string replace as well. Using pandas 2.0.3 and numpy 1.24.2.
Line causing issues:

df = df.replace({r"^\s*$": "NULL"}, regex=True)

traceback:

 File "pandas\core\series.py", line 5223, in replace
    return super().replace(
  File "pandas\core\generic.py", line 7299, in replace
    return self.replace(
  File "pandas\core\series.py", line 5223, in replace
    return super().replace(
  File "pandas\core\generic.py", line 7347, in replace
    new_data = self._mgr.replace_list(
  File "pandas\core\internals\managers.py", line 496, in replace_list
    bm = self.apply(
  File "pandas\core\internals\managers.py", line 352, in apply
    applied = getattr(b, f)(**kwargs)
  File "pandas\core\internals\blocks.py", line 770, in replace_list
    for i, ((src, dest), mask) in enumerate(zip(pairs, masks)):
  File "pandas\core\internals\blocks.py", line 748, in <genexpr>
    compare_or_regex_search(
  File "pandas\core\array_algos\replace.py", line 96, in compare_or_regex_search
    result = op(a)
  File "numpy\lib\function_base.py", line 2329, in __call__
    return self._vectorize_call(func=func, args=vargs)
  File "numpy\lib\function_base.py", line 2407, in _vectorize_call
    ufunc, otypes = self._get_ufunc_and_otypes(func=func, args=args)
  File "numpy\lib\function_base.py", line 2363, in _get_ufunc_and_otypes
    raise ValueError('cannot call `vectorize` on size 0 inputs '
ValueError: cannot call `vectorize` on size 0 inputs unless `otypes` is set

[saras-pyleusa]

This issue still seems to be occurring. I am using Pandas 2.1.4 and Numpy 1.26.2. Line thats causing the issue is:

df.replace({"\x00": ""}, regex=True, inplace=True)

Traceback:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
  result = await app(  # type: ignore[func-returns-value]
File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
  return await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 1115, in __call__
  await super().__call__(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 122, in __call__
  await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in __call__
  raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
  await self.app(scope, receive, _send)
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
  raise exc
File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
  await self.app(scope, receive, sender)
File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
  raise e
File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
  await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 718, in __call__
  await route.handle(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 276, in handle
  await self.app(scope, receive, send)
File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 66, in app
  response = await func(request)
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 274, in app
  raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
  return await dependant.call(**values)
File "/app/main.py", line 132, in mssql_to_bq
  df.replace({"\x00": ""}, regex=True, inplace=True)
File "/usr/local/lib/python3.9/site-packages/pandas/core/generic.py", line 7819, in replace
  return self.replace(
File "/usr/local/lib/python3.9/site-packages/pandas/core/generic.py", line 7867, in replace
  new_data = self._mgr.replace_list(
File "/usr/local/lib/python3.9/site-packages/pandas/core/internals/base.py", line 253, in replace_list
  bm = self.apply_with_block(
File "/usr/local/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 354, in apply
  applied = getattr(b, f)(**kwargs)
File "/usr/local/lib/python3.9/site-packages/pandas/core/internals/blocks.py", line 865, in replace_list
  masks = list(masks)
File "/usr/local/lib/python3.9/site-packages/pandas/core/internals/blocks.py", line 852, in <genexpr>
  compare_or_regex_search(
File "/usr/local/lib/python3.9/site-packages/pandas/core/array_algos/replace.py", line 98, in compare_or_regex_search
  result = op(a)
File "/usr/local/lib/python3.9/site-packages/numpy/lib/function_base.py", line 2372, in __call__
  return self._call_as_normal(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/numpy/lib/function_base.py", line 2365, in _call_as_normal
  return self._vectorize_call(func=func, args=vargs)
File "/usr/local/lib/python3.9/site-packages/numpy/lib/function_base.py", line 2450, in _vectorize_call
  ufunc, otypes = self._get_ufunc_and_otypes(func=func, args=args)
File "/usr/local/lib/python3.9/site-packages/numpy/lib/function_base.py", line 2406, in _get_ufunc_and_otypes
  raise ValueError('cannot call `vectorize` on size 0 inputs '
ValueError: cannot call `vectorize` on size 0 inputs unless `otypes` is set

Is there any workaround or something till this gets fixed?

[mtsadler-branch]

@saras-pyleusa @Xemptuous

Issue:
df.replace(thing) throws an Error when thing is not in df.

Solution:
Use df.filter(thing).empty to first check if thing is in df.

If you're code is something like:

df = df.replace(regex=thing, value="")

Change it to something like:

if not df.filter(regex=thing).empty:
    df = df.replace(regex=thing, value="")

Explanation:
We want to skip df.replace() when df == df.replace(), because the .replace() call is moot (hence size 0 inputs in the Error message).

• df.filter(regex=thing) in this case, will return a dataframe with only the things we want to replace.
• If df.filter(regex=thing) is empty, then we want to skip the .replace().
• If its not empty, then we want to execute the .replace().

---

FWIW, my initial code was:

things = [r'^\s*$', np.nan, 'nan', 'Null', 'NULL', 'null', 'None', 'NONE', 'none']
df = df.replace(regex=things, value="")

and after handling this error it got quite messy:

things = [np.nan, 'nan', 'Null', 'NULL', 'null', 'None', 'NONE', 'none']
r_white_spaces = r'^\s*$'
if not (df.filter(regex=r_white_spaces).empty and df.filter(items=things).empty):
    things.append(r_white_spaces)
    df = df.replace(regex=things, value="")

Hoping to upgrade pandas soon and simplify these lines a bit 😄 🤞🏼