DataFrame.where does not replace np.nan to None in place

[corytu]

Code Sample, a copy-pastable example if possible

import pandas as pd
import numpy as np

s = pd.Series(list(range(5)) + [np.nan])
# Replace np.nan with None
s.where(s.notnull(), None)
# 0       0
# 1       1
# 2       2
# 3       3
# 4       4
# 5    None

# The replacement fails to be done in place
s.where(s.notnull(), None, inplace=True)
s
# 0    0.0
# 1    1.0
# 2    2.0
# 3    3.0
# 4    4.0
# 5    NaN

Problem description

As the following threads discussed on SO, there are needs that "NaN"s should be replaced with "None"s so that databases can handle them correctly. Some suggested that DataFrame.where is a proper way to solve this problem.

• Replacing Pandas or Numpy Nan with a None to use with MysqlDB
• Use None instead of np.nan for null values in pandas DataFrame

However, the replacement does not take place as expected only when inplace parameter is specified as True. Considering that nothing is returned in this case, people are very likely to assume the replacement is done in place, but it's actually not, which can cause more problems in subsequent development.

Expected Output

s = pd.Series(list(range(5)) + [np.nan])
# Replace np.nan with None in place
s.where(s.notnull(), None, inplace=True)
s
# 0       0
# 1       1
# 2       2
# 3       3
# 4       4
# 5    None

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.0.final.0
python-bits: 64
OS: Darwin
OS-release: 18.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: zh_TW.UTF-8
LOCALE: zh_TW.UTF-8

pandas: 0.23.4
pytest: None
pip: 18.1
setuptools: 39.0.1
Cython: None
numpy: 1.15.1
scipy: None
pyarrow: None
xarray: None
IPython: 7.0.1
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[gfyoung]

cc @jreback

[jreback]

might be a bug

though inplace usage is generally a code smell

[TomAugspurger]

By definition, this really can't be inplace with float64 since the float array can't hold None.

@corytu if you really want inplace

s = s.astype(object)
s.fillna(s.notnull(), None, inplace=True)

but just don't use inplace in the first place.

[corytu]

@jreback @TomAugspurger

though inplace usage is generally a code smell
but just don't use inplace in the first place

I don't like inplace either but Python has lots of "inplace" operations already, such as list.append, list.sort, dict.update... I kind of want to make my coding style alike in Python.

If inplace is generally not suggested, how about deprecating that parameter for good anyway?

@TomAugspurger

By definition, this really can't be inplace with float64 since the float array can't hold None.
... if you really want inplace

s = s.astype(object)
s.fillna(s.notnull(), None, inplace=True)

1. The code you demonstrated also fails. fillna cannot be used like that. I assume you were referring to where. If so, that does the trick.

2. I had suspected the data type issue you talked about. Still it cannot explain the inconsistent behaviors showed below:

   s = pd.Series(list(range(5)) + [np.nan])
   s.where(s.notnull(), "foo", inplace=True)
   s
   # Worked
   # 0      0
   # 1      1
   # 2      2
   # 3      3
   # 4      4
   # 5    foo
   
   s = pd.Series(list(range(5)) + [np.nan])
   s.fillna("foo", inplace=True)
   s
   # Worked
   # 0      0
   # 1      1
   # 2      2
   # 3      3
   # 4      4
   # 5    foo

   Both s.where and s.fillna above turn float64 Series into an object one while replacing the missing values. So that inplace fails only when using None to replace np.nan. That seems more like a bug to me then.