BUG: SettingWithCopyError raised unsystematically (?)

[nolanbconaway]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd

pd.options.mode.chained_assignment = "raise"


def f(df):
    res = df["a"]
    is_two = df["a"] == 2
    res.loc[is_two] = None
    return res


df1 = pd.DataFrame(dict(a=range(3)))
df2 = df1.assign(b=pd.date_range("2020-01-01", "2020-01-03"))

f(df1)  # no problem
f(df2)  # SettingWithCopyError

Problem description

The function above selects a column a, finds the position of values that are equal to two and sets those values to None, then returns the a Series.

When a dataframe only contains the a column, there is no problem. When a datetime column is introduced, SettingWithCopyError is raised.

Expected Output

Because the function f does not access any column besides a, I expect it to behave identically for both dataframes.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 9d598a5
python : 3.8.5.final.0
python-bits : 64
OS : Darwin
OS-release : 19.6.0
Version : Darwin Kernel Version 19.6.0: Tue Nov 10 00:10:30 PST 2020; root:xnu-6153.141.10~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.1
numpy : 1.19.4
pytz : 2020.4
dateutil : 2.8.1
pip : 20.3.3
setuptools : 47.1.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.8.6 (dt dec pq3 ext lo64)
jinja2 : 2.11.2
IPython : 7.19.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.3
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.5.4
sqlalchemy : None
tables : None
tabulate : None
xarray : 0.16.2
xlrd : None
xlwt : None
numba : None

[nmay231]

The issue here is you are implicitly using chained-assignment. The function you wrote boils down to:

def f(df):
    df['a'].loc[df['a'] == 2] = None
    return df['a'].loc[df['a'] == 2]

That is because res = df["a"] creates a reference to the "a" column, not a copy that you are allowed to modify. There is an easy fix.

def f(df):
    res = df['a'].copy() # copy the column
    is_two = df['a'] == 2
    res.loc[is_two] = None
    return res

To be honest, I'm not certain why it only raises an error/warning with the second dataframe. Maybe it has to do with a subtle threshold where fully numeric dataframes automatically copy columns on access? E.g. maybe df['a'] is the same as df['a'].copy() for numeric dataframes? That would be my assumption.

In any case, you can look at this post for a more detailed explanation about chain-expressions.

[nolanbconaway]

Hi @nmay231 ! Thanks for looking at this. I'm aware of the causes of the SettingWithCopyWarning/Error, its fixes, etc, but the bug that I am specifically referring to in this issue is that it is raised inconsistently!

My thinking is that if one is setting with a reference, that the error should be raised regardless of what other data is in the frame. Is this not the desired behavior?