Copy-on-Write (CoW): index labels are still shared mutable state

[tswast]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

pd.options.mode.copy_on_write = True

df = pd.DataFrame({"a": [1, 2, 3]})
df

	a
0	1
1	2
2	3

s = df["a"]
s

0    1
1    2
2    3
Name: a, dtype: int64

i = df.index
i.name = "new name"
df

	a
new name
0	1
1	2
2	3

s

    new name
    0    1
    1    2
    2    3
    Name: a, dtype: int64

Issue Description

When turning on Copy-on-Write mode, I notice that index labels are some remaining shared mutable state across Series and DataFrame (and Index).

At a minimum, I'd expect the mutated index label not to affect the series "s" in the code example.

Potentially, the mutated index label wouldn't affect DataFrame either, but I could see some benefits to keeping that shared mutable state around.

Expected Behavior

Once a Series is constructed, it's index label is independent of that from the original DataFrame.

Installed Versions

INSTALLED VERSIONS

commit : 965ceca
python : 3.10.9.final.0
python-bits : 64
OS : Darwin
OS-release : 22.5.0
Version : Darwin Kernel Version 22.5.0: Mon Apr 24 20:51:50 PDT 2023; root:xnu-8796.121.2~5/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.0.2
numpy : 1.23.1
pytz : 2022.1
dateutil : 2.8.2
setuptools : 67.6.1
pip : 22.2
Cython : None
pytest : 7.3.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.4.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli :
fastparquet : None
fsspec : 2022.10.0
gcsfs : None
matplotlib : 3.5.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : 0.19.1
pyarrow : 11.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.8.1
snappy : None
sqlalchemy : 1.4.42
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : 2.3.0
pyqt5 : None

[tswast]

A related example demonstrating that triggering copy-on-write breaks the coupling between DataFrame index and Series index.

import pandas as pd

pd.options.mode.copy_on_write = True

df = pd.DataFrame({"a": [1, 2, 3]})
df

	a
0	1
1	2
2	3

This Series shares and Index object with DataFrame

s1 = df["a"]
s1

0    1
1    2
2    3
Name: a, dtype: int64

Mutating a Series breaks the link to the DataFrame index

s2 = df["a"]
s2.loc[1] = 100
s2

0      1
1    100
2      3
Name: a, dtype: int64

Question: Should this Index object be treated independently from DataFrame or is it okay that a mutation here affects the DataFrame?

i = df.index
i.name = "new name"
print(df.index.name)

new name

This Series hasn't been mutated. It shares an Index object with DataFrame.

print(s1.index.name)

new name

This Series has been mutated. It doesn't share an Index object with DataFrame.

print(s2.index.name)

None

s1.loc[1] = 100

print(s1.index.name)

new name

[lithomas1]

I think Copy on Write only applies to different pandas objects that share the same data.

Here df.index is s.index, so it shouldn't apply since they are the same object.

We could fix this issue by creating a new Index object sharing values with the original index when you index into the DF, but I don't think that's the way to go.

I'll leave this open for others to comment.

EDIT: Typo

[tswast]

Here df.index == s.index, so it shouldn't apply since they are the same object.

Isn't that the problem Copy-on-Write is solving though? It wasn't obvious when objects were views and when they weren't. IMO it's not at all intuitive that these are the same object when the objects these indexes belong to are no longer views on the same data.

[jorisvandenbossche]

I personally wouldn't change the fact that you can mutate an Series/DataFrame object's index/columns attributes (for example, that would also mean that df.index.names = ... would stop working, and that would be a quite big breaking change, and also just a behaviour that is good to keep IMO).

But when you create a new object, like creating a Series from a DataFrame column, at that point I think we should consider also not propagating mutable state of the indexes between both (independent) objects.

Here df.index is s.index, so it shouldn't apply since they are the same object.

Yes, it is correct that they are currently "identical", which explains why we see propagation of mutating their attributes.

But the question is: should they be identical? (should the df.index is df["col"].index assertion hold?)

It wasn't obvious when objects were views and when they weren't. IMO it's not at all intuitive that these are the same object when the objects these indexes belong to are no longer views on the same data.

Yes, now that we generally create new objects from indexing operations (eg also getting the same column twice doesn't give identical objects, i.e. df["col"] is df["col"] gives False), I would personally also expect that their indices are no longer identical / shared. Of course under the hood we still share the data, but just by each time creating a shallow copy.

We could fix this issue by creating a new Index object sharing values with the original index when you index into the DF, but I don't think that's the way to go.

Personally, I think that would be fine.

[lithomas1]

Yes, now that we generally create new objects from indexing operations (eg also getting the same column twice doesn't give identical objects, i.e. df["col"] is df["col"] gives False), I would personally also expect that their indices are no longer identical / shared. Of course under the hood we still share the data, but just by each time creating a shallow copy.

Ah, I forgot that we turned off the item_cache for Copy on Write.

This is fine then.