view into modified dataframe of ints causes subsequent set_value to not work properly

[jbezaire]

The following code snippet produces unexpected output:

import pandas as pd
import numpy as np
x = pd.DataFrame(data=np.zeros((5,5),dtype=np.int),columns=['a','b','c','d','e'])
x['f'] = 0
x.f.values[3]=1
y = x.iloc[np.arange(2,len(x))]
x.f.values[3]=2
print x
print x.f.values

Out[130]: 
   a  b  c  d  e  f
0  0  0  0  0  0  0
1  0  0  0  0  0  0
2  0  0  0  0  0  0
3  0  0  0  0  0  1
4  0  0  0  0  0  0
[0 0 0 2 0]

The expected output is:

Out[131]: 
   a  b  c  d  e  f
0  0  0  0  0  0  0
1  0  0  0  0  0  0
2  0  0  0  0  0  0
3  0  0  0  0  0  2
4  0  0  0  0  0  0
[0 0 0 2 0]

The behavior only manifests if the dataframe has had a column added to it (f in this case), if the columns are ints, and if the view is constructed with certain kinds of indexing (y=x.iloc[2:] works fine for instance).

Here is my version info:

INSTALLED VERSIONS

commit: None
python: 2.7.7.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-229.4.2.el7.jump.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.15.2.dev
nose: 1.3.4
Cython: 0.20.2
numpy: 1.9.0
scipy: 0.14.1rc1.dev-Unknown
statsmodels: None
IPython: 3.1.0
sphinx: 1.3b1
patsy: 0.3.0
dateutil: 2.2
pytz: 2014.10
bottleneck: 0.8.0
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.4.0
openpyxl: 2.1.4
xlrd: None
xlwt: None
xlsxwriter: None
lxml: 3.4.2
bs4: 4.3.2
html5lib: None
httplib2: None
apiclient: None
rpy2: 2.4.3
sqlalchemy: None
pymysql: 0.6.2.None
psycopg2: 2.5.4 (dt dec pq3 ext)

[shoyer]

I can reproduce this on master. Definitely a bug -- thanks for the report!

[jreback]

I know this looks like a bug, and might be. However, you are violating the guarantees. The .values objects does not guarantee that you will get a referenceable consolidated view for a DataFrame.

For a Series this is always True, and generally it is true for a DataFrame when looking at a single dtype (but this is NEVER true for multi-dtype selection as you will always get a copy), but for a single dtype view (or a view on a multi-dtype that only is a single dtype).

This implementation is lazy about consolidation (e.g. it won't actually consolidate on a single insert like this, but will later on).

So we could call it a bug if it should always consolidate on inserts (but this can severly impact performance, that's why its like this). Or say that what you are doing (e.g. by ASSIGNING to .values directly), is technically not allowed (and in any event its not good practice ever).

In [34]: x = pd.DataFrame(data=np.zeros((5,5),dtype=np.int),columns=['a','b','c','d','e'])

In [35]: x._data
Out[35]: 
BlockManager
Items: Index([u'a', u'b', u'c', u'd', u'e'], dtype='object')
Axis 1: Int64Index([0, 1, 2, 3, 4], dtype='int64')
IntBlock: slice(0, 5, 1), 5 x 5, dtype: int64

In [36]: x['f'] = 0

In [37]: x._data
Out[37]: 
BlockManager
Items: Index([u'a', u'b', u'c', u'd', u'e', u'f'], dtype='object')
Axis 1: Int64Index([0, 1, 2, 3, 4], dtype='int64')
IntBlock: slice(0, 5, 1), 5 x 5, dtype: int64
IntBlock: slice(5, 6, 1), 1 x 5, dtype: int64

In [38]: x.f.values.base
Out[38]: array([0, 0, 0, 0, 0])

In [39]: x.e.values.base
Out[39]: 
array([[0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0]])
In [41]: x._consolidate_inplace()

In [42]: x._data
Out[42]: 
BlockManager
Items: Index([u'a', u'b', u'c', u'd', u'e', u'f'], dtype='object')
Axis 1: Int64Index([0, 1, 2, 3, 4], dtype='int64')
IntBlock: slice(0, 6, 1), 6 x 5, dtype: int64

In [43]: x.values.base
Out[43]: 
array([[0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0]])

[shoyer]

@jreback the bug (IMO) is that x and x.f are inconsistent. Assigning to .values is a bit of a side point.

[jreback]

@shoyer you'd have to prove it. How are they inconsistent?

[shoyer]

@jreback compare lines 8 and 9 below:

In [2]: import numpy as np

In [3]: x = pd.DataFrame(data=np.zeros((5,5),dtype=np.int),columns=['a','b','c','d','e'])

In [4]: x['f'] = 0

In [5]: x.f.values[3]=1

In [6]: y = x.iloc[np.arange(2,len(x))]

In [7]: x.f.values[3]=2

In [8]: x
Out[8]:
   a  b  c  d  e  f
0  0  0  0  0  0  0
1  0  0  0  0  0  0
2  0  0  0  0  0  0
3  0  0  0  0  0  1
4  0  0  0  0  0  0

In [9]: x.f
Out[9]:
0    0
1    0
2    0
3    2
4    0
Name: f, dtype: int64

[jreback]

you are using .values, which violates guarantees

[jreback]

In [27]: x = pd.DataFrame(data=np.zeros((5,5),dtype=np.int),columns=['a','b','c','d','e'])

In [28]: x['f'] = 0

In [29]: x.ix[3,'f'] = 2

In [30]: x
Out[30]: 
   a  b  c  d  e  f
0  0  0  0  0  0  0
1  0  0  0  0  0  0
2  0  0  0  0  0  0
3  0  0  0  0  0  2
4  0  0  0  0  0  0

In [31]: x['f']
Out[31]: 
0    0
1    0
2    0
3    2
4    0
Name: f, dtype: int64

[shoyer]

you are using .values, which violates guarantees

It should not be possible to put a dataframe into an inconsistent state, even if you're mis-using the pandas API.

[jreback]

The state IS consistent. The issue is that the user is holding onto an incorrect reference. This is exactly SettingWithCopyError.

[TomAugspurger]

From the docs

If a DataFrame or Panel contains homogeneously-typed data, the ndarray can actually be modified in-place, and the changes will be reflected in the data structure.

So we'll need to update that or call this a bug. Perhaps we should remove that section either way and encourage users to use .loc for setting?

[shoyer]

@jreback what is the incorrect reference? Do we cache x.f instead of computing it on demand?