Wrong orientation of operations between DataFrame and range()

[toobaz]

Code Sample, a copy-pastable example if possible

In [3]: df = pd.DataFrame(np.ones((2,2)), columns=['A', 'B'])

In [4]: df + list(range(2))
Out[4]: 
     A    B
0  1.0  2.0
1  1.0  2.0

In [5]: df + pd.Series(range(2), index=df.columns)
Out[5]: 
     A    B
0  1.0  2.0
1  1.0  2.0

In [6]: df + range(2)
Out[6]: 
     A    B
0  1.0  1.0
1  2.0  2.0

Problem description

Operations where the operand is a range operate in the wrong direction compared to all other 1-d objects (I can think of).

Expected Output

Same as In [4]: and In [5]:.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: 8a69543
python: 3.5.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.0-3-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: it_IT.UTF-8
LOCALE: it_IT.UTF-8

pandas: 0.21.0rc1+19.g8a69543ab.dirty
pytest: 3.0.6
pip: 9.0.1
setuptools: None
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
pyarrow: None
xarray: None
IPython: 5.1.0.dev
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.3.0
numexpr: 2.6.1
feather: 0.3.1
matplotlib: 2.0.0
openpyxl: None
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.6
lxml: None
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.0.15
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.2.1

[jschendel]

A bit obscure, but collections.deque has the same behavior as range in this case:

In [3]: df + collections.deque([0, 1])
Out[3]:
     A    B
0  1.0  1.0
1  2.0  2.0

[toobaz]

Yeah, I bet this happens with anything which is (conceptually) 1-d but not recognized as list-like by pandas.

[jreback]

must be a too specific path somewhere, these are both considered list-like (ala is_list_like)

[jorisvandenbossche]

is_list_like indeed considers them as list-like (not saying that the actual code path is using is_list_like):

In [38]: pd.api.types.is_list_like(range(3))
Out[38]: True

[toobaz]

OK, bet lost :-)

[elsander]

I'm interested in working on this bug, but I'm struggling to figure out where the relevant code is. There's no __add__ method given for the pd.DataFrame or NDFrame classes, so I suppose it's inherited?

[elsander]

Ah-- looks like the factory is in core/ops.py. I'll take a look at this.

[elsander]

Looks like the issue is that a list has an ndim attribute, but a range doesn't, which means that they are handled differently in this block from core.internals.Block.eval():

        is_transposed = False
        if hasattr(other, 'ndim') and hasattr(values, 'ndim'):
            if values.ndim != other.ndim:
                is_transposed = True
            else:
                if values.shape == other.shape[::-1]:
                    is_transposed = True
                elif values.shape[0] == other.shape[-1]:
                    is_transposed = True

One approach might be something like this:

import collections
if isinstance(other, collections.Iterable) and not hasattr(other, 'ndim'):
    other = list(other)

But I imagine this could introduce other gotchas and possibly hurt performance.

[jschendel]

@elsander : I've got a fix for this that I'm in the process of finalizing, but if you'd rather work on it I'd be fine with that. From what I can tell, the root cause appears to be pandas' alignment process prior to when addition/multiplication/etc occurs:

pandas/pandas/core/ops.py

Lines 1150 to 1151 in a2e5400

	def _align_method_FRAME(left, right, axis):
	""" convert rhs to meet lhs dims if input is list, tuple or np.ndarray """

Specifically, the check here is a bit too narrow:

pandas/pandas/core/ops.py

Lines 1168 to 1169 in a2e5400

	if isinstance(right, (list, tuple)):
	right = to_series(right)

Basically I swapped this with is_list_like(right) and excluded DataFrame/Series (since is_list_like catches them, but we don't care about casting those). Also moved the check against ndarray to be before the is_list_like check since is_list_like catches ndarray but there's slightly different logic for ndarray (and nested the ndim == 0 case inside the ndarray check so it didn't hit the subsequent is_list_like check) .

Could be other workarounds though, so not trying to suggest what I did is the "best" approach.

[elsander]

@jschendel if you have a fix in progress, go for it! I didn't have a good workaround in mind.