pandas.concat bug with series of the same name and axis=1

[lesteve]

Kind of an edge case I would say, but still it would be nice to get an error or have an automatic renaming of the columns. If a simple error is acceptable as a first step I could definitely put together a PR.

The following snippet shows that when you concatenate a column of zeros and a column of ones the resulting dataframe has two columns of ones:

import pandas as pd
import numpy as np
ts0 = pd.Series(np.zeros(5))
ts1 = pd.Series(np.ones(5))
ts0.name = ts1.name = 'same name'
df = pd.concat([ts0, ts1], axis=1)
print df

and the output (pandas 0.10.1):

   same name  same name
0          1          1
1          1          1
2          1          1
3          1          1
4          1          1

whereas I would expect:

   same name  same name
0          0          1
1          0          1
2          0          1
3          0          1
4          0          1

Looks like this is due to pandas.tools.merge._Concatenator._get_concat_axis which doesn't check for name uniqueness, here is the relevant snippet:

names = []
for x in self.objs:
    if x.name is not None:
        names.append(x.name)
    else:
        return Index(np.arange(len(self.objs))
return Index(names)

[hayd]

Being automatic wouldn't be backwards compatible, but perhaps a suffixes argument similar to merge?

[jreback]

your can pass verify_integrity=True to catch this

edit: sorry...doens't catch anything in this case

(in the axis=0) will throw an exception

In [23]: pd.concat([ts0,ts1],axis=0,verify_integrity=True)
Exception: Indexes have overlapping values: [0, 1, 2, 3, 4]

In [24]: pd.concat([ts0,ts1],axis=0,verify_integrity=False)
Out[24]: 
0    0
1    0
2    0
3    0
4    0
0    1
1    1
2    1
3    1
4    1
Name: same name, dtype: float64
``

[lesteve]

Thinking about it a bit more I have changed my mind slightly and I feel like there is nothing wrong (it is a bit surprising though I have to say) with having a dataframe with identically named columns. After all it's just the transposed equivalent of a series having duplicate indices.

What is wrong is the df values: you concatenate a series of zeros and a series of ones and I would expect to get a dataframe where the first column has zeros and the second columns has ones. Instead you get a dataframe where both columns have ones.

Out of interest, not sure about backward-compatibility in this case, is there a use case where you do rely on this rather unexpected concat behaviour?

[jreback]

i don't use concat that way
but functionality exists

[hayd]

@lesteve I take it back about backwards compatibility, stupidly I didn't realise it was repeating the same column values. That kinda sucks.

[lesteve]

@hayd ok I thought I was missing something. I guess I could have been more explicit about the actual problem in the original message.

(I actually ended up editing the original post to make the problem more explicit)

[dieterv77]

I added a PR to fix this: #3058

[dieterv77]

Fixed with 828f9f9