Duplicated column name causes inconsistent ValueError during assignment to a unique column

[wmatern]

Code Sample, a copy-pastable example if possible

df1 = pd.DataFrame([[1,2,3,4]], columns=['C','D','D','a'])
df1['a'] = df1['a']

ValueError: Buffer has wrong number of dimensions (expected 1, got 0)

However, the following works without an error message:

df1 = pd.DataFrame([[1,2,3,4]], columns=['C','B','B','a'])
df1['a'] = df1['a']

Problem description

I believe ValueError should not be thrown in the first example. The issue appears to only occur when there is a duplicated column. Interestingly, there appears to be some link to the alphabetical order of the column names.

This appears similar to the issue reported in #21668.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.0.final.0
python-bits: 64
OS: Linux
OS-release: 4.14.76-1-lts
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: None
pip: 18.0
setuptools: 40.4.3
Cython: None
numpy: 1.15.2
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.5.0
sphinx: 1.8.0
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.3
openpyxl: 2.5.7
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: 4.2.5
bs4: None
html5lib: 1.0.1
sqlalchemy: 1.2.12
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[mroeschke]

Thanks for the report. Investigations and PR's welcome!

[adamrossnelson]

Offering an investigation which is based on a scenario in which I encountered a similar issue. I believe the issue is similar because sometimes Pandas will have problems when there are multiple columns with the same name. But not always. And sometimes when you try to make a dataframe with duplicate column names Pandas will throw an error, but not always.

>>> import pandas as pd
>>> data = {'first_name':['Adam','Tom','Sue','Pandi'],
...         'last_name':['Nelson','Jones','Pak','Sun'],
...         'topic_1':[1,2,1,1],'topic_2':[2,2,2,1]}
>>> df = pd.DataFrame(data)
>>> df.rename(columns = {'topic_1':'topic',
...                      'topic_2':'topic'}, inplace=True)
>>> df
  first_name last_name  topic  topic
0       Adam    Nelson      1      2
1        Tom     Jones      2      2
2        Sue       Pak      1      2
3      Pandi       Sun      1      1

I would have expected the df.rename method to throw an error. Because when ...

>>> choice_vals = {1:'Tourism',2:'Politics'}
>>> df['topic'].map(choice_vals)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files (x86)\Python36-32\lib\site-packages\pandas\core\generic.py", line 5180, in __getattr__
    return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'map'

To fix the problem you can repair the column names manually...

>>> df.columns = ['first_name','last_name','topic_1','topic_2']
>>> df['topic_1'] = df['topic_1'].map(choice_vals)
>>> df['topic_2'] = df['topic_2'].map(choice_vals)
>>> df
  first_name last_name   topic_1   topic_2
0       Adam    Nelson   Tourism  Politics
1        Tom     Jones  Politics  Politics
2        Sue       Pak   Tourism  Politics
3      Pandi       Sun   Tourism   Tourism

[phofl]

Was fixed, needs tests

[adamrossnelson]

Is the fix still in Dev or pending release?
I am still able to reproduce the error with Pandas 1.0.5.

[phofl]

Fix is on master, did not test other versions

[ashammad]

I just built pandas-dev at my end, and seems the issue no longer exists
pandas version: 1.2.0.dev0+1174.g4cfa97a18