Skip to content

BUG: .corr() fails when input is Int64Dtype() #33803

Closed
@akanz1

Description

@akanz1
  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • This bug exists on the master branch of pandas. see below


Code Sample, a copy-pastable example

df = pd.DataFrame({'First':[1,2,3,4,np.nan],
                   'Second':[True, True, False, False, True],
                   'Third':[2,3,8,5,6]})
print(df.dtypes)
print(df.corr())
df
df_conv = df.convert_dtypes()
print(df_conv.dtypes)
print(df_conv)
df_conv.corr() # results in the following TypeError

TypeError: float() argument must be a string or a number, not 'NAType'

Problem description

When applying convert_dtypes() to a DataFrame, columns containing NaNs (np.nan) and integer values (stored as float64) are converted to "pd.NA" and actual integers (Int64). While this is desirable, it results in an error when calling the .corr() method on the DataFrame with the new dtypes.

Before convert_dtypes():
before_conv

After convert_dtypes():
after_conv

Error message:
corr_error

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.6.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 1.0.3
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 46.1.3.post20200325
Cython : None
pytest : 5.4.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.13.0
pandas_datareader: None
bs4 : 4.9.0
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.2.1
numexpr : None
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.4.1
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : None
xlsxwriter : None
numba : 0.48.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugNA - MaskedArraysRelated to pd.NA and nullable extension arraysNumeric OperationsArithmetic, Comparison, and Logical operations

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions