BUG: assert_frame_equal(check_dtype=False) fails when comparing two DFs containing pd.NA that only differ in dtype (object vs Int32)

[michiel-de-muynck]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
from pandas.testing import assert_frame_equal

df1 = pd.DataFrame(
    {
        "x": pd.Series([pd.NA], dtype="Int32"),
    }
)
df2 = pd.DataFrame(
    {
        "x": pd.Series([pd.NA], dtype="object"),
    }
)

assert_frame_equal(df1, df2, check_dtype=False) # fails, but should succeed

Issue Description

Output of the above example:

AssertionError: DataFrame.iloc[:, 0] (column name="x") are different

DataFrame.iloc[:, 0] (column name="x") values are different (100.0 %)
[index]: [0]
[left]:  [nan]
[right]: [<NA>]

When comparing DataFrames containing pd.NA using check_dtype=False, the test incorrectly fails despite the only difference being the dtype (Int32 vs object).

Note that the values in the dataframe really are the same:

print(type(df1["x"][0])) # prints <class 'pandas._libs.missing.NAType'>
print(type(df2["x"][0])) # prints <class 'pandas._libs.missing.NAType'>

Related issues:

• assert_frame_equal not differentiating NaN and None within object dtype #18463: Similar but "opposite": here the dataframes contain different values (nan vs None) which are incorrectly treated as equal. In this issue, the dataframes contain equal values which are incorrectly treated as different.

Expected Behavior

The test should succeed, since the only difference is the dtypes, and check_dtype=False.

Installed Versions

pandas : 2.2.3
numpy : 1.26.4
pytz : 2025.2
dateutil : 2.9.0.post0
pip : 24.0
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 19.0.1
pyreadstat : None
pytest : 8.3.5
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 2.0.41
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : 0.23.0
tzdata : 2025.2
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report, this would pass if when converting the EA to a NumPy array we cast to object dtype. I haven't looked to see if this might cause issues in other cases. Since this is aimed at tests, I'm wondering if changing to object dtype is okay here.

cc @jbrockmendel @mroeschke for any thoughts.

[jbrockmendel]

this would pass if when converting the EA to a NumPy array we cast to object dtype

Yah I'm pretty sure that the behavior of df1['x'].to_numpy() casting to a float dtype was a much-discussed intentional decision. Changing that would be a can of worms.

I'm inclined to just discourage the use of a) check_dtype=False and b) using pd.NA in an object dtype column (note that df1 == df2 raises)

[rhshadrach]

@jbrockmendel - sorry, I wasn't clear. I meant just inside assert_frame_equal to use .to_numpy(dtype="object") when check_dtype=False rather than just .to_numpy(). Agreed changing the behavior of .to_numpy() is off the table.