BUG: need to block split when filling na

[jreback]

xref #14400

In [49]: df = pd.DataFrame({'c1' : list('ABC'),
    ...:                            'c2' : list('123'),
    ...:                            'c3' : np.random.randn(3),
    ...:                            'c4' : np.arange(3)})

In [50]: df
Out[50]: 
  c1 c2        c3  c4
0  A  1  0.645628   0
1  B  2 -0.841708   1
2  C  3  0.207423   2

In [51]: df.dtypes
Out[51]: 
c1     object
c2     object
c3    float64
c4      int64
dtype: object

In [52]: df.apply(lambda x: pd.to_numeric(x, errors='coerce')).fillna(df).dtypes
Out[52]: 
c1    object
c2     int64
c3    object
c4     int64
dtype: object

In [53]: df.apply(lambda x: pd.to_numeric(x, errors='coerce')).fillna(df).apply(lambda x: pd.to_numeric(x, errors='ignore')).dtypes
Out[53]: 
c1     object
c2      int64
c3    float64
c4      int64
dtype: object

so [52] should be [53] already, but since the filled block was originally float, I don't think the splitting is happening.

[mroeschke]

This looks to work on main. Could use a test I suppose

[whendo]

Should this test drop into tests/base/test_fillna.py or maybe pandastests/dtypes somewhere?

[srkds]

Should this test drop into tests/base/test_fillna.py or maybe pandastests/dtypes somewhere?

Look into #writing-tests. That might help to decide.

[nirmalmuppiri]

take

[ivonastojanovic]

@nirmalmuppiri Are you still working on this issue?

[rishikesanr]

Issue still persists in pandas version 2.0.3

df = pd.DataFrame({
'c1': list('ABC'),
'c2': list('123'),
'c3': np.random.randn(3),
'c4': np.arange(3)
})

df.apply(lambda x: pd.to_numeric(x, errors='coerce')).fillna(df).dtypes

c1 object
c2 object
c3 float64
c4 int64
dtype: object

Despite c2 being convertible to int64, it stays as object. However, column c3 is correctly identified as float64.
It seems like the fillna(df) operation might be overriding the numeric conversion for c2.

Is someone working on this issue?

[rishikesanr]

take