pd.to_numeric silently not converting when errors='coerce'

[fakio]

Code Sample, a copy-pastable example if possible

import pandas as pd
pd.to_numeric([-47393996303418497800, 'string'], errors='coerce')

Output:

array([-47393996303418497800, 'string'], dtype=object)

Problem description

to_numeric is silently not coercing 'string' to nan.

Seems similar to #17125 and #17007

pandas version 0.23.4

[chris-b1]

Yeah, the specific issue here seems to be that your number is outside of int64 space

-47393996303418497800 < np.iinfo('int64').min
# Out[18]: True

Some discussion in #17007 - I suppose the right thing do is convert to a float, even though lossy

-47393996303418497800 - int(float(-47393996303418497800))
# Out[24]: 1272

[chris-b1]

cc @gfyoung

[gfyoung]

Yeah...this is a tough one. When I initially worked on this, I really didn't want to coerce to float given how lossy that could be in cases like this. That being said, if you're passing in coerce, does that mean you're willing to accept lossiness in return for coercing every entry to some kind of number?

@fakio : What would you personally prefer?

cc @jreback

[fakio]

Some more examples I find strange:

pd.to_numeric([100000000000000000000])
array([1.e+20])

pd.to_numeric([-100000000000000000000])
OverflowError: int too big to convert

pd.to_numeric(100000000000000000000)
100000000000000000000

pd.to_numeric(-100000000000000000000)
-100000000000000000000

pd.to_numeric('100000000000000000000')
pd.to_numeric('-100000000000000000000')
pd.to_numeric(['100000000000000000000'])
pd.to_numeric(['-100000000000000000000'])
ValueError: Integer out of range. at position 0

pd.to_numeric(['10000000000000000000000000', 'string'], errors='coerce')
array([nan, nan])

pd.to_numeric([10000000000000000000000000, 'string'], errors='coerce')
array([1.e+25,    nan])

pd.to_numeric(['-100000000000000000000', 'string'], errors='coerce')
array([nan, nan])

pd.to_numeric([-100000000000000000000, 'string'], errors='coerce')
array([-100000000000000000000, 'string'], dtype=object)

[fakio]

@gfyoung the problem is converting one specific entry or having to convert everything to float because of nan?

[gfyoung]

having to convert everything to float because of nan?

@fakio : If one element is nan, all elements must become float. That's just how the internals of the arrays work from numpy. Thus, that's why you get the lossy nature as I alluded to above. I was asking you whether you would be okay with that as an end user.

Another limitation from the numpy world is that there are numeric limits to the numbers you can pass in. Those limits are np.iinfo(np.int64).min to np.iinfo(np.uint64).max. This would address why numbers like +/- 100000000000000000000 can cause issues.

That being said, your examples do point to some issues:

• We should be documenting that we can accept scalars for pd.to_numeric, which is not in the docs but is accepted in the implementation.
• The consistency of behavior between 100000000000000000000 and [100000000000000000000]. That looks buggy to me...

[fakio]

I was asking you whether you would be okay with that as an end user.

It's tough, I don't usually deal with big numbers, but I think converting to float is ok, and it is consistent with pd.to_numeric([100000000000000000000]) already doing it.

The consistency of behavior between 100000000000000000000 and [100000000000000000000]. That looks buggy to me...

and '10000000000000000000000000', they all give three different results

Also strange:

pd.to_numeric([10000000000000000000000000, '10000000000000000000000000'])
array([1.e+25, 1.e+25])

pd.to_numeric(['10000000000000000000000000'])
ValueError: Integer out of range. at position 0