PERF: maybe_convert_numeric speedup

[chris-b1]

xref #16043

Probably more to be done, but picks up some easy performance on the int path. For some reason I'm having trouble running asv, here are some timings.

setup

ints = np.array(list(range(1000000)), dtype=object)
floats = np.array([float(x) for x in range(1000000)], dtype=object)
ints_with_float = ints.copy()
ints_with_float[-1] = 1.2

Master

In [2]: %timeit pd.to_numeric(floats)
10 loops, best of 3: 132 ms per loop

In [3]: %timeit pd.to_numeric(ints)
1 loop, best of 3: 437 ms per loop

In [4]: %timeit pd.to_numeric(ints_with_float)
1 loop, best of 3: 444 ms per loop

PR

In [5]: %timeit pd.to_numeric(floats)
10 loops, best of 3: 88.9 ms per loop

In [6]: %timeit pd.to_numeric(ints)
10 loops, best of 3: 57.6 ms per loop

In [7]: %timeit pd.to_numeric(ints_with_float)
1 loop, best of 3: 392 ms per loop

[jreback]

you might be able to

cdef ival <int> val
if ival >= 0:
     ...
if ival <= iINT64_max:
    ..

[chris-b1]

I can't do exactly this because val could be larger than fits into a c int

[jreback]

the other thing is it might be worth it to actually try to cast once you have found an actual int (or a uint),

In [13]: (ints.astype('int64', casting='unsafe') == ints).all()
Out[13]: True

In [14]: %timeit (ints.astype('int64', casting='unsafe') == ints).all()
55.5 ms ± 1.15 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

At this point it would raise if you encounter a float so this is ok (you can do it in a try/except).

not sure if this is totally feasible though.

[chris-b1]

OK, I've added some fastpath logic that will try and convert the whole array at once - have to see if anything breaks!

[TomAugspurger]

This should go in 0.20, correct?

[chris-b1]

I need to work through a couple more tests, but assuming I can get this to work in a non-breaking way, yes I'd think this would be good for 0.20. `to_numeric` performance with ints has regressed quite a bit since 0.19.1 (although is more functional now, to be fair)

…

On Mon, Apr 24, 2017 at 10:07 AM, Tom Augspurger ***@***.***> wrote: This should go in 0.20, correct? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#16104 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB1b_BJWgLonT_hQzRK2-WCm3Rn7XDdOks5rzLqbgaJpZM4NFkMz> .

[codecov]

Codecov Report

Merging #16104 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #16104      +/-   ##
==========================================
- Coverage   90.84%   90.84%   -0.01%     
==========================================
  Files         159      159              
  Lines       50775    50768       -7     
==========================================
- Hits        46126    46118       -8     
- Misses       4649     4650       +1

Flag	Coverage Δ
#multiple	88.62% <ø> (-0.01%)	⬇️
#single	40.33% <ø> (-0.01%)	⬇️

Impacted Files	Coverage Δ
pandas/core/common.py	90.68% <0%> (-0.35%)	⬇️
pandas/util/testing.py	79.07% <0%> (-0.16%)	⬇️
pandas/api/types/__init__.py	100% <0%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e501e1d...9757099. Read the comment docs.

[jreback]

nice! thanks @chris-b1

[jorisvandenbossche]

Yeah, I put the 0.21 milestone on it as it was about performance, but forgot it was for a perf regressiono in 0.20dev. So certainly fine. Thanks Chris!