PERF: avoid unnecessary method call in get_indexer_non_unique() on MultiIndex

[morotti]

Hello,

There was a regression that made reindex 60x slower, in pandas v0.23 to v1.4, that was finally fixed by this PR last year
discussion #23735
commit https://github.com/pandas-dev/pandas/pull/46235/files
a benchmark was added in a later commit https://github.com/pandas-dev/pandas/pull/47221/files

That resolved the issue in get_indexer().
I was going through pandas source code and I noticed there is a second code path in get_indexer_non_unique() with the same problem.
This PR fixes the remaining code path.

I'm not super familiar with pandas, I'm not sure what might hit the default indexer vs the non unique indexer.
It would be great if you can think of something and add a benchmark.
It looks like the exact same bug and nearly the same code to me, it might just be a 60x performance improvement too 😄

@jreback
@lukemanley
@jbrockmendel

Cheers.

• closes PERF: regression in reindex. Pandas 0.23.4 is 60x slower than 0.22.0 with a MultiIndex with datetime64 #23735 (Replace xxxx with the GitHub issue number)
• [NA] Tests added and passed if fixing a bug or adding a new feature
• [NA] All code checks passed.
• [NA] Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[morotti]

Thank you,

I've done some further analysis comparing to the previous PR that fixed the first code path.
There was a batch of PRs that were merged around the same time and fixed the performance issue in 1.4, all together there was a 100x slowdown on reindex (and get_indexer) but that doesn't seem to affect get_indexer_non_unique.

I think the PR is good to merge to avoid the extra function call, though it hardly makes any difference on performance.

on a side note, there appear to be a performance regression in reindex between 1.5.3 and 2.0.3 by as much as 10%
it's quite small and close to the noise between runs.

reindex benchmark

pandas 0.22 = 0.020 seconds
pandas 1.0 = 1.431 seconds
pandas 1.3 = 2.164 seconds
pandas 1.5 = 0.020 seconds
pandas 1.5 with PR = 0.020 seconds
pandas 2.0 = 0.023 seconds
pandas 2.0 with PR  0.023 seconds

get_index_non_unique benchmark
pandas 0.22 = 0.550 seconds
pandas 1.0 = 0.071 seconds
pandas 1.3 = 0.074 seconds
pandas 1.5 = 0.063 seconds
pandas 1.5 with PR = 0.063 seconds
pandas 2.0 = 0.064 seconds
pandas 2.0 with PR = 0.064 seconds

(ignore the 1ms difference that is noise between runs)

# benchmark for reindex and get_indexer
import pandas as pd
import time
import numpy as np


if __name__ == '__main__':
    n_days = 300
    dr = pd.date_range(end="20181118", periods=n_days)
    mi = pd.MultiIndex.from_product([dr, range(1440)])

    v = np.random.randn(len(mi))
    mask = np.random.rand(len(v)) < .3
    v[mask] = np.nan
    s = pd.Series(v, index=mi)
    s = s.sort_index()

    s2 = s.dropna()

    start = time.time()

    s2.reindex(index=s.index)

    end = time.time()
    print("pandas version: %s" % pd.__version__)
    print("reindex took %s seconds" % (end - start))

# benchmark for non-unique
import pandas as pd
import time
import numpy as np


if __name__ == '__main__':
    N = 1_000_000

    mi1 = pd.MultiIndex.from_arrays([np.ones(N), np.ones(N)])
    mi2 = pd.MultiIndex.from_tuples([(1, 1)])

    start = time.time()

    mi1.get_indexer_non_unique(mi2)

    end = time.time()
    print("pandas version: %s" % pd.__version__)
    print("reindex took %s seconds" % (end - start))

[lukemanley]

Thanks. Can you remove the whatsnew note in that case and maybe retitle this PR to something like: "CLN: avoid unnecessary method call in ..."

[morotti]

adjusted commits and PR titles

[mroeschke]

Thanks @morotti