PERF: fastpaths in is_foo_dtype checks

[jbrockmendel]

xref #33364, partially addresses #33368

For exposition I used the new type checking functions in parsers.pyx.

For the implementation I chose .type and .kind checks in order to avoid needing the imports from dtypes.dtypes, which has dependencies on a bunch of other parts of the code.

In [1]: import pandas as pd                                                                                                                                                                                                                                                                                                    
In [2]: from pandas.core.dtypes.common import *                                                                                                                                                                                                                                                                                
In [3]: cat = pd.Categorical([])                                       
In [4]: arr = np.arange(5)
                                                                                                                                                                         
In [5]: %timeit is_categorical_dtype(cat.dtype)                                                                                                                                                                                                                                                                                
1.57 µs ± 34.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [6]: %timeit is_cat_dtype(cat.dtype)                                                                                                                                                                                                                                                                                        
316 ns ± 3.78 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [7]: %timeit is_extension_array_dtype(cat.dtype)                                                                                                                                                                                                                                                                            
364 ns ± 5.39 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [8]: %timeit is_ea_dtype(cat.dtype)                                                                                                                                                                                                                                                                                         
270 ns ± 7.31 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [9]: %timeit is_extension_array_dtype(arr.dtype)                                                                                                                                                                                                                                                                           
759 ns ± 5.16 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [10]: %timeit is_ea_dtype(arr.dtype)                                                                                                                                                                                                                                                                                        
199 ns ± 8.77 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

[jorisvandenbossche]

Related to the discussion we had about this on the chat, to give an example of the strategy that I tried to explain:

The current is_categorical_dtype is defined as:

def is_categorical_dtype(arr_or_dtype) -> bool:
    if arr_or_dtype is None:
        return False
    return CategoricalDtype.is_dtype(arr_or_dtype)

where CategoricalDtype.is_dtype() is the catch-all / check-all slow check.

So for this case, we can put the fast check as in this PR first:

def is_categorical_dtype(arr_or_dtype) -> bool:
    if isinstance(dtype, ExtensionDtype):
        # fast check for extension dtype
        return dtype.name == "category"
    elif isinstance(dtype, np.dtype):
        # fast check for numpy dtype (always False)
        return False
    elif arr_or_dtype is None:
        return False
    # keep the slow check if the fast ones didn't return
    return CategoricalDtype.is_dtype(arr_or_dtype)

And in that way, we don't have to change every occurrence of is_categorical_dtype internally to the fast function.

It might be that categorical is a simple example, but I would think that a similar pattern can be followed for the others as well.

[jbrockmendel]

Related to the discussion we had about this on the chat, to give an example of the strategy that I tried to explain:

Also as discussed on the chat, the advantage of the dtype-only versions is that we dont risk silently going down the slow path.

[jreback]

Related to the discussion we had about this on the chat, to give an example of the strategy that I tried to explain:

Also as discussed on the chat, the advantage of the dtype-only versions is that we dont risk silently going down the slow path.

yep i would start by using the approach outlined above.

[jreback]

this needs to be changed to use the approach @jorisvandenbossche outlined, e.g. not changing the names of the current is_* functions, just enhancing the impl with a fast path.

[jbrockmendel]

I'm happy to update the existing functions to be faster, but we should also have the strict versions internally to make sure we actually use the fastpath

[jbrockmendel]

Actually, adding the fastpath to the existing checks will actually slow them down slightly in cases where we're not passing a dtype obj

[TomAugspurger]

I think I'm OK with slowing down the non-dtype case.

I'm also OK with deprecating non-dtype entirely, but that's a larger task that can be saved for later. Short-term, I think we can rely on reviewers to catch places where we're passing non-dtype's to is_foo_dtype.

Edit: all of the above is contingent with actually being able to quickly do an is_foo_dtype on a dtype as a fastpath. I only checked for categorical. Hopefully it's doable for the rest.

[jreback]

I think I'm OK with slowing down the non-dtype case.

I'm also OK with deprecating non-dtype entirely, but that's a larger task that can be saved for later. Short-term, I think we can rely on reviewers to catch places where we're passing non-dtype's to is_foo_dtype.

Edit: all of the above is contingent with actually being able to quickly do an is_foo_dtype on a dtype as a fastpath. I only checked for categorical. Hopefully it's doable for the rest.

agreed here

[jbrockmendel]

One more doomed pitch for the strict versions: they're really nice dependency-structure-wise. ATM dtypes.common depends on dtypes.dtypes, which has runtime imports and depends on a bunch of the code. With just the strict versions, we can get dtypes.common (which we import from basically everywhere) to depend only on things "above" it in the dependency structure.

I know others dont care about the dependency structure as much as I do, will update.

[jbrockmendel]

updated+green