nunique is slower than len(set(x.dropna())) for smaller Series.

[lexual]

This is biting me when applying nunique on some groupby operations.

In one benchmark, which I'll shortly share, nunique() is slower until we get to a length of 3000.

[lexual]

See a benchmark here:

https://gist.github.com/lexual/118084593a98aa472289

[lexual]

So I'm wondering if we can use len(set(x.dropna())) where the Series length is below a certain threshold.

e.g. pseudo code.

def nunique(s):
    threshhold = 3000
    if len(s) < threshold:
        return len(set(s.dropna()))
    else:
        return current_nunique_implementation(s)

[jreback]

actually this looks changed recently

time this with

len(unique())

this was replaced by len(value_counts())

iIIRC because that handles all dtypes properly (this is in core/base.py)

[lexual]

Yes len(s.unique()) looks faster across the board, is that what nunique() is now using?

[jreback]

I think it was
was changed somewhat recently

cc @sinhrks ?

[lexual]

https://github.com/pydata/pandas/blob/master/pandas/core/base.py#L295

?

[jreback]

u want I do a pr with a vbench and change to using len(unique)
(also need to handle dropna argument)
FYI this is used by series and index which doesn't have dropna (that may be why value_counts is used(

[jreback]

yep

[lexual]

Updated my gist benchmark thing:

https://gist.github.com/lexual/118084593a98aa472289

[sinhrks]

nunique had been used value_counts prior to #6734, thus I'm not sure the actual background. it may be a reason that unique and value_counts handles NaT differently (fixed in #7424).

I agree len(series.unique()) looks better.

[jreback]

@sinhrks yep the NaT treatment might be slightly different (so maybe need to fix that!!).

@lexual since you brought this up, wnt to do a PR to revert back to using unique, and see if anything pops up?