Potential regression induced by commit 9cd5e55

[rhshadrach]

PR #58126 may have induced a performance regression. If it was a necessary behavior change, this may have been expected and everything is okay.

Please check the links below. If any ASVs are parameterized, the combinations of parameters that a regression has been detected for appear as subbullets.

• algos.isin.IsInWithLongTupples.time_isin - 14.034% (508.807us)
• algos.isin.IsinWithArange.time_isin
  • dtype=<class 'numpy.object_'>; M=1000; offset_factor=-2 - 12.132% (1.021ms)
• algos.isin.IsinWithArange.time_isin
  • dtype=<class 'numpy.object_'>; M=2000; offset_factor=-2 - 12.323% (1.027ms)
• algos.isin.IsinWithArange.time_isin
  • dtype=<class 'numpy.object_'>; M=8000; offset_factor=-2 - 11.280% (955.871us)
• categoricals.Concat.time_union - 37.484% (532.870us)
• frame_methods.Fillna.time_bfill
  • inplace=False; dtype='datetime64[ns]' - 15.926% (311.548us)
• frame_methods.Fillna.time_bfill
  • inplace=False; dtype='timedelta64[ns]' - 17.231% (329.702us)
• groupby.GroupByCythonAggEaDtypes.time_frame_agg
  • dtype='Int32'; method='max' - 21.470% (6.060ms)
• groupby.GroupByCythonAggEaDtypes.time_frame_agg
  • dtype='Int32'; method='min' - 19.713% (5.597ms)
• groupby.SumTimeDelta.time_groupby_sum_timedelta - 8.461% (328.890us)
• index_cached_properties.IndexCache.time_is_monotonic_increasing
  • index_type='MultiIndex' - 159.232% (334.134us)
• indexing.MultiIndexing.time_loc_all_lists
  • unique_levels=False - 68.380% (10.768ms)
• indexing.MultiIndexing.time_loc_all_slices
  • unique_levels=False - 85.802% (5.741ms)
• indexing.MultiIndexing.time_loc_null_slice_plus_slice
  • unique_levels=False - 72.830% (6.182ms)
• indexing.MultiIndexing.time_loc_partial_key_scalar
  • unique_levels=False - 87.309% (155.075us)
• indexing.MultiIndexing.time_loc_partial_key_slice
  • unique_levels=False - 98.589% (93.620us)
• indexing.MultiIndexing.time_loc_slice_plus_null_slice
  • unique_levels=False - 71.309% (5.589ms)
• indexing.MultiIndexing.time_xs_level_0
  • unique_levels=False - 85.092% (152.792us)
• indexing_engines.ObjectEngineIndexing.time_get_loc
  • index_type='monotonic_decr' - 6.460% (237.419us)
• multiindex_object.Append.time_append
  • dtype='datetime64[ns]' - 19.228% (325.831us)
• multiindex_object.Append.time_append
  • dtype='string' - 33.042% (525.673us)
• multiindex_object.Difference.time_difference
  • dtype='datetime' - 41.051% (410.417us)
• multiindex_object.Difference.time_difference
  • dtype='ea_int' - 17.626% (193.866us)
• multiindex_object.Difference.time_difference
  • dtype='int' - 46.220% (436.393us)
• multiindex_object.Integer.time_get_indexer_and_backfill - 41.843% (12.831ms)
• multiindex_object.Integer.time_get_indexer_and_pad - 36.088% (11.098ms)
• multiindex_object.Integer.time_is_monotonic - 77.282% (112.379ns)
• multiindex_object.SetOperations.time_operation
  • index_structure='monotonic'; dtype='datetime'; method='intersection'; sort=None - 82.267% (3.657ms)
• multiindex_object.SetOperations.time_operation
  • index_structure='monotonic'; dtype='datetime'; method='union'; sort=None - 65.657% (1.328ms)
• multiindex_object.SetOperations.time_operation
  • index_structure='monotonic'; dtype='ea_int'; method='intersection'; sort=None - 98.106% (4.134ms)
• multiindex_object.SetOperations.time_operation
  • index_structure='monotonic'; dtype='ea_int'; method='union'; sort=None - 71.847% (1.372ms)
• multiindex_object.SetOperations.time_operation
  • index_structure='monotonic'; dtype='int'; method='intersection'; sort=None - 80.503% (3.571ms)
• multiindex_object.SetOperations.time_operation
  • index_structure='monotonic'; dtype='int'; method='union'; sort=None - 61.772% (1.258ms)
• sparse.SparseArrayConstructor.time_sparse_array
  • dense_proportion=0.1; fill_value=0; dtype=<class 'object'> - 13.517% (2.279ms)
• tslibs.resolution.TimeResolution.time_get_resolution
  • unit='D'; size=1000000; tz=None - 7.399% (1.251ms)
• tslibs.resolution.TimeResolution.time_get_resolution
  • unit='D'; size=1000000; tz=datetime.timezone(datetime.timedelta(seconds=3600)) - 7.239% (1.257ms)
• tslibs.resolution.TimeResolution.time_get_resolution
  • unit='D'; size=1000000; tz=datetime.timezone.utc - 8.578% (1.437ms)
• tslibs.resolution.TimeResolution.time_get_resolution
  • unit='D'; size=10000; tz=None - 6.995% (11.901us)
• tslibs.resolution.TimeResolution.time_get_resolution
  • unit='D'; size=10000; tz=datetime.timezone.utc - 6.908% (11.753us)
• tslibs.resolution.TimeResolution.time_get_resolution
  • unit='h'; size=1000000; tz=None - 6.960% (1.182ms)
• tslibs.resolution.TimeResolution.time_get_resolution
  • unit='h'; size=1000000; tz=datetime.timezone(datetime.timedelta(seconds=3600)) - 8.014% (1.382ms)
• tslibs.resolution.TimeResolution.time_get_resolution
  • unit='h'; size=1000000; tz=datetime.timezone.utc - 8.499% (1.426ms)
• tslibs.resolution.TimeResolution.time_get_resolution
  • unit='h'; size=10000; tz=datetime.timezone.utc - 6.698% (11.416us)
• tslibs.resolution.TimeResolution.time_get_resolution
  • unit='us'; size=1000000; tz=datetime.timezone(datetime.timedelta(seconds=3600)) - 6.188% (1.183ms)
• tslibs.resolution.TimeResolution.time_get_resolution
  • unit='us'; size=10000; tz=datetime.timezone(datetime.timedelta(seconds=3600)) - 6.508% (12.626us)
• tslibs.tz_convert.TimeTZConvert.time_tz_localize_to_utc
  • size=1000000; tz=datetime.timezone(datetime.timedelta(seconds=3600)) - 11.055% (84.773us)
• tslibs.tz_convert.TimeTZConvert.time_tz_localize_to_utc
  • size=10000; tz=datetime.timezone(datetime.timedelta(seconds=3600)) - 18.081% (1.763us)

Subsequent benchmarks may have skipped some commits. The link below lists the commits that are between the two benchmark runs where the regression was identified.

Commit Range

cc @lithomas1

[rhshadrach]

Surprised by the one. Is it possible the building with NumPy 2.0rc1 induced a performance regression?

[lithomas1]

Potentially.

Can you verify that the version of numpy used at runtime was unchanged?

The is_monotonic change looks concerning.

Also, not sure how you've automated the reporting, but it would be cool to put the regression factor next to the benchmark name (some of the regressions are small ~10% change).

[rhshadrach]

Can you verify that the version of numpy used at runtime was unchanged?

Unfortunately no - unless we have a hard pin, we don't save what version was used in the benchmark.

Also, not sure how you've automated the reporting, but it would be cool to put the regression factor next to the benchmark name (some of the regressions are small ~10% change).

Great idea - I'm thinking something like

• algos.isin.IsInWithLongTupples.time_isin - 14% (0.2ms)

[rhshadrach]

@lithomas1 - I've updated the OP with the new format.