Skip to content

Stop timing NumPy in the ASV benchmarks - #3095

Open
antonwolfy wants to merge 2 commits into
masterfrom
benchmarks-drop-numpy-executor
Open

antonwolfy wants to merge 2 commits into
masterfrom
benchmarks-drop-numpy-executor

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

This PR removes the numpy executor from the ASV benchmark suites, so bench_elementwise.py, bench_linalg.py and bench_random.py now time only dpnp.

The suites track dpnp performance from commit to commit, and nothing compares the dpnp and NumPy timings with each other. A dpnp change cannot move a NumPy timing, so those points carry no signal. They did have two costs:

  • They made up half of the timed points in these three suites (470 of 940).
  • Machine noise on them was reported as a dpnp regression. A master commit that only bumped pre-commit hooks was flagged solely because of MatMul.time_matmul('numpy', 16, 'float64') at 1.22x. On a recent PR, two of the three reported regressions were NumPy points, including MatMul.time_dot('numpy', 64, 'float32') at 1.6x, with the same NumPy and MKL builds on both sides.

The change keeps the recorded results comparable:

  • executor stays as a single-value axis ('dpnp'), so result names such as time_dot('dpnp', 64, 'float32') are unchanged.
  • The setup and time_* methods are not modified, including the numpy branches some setup methods no longer use. ASV versions a benchmark by the source of these methods, so editing them would make every result incomparable with the recorded baseline. All 29 such methods are byte-identical to master.

Against a baseline recorded before this change, the NumPy points exist only on the baseline side. asv compare treats them as skipped, so --only-changed hides them and they are not counted as regressions.

benchmarks/README.md is updated to match and gets a new "NumPy is not timed" section explaining the above.

  • Have you provided a meaningful PR description?
  • Have you added a test, reproducer or referred to an issue with a reproducer?
  • Have you tested your changes locally for CPU and GPU devices?
  • Have you made sure that new changes do not introduce compiler warnings?
  • Have you checked performance impact of proposed changes?
  • Have you added documentation for your changes, if necessary?
  • Have you added your changes to the changelog?

A dpnp change cannot move a NumPy timing, so the numpy executor points
only doubled the run time of the elementwise, linalg and random suites and
let machine noise on them be reported as a dpnp regression.

Keep `executor` as a single-value axis and leave the `setup`/`time_*`
bodies untouched, so result names and ASV benchmark versions stay
comparable with the recorded history.
@antonwolfy antonwolfy added this to the 0.21.0 release milestone Oct 12, 2026
@antonwolfy antonwolfy self-assigned this Oct 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

View rendered docs @ https://intelpython.github.io/dpnp/pull/3095/index.html

@coveralls

coveralls commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Coverage Status

No base build to compare — benchmarks-drop-numpy-executor into master

@antonwolfy

Copy link
Copy Markdown
Contributor Author

Note, the 65536-size unary entries still keep making about half the runs UNSTABLE, but will be addressed in the follow-up.

@antonwolfy
antonwolfy marked this pull request as ready for review October 12, 2026 14:12
@antonwolfy

Copy link
Copy Markdown
Contributor Author

Note, the 65536-size unary entries still keep making about half the runs UNSTABLE, but will be addressed in the follow-up.

@github-actions

Copy link
Copy Markdown
Contributor

Array API standard conformance tests for dpnp=0.21.0dev13=np2py314h8d9cdd5_7 ran successfully.
Passed: 1376
Failed: 0
Skipped: 6

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants