Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend the device test suite: gradient, op-arithmetic and template tests #5273

Merged
merged 40 commits into from
Apr 12, 2024

Conversation

timmysilv
Copy link
Contributor

@timmysilv timmysilv commented Feb 27, 2024

I ran the plugin test suite against this branch so I could see which tests fail right away on which devices (done, see here, braket and quantuminspire are failing for their own reasons), and skip them for those devices. Everything is now passing as expected.

Context:
The device test suite doesn't cover a lot of PennyLane's features, so I'm adding more coverage.

Description of the Change:

  • modified CI to install interfaces if needed (basically always except for default.qubit.autograd)
  • updated test_measurements.py to test some basic arithmetic ops by comparing the result to default.qubit
  • Added 5 basic differentiation tests for each interface: test_basic_grad, test_backprop_state, test_parameter_shift, test_probs, test_multi_meas test_hessian. They do what they sound like they do 😄
  • Added a test for every single template in PennyLane 🐳 I just grabbed the example from each template's docstring, got the result with default.qubit, and I'm asserting that every other device also gets that result. If the docstring didn't have one (or it just returned 1 when there are better examples), I tried to get one from the tests for that template. I'd also try to swap qml.state() with qml.probs() whenever possible so the test would work with finite-shot devices.

Benefits:
More confidence in our plugin devices!

Possible Drawbacks:

  • Slower device tests
  • New failures to manage?
  • Some tests needed explicit skips from certain devices. I opened a story to track those explicit skips being done in the device test suite upgrade epic

[sc-57488]

Copy link
Contributor

Hello. You may have forgotten to update the changelog!
Please edit doc/releases/changelog-dev.md with:

  • A one-to-two sentence description of the change. You may include a small working example for new features.
  • A link back to this PR.
  • Your name (or GitHub username) in the contributors section.

Copy link

codecov bot commented Feb 27, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.66%. Comparing base (b2b0c36) to head (07b432f).
Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5273      +/-   ##
==========================================
- Coverage   99.67%   99.66%   -0.01%     
==========================================
  Files         404      404              
  Lines       37833    37554     -279     
==========================================
- Hits        37709    37429     -280     
- Misses        124      125       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@timmysilv timmysilv marked this pull request as ready for review February 29, 2024 22:14
@timmysilv timmysilv requested review from trbromley and a team February 29, 2024 22:14
@timmysilv timmysilv changed the title extend the device test suite with more gradient tests extend the device test suite: gradient, op-arithmetic and template tests Feb 29, 2024
@Alex-Preciado Alex-Preciado requested review from albi3ro and removed request for a team March 21, 2024 13:40
@astralcai astralcai self-requested a review March 22, 2024 19:35
Copy link
Contributor

@astralcai astralcai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Just a few small comments there.

Copy link
Contributor

@astralcai astralcai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks Good! 🎸

@timmysilv
Copy link
Contributor Author

this change will require branch protection updates because of device-tests (default.qubit.autograd, None, jax,tf,torch, 3.9), cc @albi3ro

@Alex-Preciado Alex-Preciado requested a review from albi3ro April 5, 2024 15:48
@mudit2812 mudit2812 self-requested a review April 12, 2024 18:25
@mudit2812 mudit2812 enabled auto-merge (squash) April 12, 2024 19:07
@mudit2812 mudit2812 disabled auto-merge April 12, 2024 19:07
@mudit2812 mudit2812 enabled auto-merge (squash) April 12, 2024 19:14
@mudit2812 mudit2812 merged commit f7834de into master Apr 12, 2024
38 checks passed
@mudit2812 mudit2812 deleted the more-device-tests branch April 12, 2024 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants