Fix flaky behavior of test_xeb_fidelity #6337
Conversation
Increase tolerance for fidelities calculated with and without state vector amplitudes. Avoid flaky, random-seed dependent test failures on a slightly excessive difference. Example: https://github.com/quantumlib/Cirq/actions/runs/6726576064/job/18283075054#step:6:598
pavoljuhas
requested review from
mrwojtek,
vtomole,
cduck and
a team
as code owners
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #6337 +/- ##
==========================================
- Coverage 97.83% 97.83% -0.01%
==========================================
Files 1108 1108
Lines 96352 96352
==========================================
- Hits 94270 94269 -1
- Misses 2082 2083 +1
☔ View full report in Codecov by Sentry. |
| @@ -81,7 +81,7 @@ def test_xeb_fidelity(depolarization, estimator): | |||
| f2 = cirq.xeb_fidelity( | |||
| circuit, bitstrings, qubits, amplitudes=amplitudes, estimator=estimator | |||
| ) | |||
| assert np.abs(f - f2) < 1e-6 | |||
| assert np.abs(f - f2) < 2e-6 | |||
There was a problem hiding this comment.
lets use pytest.approx instead
assert f2 == pytest.approx(f, rel=1e-6)This will assert that
assert f2 == pytest.approx(f, rel=1e-6, abs=1e-6)so that it works for all cases
There was a problem hiding this comment.
@NoureldinYosri - thank you for the suggestion, done in 9ddcbaa.
Good to know about pytest.approx it is however not used in cirq. numpy.allclose does the same thing here so I opted for that one.
There was a problem hiding this comment.
relative tolerance works only for large numbers. for small number absolute tolerance is needed so
np.allclose(f, f2, atol=1e-6, rtol=1e-6)There was a problem hiding this comment.
In this case the initial absolute value comparison was better as the fidelity is supposed to be on the [0, 1] interval.
For some reason the round-off errors on Mac got it over the threshold.
Increase tolerance for fidelities calculated with and without state vector amplitudes. Avoid flaky test failures due to round-off errors. Example: https://github.com/quantumlib/Cirq/actions/runs/6726576064/job/18283075054#step:6:598
Increase tolerance for fidelities calculated with and without state
vector amplitudes. Avoid flaky, random-seed dependent test failures
on a slightly excessive difference.
Example: https://github.com/quantumlib/Cirq/actions/runs/6726576064/job/18283075054#step:6:598