Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too tight tolerance in one of the tests? #295

Closed
KristofferC opened this issue May 15, 2019 · 2 comments
Closed

Too tight tolerance in one of the tests? #295

KristofferC opened this issue May 15, 2019 · 2 comments

Comments

@KristofferC
Copy link

In the PkgEval run for the 1.2 release branch Convex.jl errored with

max atom: Test Failed at /root/.julia/packages/Convex/HwTSz/test/test_lp.jl:61
  Expression: ≈(p.optval, max(max_a, max_b), atol=10TOL)
   Evaluated: 0.9898189279146101 ≈ 1.0 (atol=0.01) 

Running the tests locally, they pass so I think the tolerance is perhaps a bit too tight?

@ericphanson
Copy link
Collaborator

ericphanson commented Jun 28, 2019

Looking at the git history, the tolerance for this test was already pushed up by 10x in #239. From the numbers in #295 (comment), something like atol=0.011 might be enough here (whereas it's currently atol=0.01) so maybe it doesn't need that much more, but the tolerance is already pretty high so maybe there's some other underlying problem.

So one option might be to revert the tolerance to the standard atol=1e-3, set it to @test_broken and open an issue for it. (I guess @test_broken might not work if it passes in some solvers but not others, in which case we could just comment out the test). Or just bump the tolerance up again.

@odow
Copy link
Member

odow commented Feb 23, 2022

Closing as stale. There are going to be tolerance issues with the tests, particularly when underlying solver versions change. The default tolerances are already quite loose. We've also fixed a lot of the discrepancies in binary distributions by using JLLs.

@odow odow closed this as completed Feb 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

3 participants