-
Notifications
You must be signed in to change notification settings - Fork 217
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Going below sqrt(eps) #631
Comments
Sort-of relevant: JuliaNLSolvers/LineSearches.jl#120 |
Very relevant! So, now with JuliaNLSolvers/LineSearches.jl#123 in, with the code above and the default initial alpha, we have:
So that looks like a good test case ;-) |
After JuliaNLSolvers/LineSearches.jl#136, HagerZhang now doesn't error ;) using Optim, LineSearches, LinearAlgebra
n = 10
M = randn(n,n)
M = I+M'M
A = (M+M')/2
b = randn(n)
f(x) = dot(x,A*x) + b'x
g(x) = 2*A*x + b
g!(stor,x) = copyto!(stor,g(x))
x0 = randn(n)
Optim.optimize(f, g!, x0, Optim.ConjugateGradient(linesearch=LineSearches.HagerZhang()), Optim.Options(show_trace=true,allow_f_increases=true,f_tol=-1.0, g_tol=-1.0, x_tol=-1.0)) |
Some general thoughts: Although for simple problems HZ is very impressive, I've found that in "practical" cases, Hager Zhang doesn't do much better than the others. When the optimisers crash, then I normally hack something together, but the solution is always to avoid objective evaluations altogether. Some of the following work:
|
Makes you wonder: was it the actual bug fix, or the handling around zero ? :) Probably the latter, but... |
Well the latter was a bug too :) |
I still get various errors like
(with the code pasted above). That also happens with other optimizers. BackTracking does not error, but stagnates around sqrt(eps). @cortner That depends somewhat on the optimizer, but for LBFGS I agree with you. Line searches should only be used to add some robustness, not help convergence. It seems a simple backtracking strategy works best. |
I get the line search failure warning but it exits gracefully, no errors, with |
The desired output is that the method converges efficiently to machine precision (ie a gradient of the order of machine precision). Currently HZ achieves this, but the other linesearches don't. After that, it's just a stress test that might help catch bugs. |
I've noted the point about the lower bound on the step size and will keep this in mind going forward, but otherwise this isn't really actionable. Thanks for the discussion. |
A number of linesearches get into various trouble when going below the dreaded sqrt(eps) precision. Example:
Change the linesearch and method to explore a variety of different bugs (JuliaNLSolvers/LineSearches.jl#123 is just the first one, there are others hidden behind).
It is not completely clear what to do in this regime, in which differences of objective functions don't make much sense. Of course this is an extreme precision regime, but it's sometimes useful to go there. HagerZhang seems to be consistently able to converge to machine precision, I'm not sure how. A simple solution would be to have a minimum linesize of
C*eps(T)/dphi_0
, which should be OK in most cases?cc @cortner, who has experience in this matter
The text was updated successfully, but these errors were encountered: