Optim of Tomorrow

[pkofod]

aka Optim Roadmap 2.0 aka Make Optim Greater Still...

These lists are up for discussion, I just figured I would start that discussion now. JuliaLang v1.0 is approaching rapidly, with v0.6 feature freeze not far into the future. I am not sure we should tag Optim v1.0 the day JuliaLang starts shipping as non-beta, but it would be cool to have a lot of the current issues sorted out.

Let the number of issues guide you as to what I will tackle first :)

General

• Replace df.f(x), df.g!(x, stor) and df.h!(x, stor) with value(df, x), gradient!(df, x) and hessian!(df, x) where the old storage argument is embedded in dfswitch argument order for gradient and hessian evaluations? #156 optim-julep: define AbstractProblem and make use of it in optimizers #163 Finite differences: shouldn't we count f_calls? #219 Add limits to f_calls, g_calls and h_calls #241 Changes to DifferentiableFunction API #287 OEP: function-objects should keep track of their own call counts #305 OEP: more general state for user-supplied functions #306
• Restore matrix api ref Inappropriate StackOverflowError with unexpected array type #198 Fix problem dimensions inside the method #312
• Figure out what to do about tests and benchmarks. The current idea seems to be to have some testing here, and then extensive benchmarking and testing in OptimTests.jl. Relevant issues and PRs: continuous performance testing jump-dev/JuMP.jl#42 Add unconstrained benchmarks. OptimTests.jl#7
• Move problems and some testing infrastructure out of Optim
• Add limits to f_/g_/h_calls Add limits to f_calls, g_calls and h_calls #241
  /hessian! interface Can I maximize a function using Optim.jl? #308 (I intend to implement this together with the first bullet) through a keyword and a new function
• Figure out if we want to integrate with JuMP/MathProgBase usage with JuMP? #107 Generalize Optim.jl to be an interface for nonlinear optimization #309 (see for mathprogbase at least https://github.com/JuliaNLSolvers/OptimMPB.jl)
• Better/additional/flexible stopping criteria? misreported non-convergence of convex function #374

Documentation

• General QA on the contents
• Identify weak pages
• Add references

Constrained

• Read through grants.gov to find money for @timholy so he can take a sabbatical from his boring image lab, and work on his fun fun fun constrained optimizer instead! (@anriseth did it for free instead :) )

Solvers (up for grabs if the person is willing to "maintain")

• Subplex Feature Request: Subplex #272
• Parallel NelderMead Feature Request: parallel Nelder-Mead #268

In general I think we might want to delete feature/solver requests and have a single issue for them, as the requesters are rarely the ones willing to write up and maintain anyways, so the issues always end up idle.

More
... and more! I think 2017 is going to be a good year for Optim.

[cortner]

I'd like to add

• rewrite line searches as types (I realise this is LineSearches.jl, but it can only go hand in hand with Optim.jl); see LineSearchOptions LineSearches.jl#9
• an intelligent objective that avoids multiple evaluations; Multiple f evaluations for same state #282
• a benchmark function that, given a small set of objectives, tests a variety of methods maybe with different parameters, and recommends the optimal configuration, which can then be reused for multiple future runs.

[pkofod]

Those are good points.

Dispatch based linesearches
The first one has to go hand in hand with NLsolve as well I guess, @KristofferC . It would be great to make an interface such that it would be easy to call a solver here with a linesearch not in LineSearches.jl, and I think the dispatch based idea you have in mind does exactly that. This would to some degree separate the work here and in LineSearches, to the benefit of both parties.
Function evaluations
I think the second point relates to the DifferentiableFunction-rewrite. We could certainly make it so that value(df, x) only actually evaluated f it x was different from last time. This would solve it right?

function value(df, x) 
    if is_new(x, df.last_x)
        df.f_calls += 1
        df.f_x = df.f(x)
        copy!(df.last_x, x)
    end
    df.f_x
end

Benchmarks
The benchmarking-function sounds fun, but that should probably be started outside of Optim. It can also be started as a PR here, but I think it would need some work before we can merge is as an official part of this package. You are thinking something like recommend(problems) where problems is some collection of solver/starting point/parameter specifications? I guess it could also be based on CUTEst problems and more. That is, some sort of recommender system in a stored state based on the large problem collection, and then you could add your own, which would maybe have a relatively large weight attached and update the parameters of the recommender system. I'm not sure how well it would work in practice, but we could start the discussion over at JuliaML (gitter?).

[cortner]

Function evaluations
I think the second point relates to the DifferentiableFunction-rewrite. We could certainly make it so that value(df, x) only actually evaluated f it x was different from last time. This would solve it right?

Yes. A key advantage is that it would remove the need for passing function values into line searches and other things.

[ChrisRackauckas]

I would like to add:

• Less strict typing of inputs.

For the native Julia methods, there's no reason to restrict to Array{T,N}. AbstractArrays, which by definition should have a linear index, seem like they would work fine in the methods that I checked. You wouldn't be able to do a type-check until v0.6 since it would require triangular dispatch. But I would really like to start optimizing some crazy things like MultiScaleArrays, and it seems the only thing blocking it is the type-checks requiring it to be a traditional array. Note that the same change should make it compatible with things like SharedArrays, which would be an early step to parallelism.

[timholy]

Congrats!! 🎆

[alexandrebrilhante]

I will attempt to work on #272. I'll look at the C implementation in NLopt.