Integrate gaussian_newton and levenberg_marquardt from optimtool.

[linjing-lab]

Current situation

The gaussian_newton and levenberg_marquardt are two algorithms for calculation of multiple residual function groups, like underlined code,

import optimtool.unconstrain as ou
from optimtool.base import sp, np
x = sp.symbols("x1:5")
res1 = 0.5*x[0] + 0.2*x[1] + 1.5*x[0]**3 - 2*x[2] # Damped oscillator with nonlinear term
res2 = 3*x[0] + x[1]**2 - x[2]**2 - 0.1*np.random.normal() # Coupled oscillator system
res3 = x[2]*(1 - x[3]) - x[0]*x[1] + 1.5*sp.sin(x[3]) # Predator-prey like interaction
res4 = x[3]*(x[0] - x[1]) + 0.5*sp.exp(-x[2]) - 2.0*x[1] # Delay differential equation component
ou.nonlinear_least_square.levenberg_marquardt([res1, res2, res3, res4], x, (1.0, 0.5, 0.2, 0.8), verbose=True, epsilon=1e-3)

Desired Situation

I think optimtool can provide many hybrid algorithms for optimagic, that nonlinear least squares algorithms can also serve as the optional solutions for optimagic ecology.

Proposed implementation

My implementation of optimization is in optimtool, barzilar_borwein and BFGS algorithms are two portable cases of them, which will provide more optimizers for optimagic.

[janosg]

Hi @linjing-lab thanks for your feature request. I was not aware of optimtool but I am always happy to add more algorithms to optimagic.

Would you like to create a Pull request that adds the suggested optimizers to optimagic?

You can follow this guide and tag me if you have any questions.

I think we should start with levenberg_marquardt but I am also open to adding more optimizers.