[WIP] Feat: Add COBYLA optimizer

[yashnator]

Motivation

COBYLA (Constrained Optimisation BY Linear Approximations) is an iterative, derivative-free algorithm for solving constrained optimisation problems. It works by constructing and updating linear approximations of both the objective function and the constraints. These approximations are formed using function values evaluated at the vertices of a carefully structured simplex. At each iteration, the algorithm solves a linear programming subproblem within a trust region. As the method approaches a constrained optimum, the size of the trust region is gradually reduced to refine the solution.

• Type of optimisation: local
• COBYLA can be computationally expensive for higher-dimensional problems, noisy functions and non-linear constraints.

Reference:

Powell, M.J.D. (1994). A Direct Search Optimization Method That Models the Objective and Constraint Functions by Linear Interpolation. In: Gomez, S., Hennart, JP. (eds) Advances in Optimization and Numerical Analysis. Mathematics and Its Applications, vol 275. Springer, Dordrecht.

Source: https://doi.org/10.1007/978-94-015-8330-5_4

Description of the changes

• Add and COBYLA class to local_opt

• Add example to examples/optimization_algorithm

Fixes #73