Pre-release: beta version of the benchmark

[stephane-caron]

This is the last pre-release of qpsolvers_benchmark, a benchmark for quadratic programming (QP) solvers available in Python. The main upgrade since alpha is that the benchmark now checks both residuals and the duality gap, i.e., full optimality conditions.

The goal of this pre-release is to request feedback from QP solver developers before the initial release. The initial release will in no way be the end: it will be advertised around, but the benchmark is designed to be iteratively refined by the community in the long run.

You can already help at this stage! Here are some suggestions:

• Take a look at the README and current reports produced for the Maros-Meszaros and Maros-Meszaros dense subset test sets.
• Look at the benchmark methodology and join the discussions.
• Submit a new problem to the GitHub free-for-all, for instance one that reflects the applications you are working on.

Added

• Check dual residual
• Check duality gap
• Document all benchmark functions
• Main script: new hist plot command
• ProblemNotFound exception
• Results by settings in reports
• Write benchmark version in reports

Changed

• Benchmark script takes test set as first argument
• Maros-Meszaros: empty equality constraints are now set to None
• ProxQP: re-run benchmark with ProxQP 0.3.2
• Refactor Report class and run function
• Report encoding is now UTF-8
• Switch to qpsolvers v2.7
• Test set descriptions are now mandatory

Fixed

• Conform to linter standards
• Sparse matrix conversion

---

This discussion was created from the release v0.1.0-beta.

[stephane-caron]

Dear open-source solver developers and maintainers:

• CVXOPT: @martinandersen
• ECOS: @adomahidi @SteveDiamond
• HiGHS: @feldmeier
• OSQP: @bstellato @gbanjac
• ProxQP: @Bambade @jcarpent
• qpOASES: @jferreau @apotschka
• qpSWIFT: @abhishek-pandala
• quadprog: @preeyan @leonwehrhan (@rmcgibbo FYI)
• SCS: @bodono

This is a request for comments 🙂 If you were mentioned, this benchmark compares your solver with the others.

• Methodology is described in the README and discussions.
• Results are compiled in reports for the Maros-Meszaros and Maros-Meszaros dense subset test sets.

The benchmark is open to feedback and contributions. The goal is to help users compare and select the best solver(s) for their respective problems. If you believe the performance of your solver is not accurately represented one of the reports, open an issue and we will look at the data, reproducing the benchmark on your machine, or investigate the code.

NB: there are already some known limitations. Work is planned for those that are already assigned (they will be done before the initial release).