Move optional sage optimization backends (COIN, CPLEX, Gurobi) to separate Cython packages to remove OptionalExtension problems

[mkoeppe]

sage.numerical.backends depends on very little from Sage. We propose to split out the optional backends (COIN, CPLEX, Gurobi) to separate Cython packages.

With this ticket, sagelib no longer has the optional dependency on cbc. Instead, these new packages depend on the sage package (i.e., sagelib).

This makes it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this will also simplify packaging binary distributions.

The new packages are on PyPI:

wget -P upstream https://files.pythonhosted.org/packages/29/8a/65fd90a890fcfb56166c4071aa7a4fb5b865d313304f60474870f0e89998/sage_numerical_backends_cplex-9.0b12.tar.gz
wget -P upstream https://files.pythonhosted.org/packages/2c/5c/e1532bb6cde28cf86ebffd408ccb500d7b846bc29bbd1c5e479a2f18acb6/sage_numerical_backends_gurobi-9.0b12.tar.gz
wget -P upstream https://files.pythonhosted.org/packages/17/f4/308d3e151d82daf2c58ca26bce157fac31f763b2b60b5173243c09291e0a/sage_numerical_backends_coin-9.0b12.tar.gz

Development here:

• https://github.com/mkoeppe/sage-numerical-backends-cplex
• https://github.com/mkoeppe/sage-numerical-backends-coin
• https://github.com/mkoeppe/sage-numerical-backends-gurobi

See also:

• Follow-up: Move sage optimization backend framework (sage.numerical.backends) to separate Cython packages #28920: Move sage optimization backend framework (sage.numerical.backends) to separate Cython packages
• Meta-ticket: Use Python optimization interfaces: CVXPY, SCIP, or-tools, PuLP, Pyomo, cylp... #26511 - Meta-ticket: Use Python optimization interfaces: PuLP, Pyomo, cylp...
• https://groups.google.com/forum/#!topic/sage-packaging/_FLDkqD2X8c

CC: @isuruf @saraedum @kiwifb @dimpase @sagetrac-tmonteil @videlec @seblabbe

Component: numerical

Author: Matthias Koeppe

Branch: 1279897

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/28175

[dcoudert]

comment:1

Can you be more precise on your plans.

These backends are used in graphs. So what will happen if linear programming backends are in a separate package ?

[mkoeppe]

comment:2

Replying to @dcoudert:

These backends are used in graphs. So what will happen if linear programming backends are in a separate package ?

In my plan, the backends would be a standard package of the sage distribution, and the cython modules it provides would be automatically imported when MixedIntegerLinearProgram is used.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,9 +1,37 @@
-`sage.numerical.backends` depends on very little from Sage and should become a separate Cython package. This would make it easier to reconfigure it when new solvers are installed and eliminate some optional extensions from sagelib.
+`sage.numerical.backends` depends on very little from Sage and should become a separate Cython package. This would make it easier to reconfigure it when new solvers are installed and eliminate some optional extensions from sagelib, which would simplify packaging binary distributions.
+
+From `src/module_list.py`:
+
+```
+    OptionalExtension("sage.numerical.backends.gurobi_backend",
+              ["sage/numerical/backends/gurobi_backend.pyx"],
+              libraries = ["gurobi"],
+              condition = os.path.isfile(SAGE_INC + "/gurobi_c.h") and
+                  os.path.isfile(SAGE_LOCAL + "/lib/libgurobi.so")),
+
+    OptionalExtension("sage.numerical.backends.cplex_backend",
+              ["sage/numerical/backends/cplex_backend.pyx"],
+              libraries = ["cplex"],
+              condition = os.path.isfile(SAGE_INC + "/cplex.h") and
+                  os.path.isfile(SAGE_LOCAL + "/lib/libcplex.a")),
+
+    OptionalExtension("sage.numerical.backends.coin_backend",
+              ["sage/numerical/backends/coin_backend.pyx"],
+              language = 'c++',
+              libraries = ["Cbc", "CbcSolver", "Cgl", "Clp", "CoinUtils",
+                           "OsiCbc", "OsiClp", "Osi"] + lapack_libs,
+              library_dirs = lapack_library_dirs,
+              include_dirs = lapack_include_dirs,
+              package = 'cbc'),
+```
+
 
 Details:
 - `get_solver` uses base_ring
 - GLPKBackend depends on `sage.libs.glpk`, which should also become a separate Cython package (see https://en.wikibooks.org/wiki/GLPK/Python for a list of other glpk bindings)
-
- 
+- For Gurobi and CPLEX, we could alternatively get rid of these Cython modules entirely by making use of their standard Python API instead.
 
 
+
+
+

[kiwifb]

comment:5

For glpk it looks like https://github.com/biosustain/swiglpk/releases is the best maintained. Excluding sage and cvxopt.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -31,6 +31,8 @@
 - GLPKBackend depends on `sage.libs.glpk`, which should also become a separate Cython package (see https://en.wikibooks.org/wiki/GLPK/Python for a list of other glpk bindings)
 - For Gurobi and CPLEX, we could alternatively get rid of these Cython modules entirely by making use of their standard Python API instead.
 
+See also:
+- #26511 - Meta-ticket: Use Python optimization interfaces: PuLP, Pyomo, cylp...
 
 
 

[mkoeppe]

Description changed:

--- 
+++ 
@@ -33,7 +33,7 @@
 
 See also:
 - #26511 - Meta-ticket: Use Python optimization interfaces: PuLP, Pyomo, cylp...
+- https://groups.google.com/forum/#!topic/sage-packaging/_FLDkqD2X8c 
 
 
 
-

[mkoeppe]

comment:8

I have prepared a first module, https://github.com/mkoeppe/sage-numerical-backends-coin.
It depends on Sage (as it inherits from sage.numerical.backends.generic_backend.GenericBackend).

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,4 +1,4 @@
-`sage.numerical.backends` depends on very little from Sage and should become a separate Cython package. This would make it easier to reconfigure it when new solvers are installed and eliminate some optional extensions from sagelib, which would simplify packaging binary distributions.
+`sage.numerical.backends` depends on very little from Sage. We propose to split out the optional backends to separate Cython packages. This would make it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this  would also simplify packaging binary distributions.
 
 From `src/module_list.py`:
 

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,35 +1,24 @@
-`sage.numerical.backends` depends on very little from Sage. We propose to split out the optional backends to separate Cython packages. This would make it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this  would also simplify packaging binary distributions.
+`sage.numerical.backends` depends on very little from Sage. We propose to split out the optional backends (COIN, CPLEX, Gurobi) to separate Cython packages. This would make it easier to reconfigure when new solvers are installed, as sagelib does not have to be recompiled. By eliminating the use of the optional extensions mechanisms from sagelib, this will also simplify packaging binary distributions.
 
-From `src/module_list.py`:
+The new packages are on PyPI:
 
 ```
-    OptionalExtension("sage.numerical.backends.gurobi_backend",
-              ["sage/numerical/backends/gurobi_backend.pyx"],
-              libraries = ["gurobi"],
-              condition = os.path.isfile(SAGE_INC + "/gurobi_c.h") and
-                  os.path.isfile(SAGE_LOCAL + "/lib/libgurobi.so")),
+wget -P upstream https://files.pythonhosted.org/packages/2a/1f/4fc55a0013bc5050ecf253b8bce1c4548380c15d57310a1fae5841405984/sage_numerical_backends_cplex-9.0b9.tar.gz
+wget -P upstream https://files.pythonhosted.org/packages/72/07/8b52ab9411caec6f4593decebc5660c6ce955f437510d8bd58533866ff31/sage_numerical_backends_gurobi-9.0b9.tar.gz
+wget -P upstream https://files.pythonhosted.org/packages/40/01/0b9580bc9db5671cc2446ea089ba1f28bacf85ef0e9f05a4a781bf9dcd7d/sage_numerical_backends_coin-9.0b9.tar.gz
+``` 
 
-    OptionalExtension("sage.numerical.backends.cplex_backend",
-              ["sage/numerical/backends/cplex_backend.pyx"],
-              libraries = ["cplex"],
-              condition = os.path.isfile(SAGE_INC + "/cplex.h") and
-                  os.path.isfile(SAGE_LOCAL + "/lib/libcplex.a")),
+Development here:
+- https://github.com/mkoeppe/sage-numerical-backends-cplex
+- https://github.com/mkoeppe/sage-numerical-backends-coin
+- https://github.com/mkoeppe/sage-numerical-backends-gurobi
 
-    OptionalExtension("sage.numerical.backends.coin_backend",
-              ["sage/numerical/backends/coin_backend.pyx"],
-              language = 'c++',
-              libraries = ["Cbc", "CbcSolver", "Cgl", "Clp", "CoinUtils",
-                           "OsiCbc", "OsiClp", "Osi"] + lapack_libs,
-              library_dirs = lapack_library_dirs,
-              include_dirs = lapack_include_dirs,
-              package = 'cbc'),
-```
+---
 
-
-Details:
+(In a follow-up ticket, one could consider moving the entire backend machinery to a separate packages. Some notes regarding this:
 - `get_solver` uses base_ring
-- GLPKBackend depends on `sage.libs.glpk`, which should also become a separate Cython package (see https://en.wikibooks.org/wiki/GLPK/Python for a list of other glpk bindings)
-- For Gurobi and CPLEX, we could alternatively get rid of these Cython modules entirely by making use of their standard Python API instead.
+- GLPKBackend depends on `sage.libs.glpk`, which should also become a separate Cython package (see https://en.wikibooks.org/wiki/GLPK/Python for a list of other glpk bindings).)
+(In another follow-up ticket, for Gurobi and CPLEX, we could alternatively get rid of these Cython modules entirely by making use of their standard Python API instead.)
 
 See also:
 - #26511 - Meta-ticket: Use Python optimization interfaces: PuLP, Pyomo, cylp...

[dimpase]

comment:10

Is there a git branch to try, replacing Sage's current COIN-OR backend with this one?

[mkoeppe]

comment:11

Not yet (working on it right now), but the 3 backends already pass their test suites separately.

[mkoeppe]

Branch: u/mkoeppe/move_optional_sage_optimization_backends_to_separate_cython_packages_to_remove_optionalextension_problems

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

437fec5

Fixup dependencies

[sagetrac-git]

Commit: 437fec5

[mkoeppe]

comment:14

Now the packages can be installed using sage -i -c... please test