Data formatting error Exception occurs when using this solver from CVXPY

[AtsushiSakai]

When using CLARABEL instead of ECOS as the solver for this rocket landing simulation created with CVXPY, the following error occurs.
I'm not sure this is Clarabel issue or CVXPY issue, but I would like to know workaround for it. Thanks!!

['CLARABEL', 'ECOS', 'ECOS_BB', 'OSQP', 'SCIPY', 'SCS']
start!!
------------------ Iteration 01 ------------------
delta_norm 55.857422448036154
sigma_norm 9.900000001863756
nu_norm 8.91953005382578e-15
Time for iteration 0.27562594413757324 s
------------------ Iteration 02 ------------------
Traceback (most recent call last):
File "/Users/atsushi_sakai/Library/CloudStorage/Dropbox/Program/python/PythonRobotics/AerialNavigation/rocket_powered_landing/rocket_powered_landing.py", line 674, in
main()
~~~~^^
File "/Users/atsushi_sakai/Library/CloudStorage/Dropbox/Program/python/PythonRobotics/AerialNavigation/rocket_powered_landing/rocket_powered_landing.py", line 642, in main
problem.solve()
~~~~~~~~~~~~~^^
File "/Users/atsushi_sakai/Library/CloudStorage/Dropbox/Program/python/PythonRobotics/AerialNavigation/rocket_powered_landing/rocket_powered_landing.py", line 539, in solve
self.prob.solve(verbose=verbose_solver,
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
solver=solver)
^^^^^^^^^^^^^^
File "/Users/atsushi_sakai/Library/CloudStorage/Dropbox/Program/python/PythonRobotics/venv/lib/python3.13/site-packages/cvxpy/problems/problem.py", line 600, in solve
return solve_func(self, *args, **kwargs)
File "/Users/atsushi_sakai/Library/CloudStorage/Dropbox/Program/python/PythonRobotics/venv/lib/python3.13/site-packages/cvxpy/problems/problem.py", line 1183, in _solve
solution = solving_chain.solve_via_data(
self, data, warm_start, solver_verbose, kwargs)
File "/Users/atsushi_sakai/Library/CloudStorage/Dropbox/Program/python/PythonRobotics/venv/lib/python3.13/site-packages/cvxpy/reductions/solvers/solving_chain.py", line 475, in solve_via_data
return self.solver.solve_via_data(data, warm_start, verbose,
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
solver_opts, problem._solver_cache)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/atsushi_sakai/Library/CloudStorage/Dropbox/Program/python/PythonRobotics/venv/lib/python3.13/site-packages/cvxpy/reductions/solvers/conic_solvers/clarabel_conif.py", line 346, in solve_via_data
solver = updated_solver()
File "/Users/atsushi_sakai/Library/CloudStorage/Dropbox/Program/python/PythonRobotics/venv/lib/python3.13/site-packages/cvxpy/reductions/solvers/conic_solvers/clarabel_conif.py", line 342, in updated_solver
_solver.update(P=P, q=q, A=A, b=b, settings=newsettings)
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: Data formatting error

Version Info:

• Python: 3.13
• CVXPY: 1.6.5
• CLARABEL: 0.10.0

[goulart-paul]

I think there was a similar issue reported previously in cvxpy that was caused by an internal formatting problem with the P matrix. Are you using the most recent cvxpy version?

[AtsushiSakai]

Thank you for your comment.

Are you using the most recent cvxpy version?

Yes. I'm using the latest one (CVXPY 1.6.5)

Do you think this issue should be reported in the CVXPY repository?

[goulart-paul]

Perhaps yes. The issue appears to be that the solver is receiving badly formatted data (likely one of A or P). That's either because it is the wrong size or because the sparsity of the update matrix doesn't match what the solver was initialized with.

If you have a minimal example that reproduces it that would be very helpful. Also fine just to report it here through since I am generally the one maintaining the solver wrapper in any case.

[AtsushiSakai]

Thank you!!.

Perhaps yes.

I created an issue ticket in the CVXPY repo
cvxpy/cvxpy#2800

If you have a minimal example that reproduces it that would be very helpful.

This is how to reproduce this issue.

1. Install CVXPY, SciPy, Matplotlib
2. Run this script
   https://github.com/AtsushiSakai/PythonRobotics/blob/b6f0b02aef7739ed2bf8bbf3c3cc43fad41c0747/AerialNavigation/rocket_powered_landing/rocket_powered_landing.py

Also, this CI also reproduced this issue: https://github.com/AtsushiSakai/PythonRobotics/actions/runs/14785601975/job/41513352656?pr=1209#step:6:256

[Transurgeon]

Perhaps yes. The issue appears to be that the solver is receiving badly formatted data (likely one of A or P). That's either because it is the wrong size or because the sparsity of the update matrix doesn't match what the solver was initialized with.

I just debugged a bit this problem instance and it seems like the data is in the right format.

A.shape == (5955, 3377)
P.shape == (3377, 3377)
b.shape == (5955,)
q.shape == (3377,)
# presumably x.shape == (3377,)

Also, I noticed that P actually has no non-zero entries (meaning no quadratic objective). This leaves the only possibility that the sparse structure of the A matrix has changed.
@AtsushiSakai I didn't look much into the problem instance you posted (it looks quite complicated!), but is there anything that could have caused this sparsity change in subsequent iterations?

[goulart-paul]

Putting some printfs in the data updating code that is eventually reached within rust, and I can see that the solver is halting specifically during the update of the A matrix. When the solver compares the update values for A vs the ones originally used to construct the solver instance, the data passed for the update has about 400 more nonzeros than the one being carried internally. About 100 of those are in the first column.

[AtsushiSakai]

Thank you both!!
I’m not sure, but the sparsity pattern of the matrix A may vary across iterations to solve control problem at each time step. The matrix is regenerated based on runtime data.

I’m not an expert of numerical solvers.
Do you mean that CLARABEL does not support this kind of problem?

[goulart-paul]

Correct. The solver will only allow an update if the sparsity pattern does not change.

There are two options for you here:

1. If you know the pattern of all entries in A that could conceivably be nonzero, then you can initialize the solver with problem data where the sparsity pattern reflects that pattern. In that case updating the values will work. It is fine if values themselves are assigned the numerical value 0.0 at any point.

2. Just make a new solver object whenever the problem data changes. There is little advantage to trying to update the solver with problem data that has a different sparsity pattern than what it was initialized with anyway. Most of the reason why updating is faster is that certain setup steps, particularly analysis of the problem sparsity pattern and allocation of memory based on the expected sparsity of matrix factorisations, can be skipped when updating. If you change the sparsity pattern then none of that works.