Repository navigation
Different solutions on different systems #3972
Description
Since the removal of the high tolerance in #3964, we have been noticing intermittent failures of the regression test in the CI on main. This is surprising because, on main, the regression tests for a commit abcd will use the assets just generated for that same commit abcd--there should be no difference.
I have explored whether this was non-determinism in the solver by changing to use the Clarabel quadratic programming problem solver #3971 however the problems persisted. That is not to say this was a bad decision and should be reversed, Clarabel is what we were recommended to use some years ago by the OSQP developers.
I have then checked whether the data going into and out of the QPP solve is exactly the same. On my Mac, the arrays going into and out of the QPP in the LT-NOF case are exactly the same at all iterations, down to the last decimal place. However, when I compare my Mac and Linux VM, there are slight very small differences (attached below). This difference is present in the data coming from PROCESS on the first iteration and therefore does not result from VMCON/QPP solver. As the solver runs, this error accumulates by a couple of orders of magnitude.
To verify that this is not new, I have gone back to a version of PROCESS before this change (58b2c96), downversioned to PyVMCON 2.3.1 and CVXPY 1.6. This is what the CI would have run using when it reliably passed. I then ran this test again between my Mac and Linux VM. The results show that there is indeed a difference between my Mac and Linux VM from the first iteration for the data coming out of PROCESS. The MAEs for the PROCESS data (deq, ie, die) on the first iteration are the same as the previous experiment. Unlikely in the previous experiments, the differences remain around the same order of magnitude and remains below the reporting threshold for PROCESS' test suite, even when running with 0% tolerance. Indeed in our test suite we do not use an exact 0 tolerance for this reason
PROCESS/tests/regression/test_process_input_files.py
Lines 192 to 202 in 8dd9e07
It is my opinion that reducing the QPP solver tolerance has just made apparent an issue which has existed in PROCESS all along. There are slight discrepancies between our results on different systems, but a lower QPP tolerance causes these to accumulate over a PROCESS run and results in very slightly different solver paths and hence solutions.
Differences in solve_qsp data between Mac and Linux VM for a LT-NOF PROCESS run using Clarabel QPP solver
deq@0 is different: MAE 1.6653345369377348e-13
ie@0 is different: MAE 9.472422846101836e-13
die@0 is different: MAE 4.274358644806853e-12
delta@0 is different: MAE 1.6758816556716738e-13
lamda_equality@0 is different: MAE 2.0961010704922955e-13
lamda_inequality@0 is different: MAE 6.326015812316642e-14
x@1 is different: MAE 1.674216321134736e-13
B@1 is different: MAE 2.8350655156827997e-12
f@1 is different: MAE 2.1316282072803006e-14
df@1 is different: MAE 4.196643033083092e-14
eq@1 is different: MAE 2.7755575615628914e-14
deq@1 is different: MAE 4.313216450668733e-13
ie@1 is different: MAE 6.543654507140673e-13
die@1 is different: MAE 2.3058943643405883e-11
delta@1 is different: MAE 2.0623919239071142e-12
lamda_equality@1 is different: MAE 7.405187574249794e-13
lamda_inequality@1 is different: MAE 8.139044993527023e-13
x@2 is different: MAE 2.0881074647149944e-12
B@2 is different: MAE 3.877786980410747e-11
f@2 is different: MAE 4.707345624410664e-14
df@2 is different: MAE 4.196643033083092e-14
eq@2 is different: MAE 6.061817714453355e-14
deq@2 is different: MAE 5.402345237826012e-13
ie@2 is different: MAE 6.633915639042698e-11
die@2 is different: MAE 1.3681287214240001e-09
delta@2 is different: MAE 1.4854523860963198e-12
lamda_equality@2 is different: MAE 1.201594379551807e-12
lamda_inequality@2 is different: MAE 6.062511603843745e-13
x@3 is different: MAE 8.271161533457416e-13
B@3 is different: MAE 5.5859317171780276e-11
f@3 is different: MAE 2.107203300738547e-13
df@3 is different: MAE 2.1538326677728037e-14
eq@3 is different: MAE 2.90878432451791e-14
deq@3 is different: MAE 3.643751966819764e-13
ie@3 is different: MAE 1.4147460980495907e-11
die@3 is different: MAE 2.496385320682748e-10
delta@3 is different: MAE 1.1022294188478554e-12
lamda_equality@3 is different: MAE 1.049249576112743e-11
lamda_inequality@3 is different: MAE 2.063987869505013e-12
x@4 is different: MAE 5.626610288800293e-13
B@4 is different: MAE 5.4939164328970946e-11
eq@4 is different: MAE 5.240252676230739e-14
deq@4 is different: MAE 6.319389456166391e-13
ie@4 is different: MAE 9.53481738008577e-12
die@4 is different: MAE 1.2168399621259596e-10
delta@4 is different: MAE 4.3482535280747747e-13
lamda_equality@4 is different: MAE 3.824274230623814e-12
lamda_inequality@4 is different: MAE 2.698188894534326e-13
x@5 is different: MAE 4.0101255649460654e-13
B@5 is different: MAE 6.160050247672189e-11
eq@5 is different: MAE 3.774758283725532e-15
deq@5 is different: MAE 2.7400304247748863e-13
ie@5 is different: MAE 3.019806626980426e-13
die@5 is different: MAE 3.26405569239796e-11
delta@5 is different: MAE 3.207095266032122e-10
lamda_equality@5 is different: MAE 9.877559855112072e-10
lamda_inequality@5 is different: MAE 3.334670087805358e-10
Differences in solve_qsp data between Mac and Linux VM for a LT-NOF PROCESS run using OSQP QPP solver (very high relative tolerance), VMCON 2.3.1, CVXPY 1.6, and an older version of PROCESS
deq@0 is different: MAE 1.6653345369377348e-13
ie@0 is different: MAE 3.1086244689504383e-15
die@0 is different: MAE 4.274358644806853e-12
delta@0 is different: MAE 1.8360313269738526e-13
lamda_equality@0 is different: MAE 1.9129142714291447e-13
lamda_inequality@0 is different: MAE 8.171241461241152e-14
x@1 is different: MAE 1.836308882730009e-13
B@1 is different: MAE 4.397149311330395e-12
f@1 is different: MAE 1.7985612998927536e-14
df@1 is different: MAE 1.0502709812953981e-13
eq@1 is different: MAE 2.4646951146678475e-14
deq@1 is different: MAE 5.344613640545504e-13
ie@1 is different: MAE 3.232969447708456e-13
die@1 is different: MAE 2.0210277895671425e-11
delta@1 is different: MAE 3.3911068397785016e-13
lamda_equality@1 is different: MAE 2.0672352718520415e-13
lamda_inequality@1 is different: MAE 8.836265052991621e-13
x@2 is different: MAE 3.6648462042876417e-13
B@2 is different: MAE 7.680611702198803e-11
f@2 is different: MAE 1.48991929904696e-13
df@2 is different: MAE 4.196643033083092e-14
eq@2 is different: MAE 6.261657858885883e-14
deq@2 is different: MAE 3.02174951727352e-13
ie@2 is different: MAE 2.128741627416275e-12
die@2 is different: MAE 3.342037757647631e-11
delta@2 is different: MAE 3.720634911275056e-13
lamda_equality@2 is different: MAE 1.6369128275073308e-12
lamda_inequality@2 is different: MAE 5.029865413064272e-13
x@3 is different: MAE 4.984901380566953e-13
B@3 is different: MAE 7.907985377642035e-11
f@3 is different: MAE 5.10702591327572e-14
eq@3 is different: MAE 7.815970093361102e-14
deq@3 is different: MAE 4.289901767151605e-13
ie@3 is different: MAE 7.254197242900773e-13
die@3 is different: MAE 2.4063417924935493e-11
delta@3 is different: MAE 4.373723605510804e-13
lamda_equality@3 is different: MAE 4.987676938128516e-12
lamda_inequality@3 is different: MAE 8.842232501748981e-13
x@4 is different: MAE 6.967759702547482e-13
B@4 is different: MAE 7.737455121059611e-11
eq@4 is different: MAE 2.4646951146678475e-14
deq@4 is different: MAE 8.699707620962727e-13
ie@4 is different: MAE 1.535327420754129e-12
die@4 is different: MAE 3.527134140313137e-11
delta@4 is different: MAE 1.2453059416994705e-13
lamda_equality@4 is different: MAE 1.3394840792102514e-13
lamda_inequality@4 is different: MAE 3.3113052230748785e-14
x@5 is different: MAE 6.7390537594747e-13
B@5 is different: MAE 6.901146321069973e-11
eq@5 is different: MAE 6.661338147750939e-16
deq@5 is different: MAE 5.551115123125783e-13
ie@5 is different: MAE 6.225020499073253e-13
die@5 is different: MAE 1.7564616427989677e-11
delta@5 is different: MAE 2.93050306243714e-13
lamda_equality@5 is different: MAE 5.5297454983060934e-14
lamda_inequality@5 is different: MAE 3.992444395917172e-14
x@6 is different: MAE 7.194245199571014e-13
B@6 is different: MAE 5.74296166178101e-11
f@6 is different: MAE 4.440892098500626e-16
eq@6 is different: MAE 6.661338147750939e-16
deq@6 is different: MAE 4.969358258222201e-13
ie@6 is different: MAE 8.643086246706844e-13
die@6 is different: MAE 2.0371260234242072e-11
delta@6 is different: MAE 1.6653692314072543e-13
lamda_equality@6 is different: MAE 4.535130951333066e-14
lamda_inequality@6 is different: MAE 3.3518327002823867e-14
Activity
timothy-nunn commentedon Nov 6, 2025
I do not believe it is good software practice to re-lower the solver tolerance just so that our test suite can pass.
Now that we know these differences are a product of the system PROCESS runs on, I believe we should increase the test tolerance such that these differences are not flagged. This will provide us time to discuss if there is a possible solution to these accumulated differences--there may not be.
timothy-nunn commentedon Nov 12, 2025
@jonmaddock sorry I forgot to tag you into this. If you are satisfied with my reasoning above (i.e. you do not have a concern that there is a bug present in PROCESS causing these failures) and are happy with #3973, please go ahead and close this issue.
timothy-nunn commentedon Jan 7, 2026
@je-cook I spoke with Jon about this and given my investigation above I think we were both happy. Its no secret that PROCESS (and every other numeric code) is slightly different on different systems. We are not getting failures anymore because GitHub has finished their Linux runner upgrades so they are all using the same software; now the regression assets are created on the exact same system that runs the tests. Are you happy for this issue to be closed?