-
Notifications
You must be signed in to change notification settings - Fork 767
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Incremental Smoothing with Plane Factor occasionally fails #561
Comments
This is a great example! Before o look at it in detail: does the optimization work if you just do gauss-Newton optimization rather than iSAM? |
Yes, exact same behavior with Gauss-Newton. Here is the listing:
|
Marco, I added your test in #564 |
Here is the output of the second listing (Gauss-Newton) with the offending line through the command: Output:
Without the offending line, the error count goes to 1:
Note that I wasn't able to reproduce the success by commenting out both the plane priors. I will try later on with the test you have made. This was done on my small main but it shouldn't make much difference. |
Hmmm. I'm not an expert on debugging memory errors but don't see crazy things here at first glance. Still, the fact that you can't get error by commenting out prior again points to non-determinism. I'll take another look after teaching but please pore over the code as well? My bet is on some subtle bug in OrientedPlane3 or OrientedPlane3Factor, and non-determinism points to either uninitialized memory or worse, overwriting some nearby memory. I did use Debug, however, and I think that adds bounds checking in Eigen - but nothing triggered. |
TBB is famous for generating false positives with Valgrind. Could you try reproducing without TBB? That would make the stacktrace a bit cleaner. |
@ProfFan you are right. I've recompiled gtsam without TBB and run the test @dellaert made. The stack is clear of errors:
If I recompile with TBB, then I get 3 errors (which I spare you). |
Eigen's Debug mode run time bound checks are not helpful here because the code path is completely different under Release. I would suggest an uninitialized variable (which is the nastiest since both valgrind and ASAN cannot detect). I have work on my hand so I can't investigate so this is a pointer for anyone who's working on this. |
It fails under Debug, too, though... Maybe that's a hint? |
Running the new test in a debug build (u18.04, g++7) with valgrind gives no suspicious memory access. Neither as is in git, nor uncommenting the p2_prior factor. In that latter case the test failed, of course, but it looks more like a logic or math problem than an uninitialized/corrupted memory issue. |
The augmented information matrix on p1-x0 has large (but not absurdly large) negative eigenvalues, btw...
|
Hmmm. Marco, I propose you print out the singular values of the entire information matrix: it should be positive definite. If not, we should figure out why. Don’t use the augmented part if you use the method @jlblancoc refers to, just the top-left block, omitting the last row and column. |
Hi Frank, Thanks again for the feedback. I added the following code to your example to calculate the singular values (I didn't have permission to push to the branch):
I also added the Please let me know if this isn't the best way to calculate this an I can run it again. Without the priors on the planes the output is:
With a prior only on p1:
With a prior on both p1 and p2:
|
OK, so this is interesting, and rather poorly conditioned (both very
high and low singular values). Can you hypothesize why there is such a
discrepancy ? Why are the lowest directions so poorly constrained? What
variables do they correspond to?
On October 18, 2020, GitHub ***@***.***> wrote:
Hi Frank,
Thanks again for the feedback.
I added the following code to your example to calculate the singular
values (I didn't have permission to push to the branch):
Matrix info = graph.linearizeToHessianFactor(initialEstimate)-
>information(); Eigen::JacobiSVD<Matrix> svd(info,
Eigen::ComputeThinU); std::cout << "Singular Values:\n" <<
svd.singularValues().transpose() << std::endl; std::cout << "Matrix
U:\n" << svd.matrixU() << std::endl;
I also added the GaussNewtonParams to the optimizer as you mentioned
here
<#564 (comment)>.
Please let me know if this isn't the best way to calculate this an I
can run it again.
Without the priors on the planes the output is:
errorThreshold: 1.34864e-26 <? 0 absoluteDecrease: -6.93331028744e-27
<? 1e-05 relativeDecrease: -1.05802556953 <? 1e-05 iterations: 2 >?
100 Singular Values: 7227856.57593 3900046.66513 11554.603727
11447.8952301 10004.1893614 10000 9999.992414 9997.42007424
876.885178384 463.629436563 0.117115818303 0.0584303566143 Matrix U:
3.05749155202e-08 0.000203902807992 0.043944479922 -0.00133561731752
0.00327181364483 -8.13856900573e-14 -1.86582853059e-05 -
3.16663230232e-06 0.106971160401 0.993284210451 1.48677042265e-05 -
1.61163266599e-06 0.000150569143325 0.999874039535 -1.04508237136e-05
7.61870498253e-06 1.96616620193e-05 -5.28274821756e-13 -
0.000639944543154 -0.0112168372323 -2.19274190229e-05 -
0.000202567354402 0.00101435534489 -0.0111617400842 1.68764240573e-06
0.0112070304694 -1.54012737984e-05 -0.000478990846787 2.14200557276e-
05 -2.35713441824e-15 -6.83693024657e-06 -0.000125797549625 -
1.13078314566e-05 1.85379398803e-06 -0.0904992302243 0.995833342313
0.998750564567 -0.000279062238723 -0.00425225577754 6.80738691313e-05
0.000173737506859 5.12382147205e-13 0.000621406264083 -0.0115045141176
0.0468019236689 -0.00485248849205 -0.0114597193946 -0.00104093244 -
0.0472386502513 1.31983933644e-05 -0.0899197346123 0.00123405396733
0.00322719142037 7.02600019382e-14 -4.84575065675e-05 0.00054336790131
0.9895172693 -0.102596268418 0.000540122053905 5.96455652937e-05
0.0115189560064 -3.21851829849e-06 -1.36086058062e-05 -
0.000964913738444 -2.21415804757e-05 5.60199194881e-15 6.83580815522e-
06 -0.00013260883655 -5.94154734593e-07 -1.53174371527e-05
0.995829503712 0.0904984175442 7.7307419102e-05 7.43572915294e-05
0.813574265572 0.56837703658 0.0920197314187 -1.66783091651e-12 -
0.000641683245406 0.00681276961955 0.0684733214383 -0.0429068723044
0.000533004251426 0.000333304664153 0.000107986943198
0.000100831065696 -0.568317890614 0.820638180575 0.00663674256352
9.05842138274e-13 -0.000200109211003 0.00924311059846 -0.0494189235086
0.0315470359615 0.000746807763752 0.000453070406121 -5.16349440162e-06
-1.24301092223e-05 -0.0712522635495 -0.0580178575669 0.995706232728 -
4.60585274124e-12 -0.00573296628998 0.00104517090576 -0.00956546816723
0.000824554774567 -2.9987268781e-05 -5.30283529548e-05
0.00121014453295 0.000717594935947 -1.79246945802e-05 -
0.00104230353744 -0.00227821376876 0.925362353338 -0.369407708441
0.0850541215878 7.54385827748e-08 7.23604838035e-08 -1.15573562131e-06
-4.27268452632e-07 -0.00031795517522 0.000848714700754 -
5.85658021976e-06 -7.56605307881e-05 0.0053213985507 0.368867026621
0.929190529936 0.0226479865482 8.05147599797e-08 -2.73422722084e-08
4.01664135017e-07 -4.24897565619e-07 0.011468542568 0.0111775125185 -
0.000214457843987 -0.0113527533003 -0.00166198509197 -0.0874164269234
0.0104300936341 0.995922398508 1.13831276344e-06 6.50610106653e-07 -
1.05393644222e-05 -6.3158476992e-06
With a prior only on p1:
converged errorThreshold: 0.0630133707 <? 0 absoluteDecrease:
1.08871245352e-13 <? 1e-05 relativeDecrease: 1.72774832075e-12 <? 1e-
05 iterations: 3 >? 100 Singular Values: 7227856.57593 3900048.28587
11554.6068581 11447.895233 10004.1893787 10000 9999.99241466
9997.42027821 876.903800868 465.228801703 0.118107841678
0.097636936477 Matrix U: 3.0574930447e-08 0.000203902808013
0.0439508961498 -0.00133577499349 0.00327236228026 -9.17320066784e-14
-1.86613148709e-05 -3.1665143594e-06 0.107386723055 0.993239082816
1.48655755096e-05 1.07834713144e-06 0.000150569216665 0.999874039638 -
1.04521337561e-05 7.61875257792e-06 1.96621800382e-05 -8.80834660302e-
13 -0.000639995035471 -0.0112168298853 -2.2012160763e-05 -
0.000202558084258 0.00300614452676 -0.010797052687 1.68764285858e-06
0.0112070259271 -1.54008947176e-05 -0.00047899254574 2.14201111227e-05
-6.11941717528e-15 -6.83636036435e-06 -0.000125777584024 -
1.13077011445e-05 1.85637452258e-06 -0.268203721303 0.963296901437
0.998750564567 -0.000279062258366 -0.0042522533701 6.80700019074e-05
0.000173738994954 8.60528572836e-13 0.000621354215108 -0.0115045169203
0.0467998922282 -0.00487204416798 -0.0110853912383 -0.00308590464483 -
0.0472386502513 1.3198394294e-05 -0.0899196835801 0.00123397216971
0.00322720977533 4.31728603128e-14 -4.84550602443e-05
0.000543368296285 0.989474319248 -0.103009714795 0.000520576887842
0.000155857944773 0.0115189560064 -3.21851852506e-06 -1.36077564132e-
05 -0.000964913750218 -2.21415767839e-05 9.43369315503e-15
6.83520758479e-06 -0.000132608868809 -5.97663041291e-07 -
1.52665262073e-05 0.963293271661 0.268202231189 7.7307419105e-05
7.43572605427e-05 0.81357354761 0.568377768558 0.0920195920786 -
2.02654104557e-12 -0.000641649017657 0.00681277769308 0.0684547864806
-0.0429406494304 0.000464334894702 0.000423770267717 0.000107986943202
0.000100831023677 -0.568318455404 0.820637668631 0.00663684588997
7.92409850119e-13 -0.000200067785724 0.00924311193524 -0.0494052946495
0.0315714941146 0.000653096069585 0.000580050748984 -5.16349440211e-06
-1.24301040434e-05 -0.0712520636577 -0.0580179259469 0.995706243123 -
7.79520704277e-12 -0.00573293329634 0.00104525191259 -0.0095651346898
0.000828450376876 -1.99563658608e-05 -5.75586996912e-05
0.00121014453298 0.000717594636806 -1.79237957276e-05 -
0.00104230355817 -0.00227820796801 0.925362353133 -0.369407324212
0.085055792746 7.54717451094e-08 7.23541449287e-08 -1.05965350122e-06
-6.29471514346e-07 -0.000317955175187 0.000848714347192 -
5.85650136757e-06 -7.56605432021e-05 0.00532137086733 0.368867027137
0.929190632338 0.0226437829842 8.05026446805e-08 -2.73816196514e-08
4.71963816561e-07 -3.47137775699e-07 0.0114685425685 0.01117750786 -
0.000214447996293 -0.0113527535721 -0.00166204050134 -0.0874164269179
0.01043459941 0.995922351269 1.13861268671e-06 6.50376973562e-07 -
9.22562952533e-06 -8.12818531459e-06
With a prior on both p1 and p2:
converged errorThreshold: 0.136903693 <? 0 absoluteDecrease:
2.94042568072e-13 <? 1e-05 relativeDecrease: 2.1478059549e-12 <? 1e-05
iterations: 3 >? 100
/home/david/git/gtsam/gtsam/slam/tests/testOrientedPlane3Factor.cpp:240:
Failure: "expected 0 but was: 0.13690369346500605" Singular Values:
7227858.19665 3900048.28587 11554.6199972 11447.8952356 10004.1893956
10000 9999.99241529 9997.42049326 878.494607302 465.245975706
0.157325516372 0.0986290479126 Matrix U: 3.05749086567e-08
0.000203902808013 0.0439507914704 -0.00133569922174 0.00327238046304 -
8.84373407917e-14 -1.86613463037e-05 -3.16641055135e-06 0.106989278445
0.993281977841 1.48179572848e-05 -1.61249565849e-06 0.000150569109526
0.999874039638 -1.04521298542e-05 7.6187360439e-06 1.96628462572e-05 -
7.70381835845e-13 -0.000639944308141 -0.0112168327793 -2.19313596594e-
05 -0.000202566831135 0.00101416696134 -0.0111617524172 1.6876416577e-
06 0.0112070259271 -1.5400069078e-05 -0.000478992571968
2.14201153885e-05 -4.94257902279e-15 -6.83579111928e-06 -
0.000125777613802 -1.12893257227e-05 1.84808714787e-06 -
0.0904824397435 0.995834868107 0.998750564626 -0.000279062122488 -
0.00425289384695 6.80717321311e-05 0.000173765252851 7.4792136335e-13
0.000621405968666 -0.0115045117644 0.0468017832124 -0.00485328346898 -
0.0114597339476 -0.00104073962426 -0.0472386502541 1.3198387867e-05 -
0.0899332244992 0.001234007912 0.00322775093607 5.18613717093e-14 -
4.84606423781e-05 0.000543368026424 0.989514300024 -0.102613063915
0.000540128626446 5.96178426612e-05 0.0115189534872 -3.21851565308e-06
-1.36061678914e-05 -0.00096491716022 -2.21416967848e-05
8.19249379072e-15 6.83466921356e-06 -0.000132587826896 -
5.93287005414e-07 -1.52677435235e-05 0.995831029468 0.0904816271279
7.7307401743e-05 7.43572605623e-05 0.813571573724 0.568379157644
0.0920198780335 -1.91076346742e-12 -0.000641679794524 0.00681278021421
0.0684828465973 -0.0429142975077 0.000533013714559 0.00033329546665
0.00010798691895 0.000100831023704 -0.568319132496 0.820636698274
0.00663663887277 8.2963460539e-13 -0.000200108254506 0.00924311158016
-0.0494257961254 0.031552477011 0.000746816304715 0.000453060102503 -
5.16349324192e-06 -1.24301040447e-05 -0.0712521378995 -0.0580180437144
0.995706216264 -6.77714344402e-12 -0.00573291923612 0.00104528444433 -
0.00956698616148 0.000824768609232 -2.99882940619e-05 -5.30278554026e-
05 0.0012101442612 0.000717594637112 -1.79219972615e-05 -
0.00104230360176 -0.0022782060513 0.925362353198 -0.369407708743
0.0850541220102 7.54642914084e-08 7.23822983471e-08 -1.15782410858e-06
-4.28713338982e-07 -0.000317955103855 0.000848714347112 -
5.8563302783e-06 -7.56605503577e-05 0.00532135215213 0.368867026973
0.929190530089 0.0226479854579 8.05407923734e-08 -2.73520097114e-08
4.02370554863e-07 -4.26347525692e-07 0.0114685399925 0.0111775078629 -
0.000214428236363 -0.0113527540638 -0.00166209918326 -0.0874164269195
0.0104300954127 0.995922398378 1.13869475423e-06 6.50799939747e-07 -
1.0558491634e-05 -6.33726113327e-06
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#561 (comment)>, or
unsubscribe <https://github.com/notifications/unsubscribe-
auth/ACQHGSINWP4HBLA4P4KLDHTSLMEOXANCNFSM4SLGTCZQ>.
|
Just realised this issue was still open - it was resolved in #680. |
Hi all,
I'm experiencing some problems with isam2 optimization with plane factors.
Occasionally, the system throws an
gtsam::IndeterminantLinearSystemException
when two planes are present. But if I comment one of them, the system converges.Description
Let's consider the following minimal example (or at least as minimal as I could get to reproduce the error).
The code instantiates a prior pose, two planes, and the priors for the planes.
All the values used are perfect: the measurements and the priors have no errors from the ground truth, so the system should converge immediately to the solution. However, with two planes the system throws a
gtsam::IndeterminantLinearSystemException
. If I comment one of the two planes, no exception is thrown.Steps to reproduce
Expected behavior
The system should converge to the same values of the priors
Environment
Ubuntu 18.04 equipped with GTSAM 4.0.3 stock.
Please note though that the problem persists even when using the branch that fixes the derivatives by @dwisth here.
Additional information
gtsam/nonlinear/ISAM2Clique.cpp
The text was updated successfully, but these errors were encountered: