Optimizer Eigen assertion when calibrating with "none" distortion models

[caselitz]

Hi all,

I experienced the following issue when trying to kalibr_calibrate_cameras:

kalibr_calibrate_cameras --bag ~/Downloads/BF2M2020S23-1.bag --topics /cam0/image_raw --models ds-none --target ~/Desktop/target.yaml --verbose
importing libraries
Initializing cam0:
	Camera model:	  ds-none
	Dataset:          /home/caselitz/Downloads/BF2M2020S23-1.bag
	Topic:            /cam0/image_raw
	Number of images: 253
Extracting calibration target corners

... 

  Extracted corners for 253 images (of 253 images)                              
[DEBUG] [1540407694.373795]: calibrateIntrinsics: intrinsics guess: [  0.           0.5        328.19045165 328.19045165 639.5
 511.5       ]
[DEBUG] [1540407694.378700]: calibrateIntrinsics: distortion guess: []
[DEBUG] [1540407694.387976]: calibrateIntrinsics: adding camera error terms for 253 calibration targets
[DEBUG] [1540407703.494753]: calibrateIntrinsics: added 35793 camera error terms
No linear system solver set in the options. Defaulting to the sparse_cholesky solver
Using the sparse_cholesky linear system solver
Using the levenberg_marquardt trust region policy
[DEBUG] [1540407703.499829]: Before optimization:
[DEBUG] [1540407705.404647]:  Reprojection error squarred (camL):  mean 49.7943951178, median 3.70118328943, std: 171.027015386
No linear system solver set in the options. Defaulting to the sparse_cholesky solver
Using the sparse_cholesky linear system solver
Using the levenberg_marquardt trust region policy
Initializing
Optimization problem initialized with 508 design variables and 35793 error terms
The Jacobian matrix is 71586 x 1524
[0.0]: J: 1.78229e+06
python: /usr/include/eigen3/Eigen/src/Core/DenseCoeffsBase.h:408: Eigen::DenseCoeffsBase<Derived, 1>::Scalar& Eigen::DenseCoeffsBase<Derived, 1>::operator[](Eigen::Index) [with Derived = Eigen::Matrix<double, -1, 1>; Eigen::DenseCoeffsBase<Derived, 1>::Scalar = double; Eigen::Index = long int]: Assertion `index >= 0 && index < size()' failed.
Aborted (core dumped)

So I guess the assertion is raised somewhere in this function.

I tried the available camera models (except pinhole) and all *-none models gave me the error. Interestingly, omni-radtan did not!

I tried with my own dataset but then switched to another one from this site (BF2M2020S23). Error also occurs here, so I guess the dataset is not the reason.

Do you think it might be related to the OS (Ubuntu 18.04) or Eigen version (3.3.4)?
@VladyslavUsenko and @NikolausDemmel - when you did the calibrations for your paper, e.g. ds-none, which versions did you use? Do you think the issue could be related to none distortion model, as - if I remember correctly - this was introduced with the PR for the new camera models (thanks for contributing this to kalibr!)?

Thanks for your help!
Best, Tim

PS: I will investigate were the assertion is raised exactly and see if using an older Eigen version helps. Just wanted to open this issue in case someone with more insight can help already. :)

[caselitz]

Ok, so far I tracked it down to this line (called here), so ist must be in the updateImplementation() of some DesignVariable. Probably distortionDesignVariable or so...

[NikolausDemmel]

Hi Tim,

yes, I think this might be related to the Eigen and compiler version. Without knowing exactly, sounds to me like something goes wrong when there are 0 distortion coefficients. Maybe the code still creates a design variable with dimension 0 or something and in newer eigen versions there is an explicit check for something. Could also be a bug in older eigen version that just doesn't show.

Its true that our PR introduced all the ...-none variants, so maybe somehow this is not handled correctly.

We tested only on Ubuntu 16.04, which has a much older eigen version. I think we also tried on macos though, which has much more recent eigen from Homebrew, but I'm not sure. But it also has a different compiler.

Looks like this needs some deeper investigation. I'm not sure when I'll have the time though, unfortunately. Maybe you could try on 16.04 to confirm that it works there (virtual machine or docker?).

Which version of Kalibr are you using? Current master?

[caselitz]

Hi Niko,

thanks for your reply. What you say regarding the design variable sound reasonable to me.

I now tried on a different machine, Ubuntu 16.04, with two Eigen versions. The newest commit from the repo (3.3.90) and libeigen3-dev from the Ubuntu packages (3.2.92). Compiler is gcc 5.4.0. The assertion occurs in both cases. [Edit: Also tried 3.2.10 now, error occurs as well.]

Which "much older eigen version" did you try? Am I missing something or did you mean 3.2.92 which seems to be the current one of Ubuntu 16.04 and which I feel is not so much older?

Yes, I always used current master of Kalibr.

Maybe I can look into the issue next week again. However, any feedback - whenever you find the time - is highly appreciated. :)

[caselitz]

Also wanted to try Eigen 3.2.0 (as used in Ubuntu 14.04), but I already ran into issues during compilation (Boost related, kalibr issue here).

[VladyslavUsenko]

Hi Tim,

Could you provide the full list of commands that you try to run? This way it will be easier to solve the issue. The output you provided seems to come from the kalibr_calibrate_imu_camera and not from kalibr_calibrate_cameras. If this is the case, you should not use it on BF2M2020S23-1.bag dataset because it has no IMU data in it.

If you want to test the IMU calibration, you should try the TUM-VI or Euroc example from here: https://vision.in.tum.de/research/vslam/double-sphere

P.S. I've tested current master with ubuntu 16.04 and eigen 3.3.4 and kalibr_calibrate_cameras works for me with BF2M2020S23-1.bag.

[VladyslavUsenko]

Sorry, looks like it is kalibr_calibrate_cameras but with verbose output. I will try to test it on 18.04. Could you also write the cmake flags that you use? In particular CMAKE_BUILD_TYPE.

[caselitz]

Hi Vladyslav,

thanks for your answer! Sorry for my late response, didn't notice the reply (does GitHub stop to send eMail notifications after a certain number of replies?).

Sorry, looks like it is kalibr_calibrate_cameras but with verbose output.

Correct, definitely no IMU stuff involved. ;)

I will try to test it on 18.04.

Maybe you can save the effort as I also encounter the problem on 16.04 - so right now I feel it is not related to the OS version.

Could you also write the cmake flags that you use? In particular CMAKE_BUILD_TYPE.

What I do is to clone the kalibr repo into my catkin_ws/src and do catkin_make. I guess that should default to Release. No special cmake flags.

P.S. I've tested current master with ubuntu 16.04 and eigen 3.3.4 and kalibr_calibrate_cameras works for me with BF2M2020S23-1.bag.

I really wonder what's the difference in our setup... maybe I'm doing something stupid, hm.
As I said, I already tried on two machines:

1. 18.04, (default) Eigen 3.3.4, ROS Melodic, default CMake, default Boost
2. 16.04, multiple Eigen (smaller and bigger 3.3.4), ROS Kinetic, CMake 3.10.2, Boost 1.58.0

I also feel it is not related to the dataset, because I tried with yours and my own one.

I will try on 16.04 with Eigen 3.3.4 and CMake 3.5.1. Other than that I don't really know what to do. Would be nice if you could reproduce my issue. Can you think of anything else that might be crucial?

Anyways, thanks for your help so far,
Tim

[VladyslavUsenko]

Can you try to run catkin config --cmake-args -DCMAKE_BUILD_TYPE=Release and rebuild the workspace. This way we can make sure we use the same flags (It comes from the installation wiki https://github.com/ethz-asl/kalibr/wiki/installation).

[caselitz]

Ok, I'll try that.

CMake 3.5.1 did not help.

Btw. I noticed a warning during compilation (even though I think it's unrelated):
/home/caselitz/catkin_ws/src/kalibr/aslam_cv/aslam_cameras/include/aslam/cameras/implementation/DoubleSphereProjection.hpp:508:42: warning: unused parameter ‘p’ [-Wunused-parameter]

[NikolausDemmel]

Can you try to run catkin config --cmake-args -DCMAKE_BUILD_TYPE=Release

Yes, please try that. Default for catkin is the same as the cmake default, i.e. no flags (I think), but definitely not Release.

Just to confirm: Are you using caktin_make, or catkin build?

Lastly, are there any other packages in your catkin workspace, or just the Kalibr repository?

[VladyslavUsenko]

I could reproduce the issue. Looks like it hits this assert only when compiled with default flags. The pull request above should fix it even for default flags, but you should in general use Release otherwise it is very slow..

[caselitz]

Default for catkin is the same as the cmake default, i.e. no flags (I think), but definitely not Release.

Your right, I just always default to Release in my own CMake files if no build type is specified. Probably that's why I was guessing that...

Just to confirm: Are you using caktin_make, or catkin build?

I used caktin_make. Your advice based on catkin tools (catkin config --cmake-args -DCMAKE_BUILD_TYPE=Release) only seems to affect catkin build, so I switched to it to resolve the issue.

Should have thought about this earlier. Just recently ran into another Eigen assertion that was only raised in Debug and not Release...

...anyways: thank you for your support! Nice that you could reproduce and sent a PR!