Error when run main_slam.py with FeatureTrackerConfigs.LIGHTGLUE

[sentry0501]

I built pyslam_cuda image with your rosdocker repository.
I successfully run main_slam.py with default config but when I change the FeatureTrackerConfigs.ORB2 to FeatureTrackerConfigs.LIGHTGLUE, I met this error

2024-12-23 12:11:50.931383: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-12-23 12:11:50.932538: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-12-23 12:11:50.954171: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-12-23 12:11:51.379938: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
importing pygame
pygame 2.6.0 (SDL 2.28.4, Python 3.8.10)
Hello from the pygame community. https://www.pygame.org/contribute.html
initialising pygame
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM default
 dataset_factory - sensor_type: MONOCULAR
 read timestamps from /home/benbp/pyslam/./videos/kitti06/times.txt
Processing Video Input
num frames:  1101
fps:  10.0
using groundtruth:  video
Using depth_threshold = 21.486026
 feature_tracker_config:  {'num_features': 2000, 'num_levels': 1, 'scale_factor': 1.2, 'detector_type': FeatureDetectorTypes.SUPERPOINT, 'descriptor_type': FeatureDescriptorTypes.SUPERPOINT, 'sigma_level0': 1.0, 'match_ratio_test': 0.7, 'tracker_type': FeatureTrackerTypes.LIGHTGLUE}
 loop_detection_config:  {'global_descriptor_type': GlobalDescriptorType.DBOW3, 'local_feature_manager_config': None, 'local_descriptor_aggregation_type': LocalDescriptorAggregationType.DBOW3, 'vocabulary_data': <loop_detector_vocabulary.DBow3OrbVocabularyData object at 0x7ccd203b9610>}
num_levels:  1
using opencv  4.10.0
SuperPoint weights: /home/benbp/pyslam/thirdparty/superpoint/superpoint_v1.pth
SuperPoint using  cuda
<feature_superpoint.SuperPointOptions object at 0x7cccb8adb3d0>
SuperPointFeature2D
==> Loading pre-trained network.
==> Successfully loaded pre-trained network.
 using same detector and descriptor object:  SUPERPOINT
 WARNING: using NON-ORIENTED features:  SUPERPOINT - SUPERPOINT  (i.e. kp.angle=0)
num_levels:  1
device:  cuda
 matcher: LightGlueFeatureMatcher
set_rlimit(): Current soft limit: 1048576, hard limit: 1048576
set_rlimit(): Updated soft limit: 4096, hard limit: 1048576
MultiprocessingManager: start method: fork
MultiprocessingManager: start method: fork
 VocabularyData: downloading vocabulary ORB2 from: https://drive.google.com/uc?id=13xmRtop_ow3aPtv3qCT5beG19_mlogqI
downloading https://drive.google.com/uc?id=13xmRtop_ow3aPtv3qCT5beG19_mlogqI to /home/benbp/pyslam/data/ORBvoc.dbow3
MultiprocessingManager: start method: fork
groundtruth shape: (1099, 3)
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM default
MultiprocessingManager: start method: fork
Mplot2d: starting the process on figure: 1
MultiprocessingManager: start method: fork
Mplot2d: starting the process on figure: 2
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-benbp'
MultiprocessingManager: start method: fork
Mplot2d: starting the process on figure: 3
..................................
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-benbp'
image: 0, timestamp: 0.0, duration: 0.1044989
 @tracking MONOCULAR, img id: 0, frame id: 0, state: NO_IMAGES_YET
img.shape: (370, 1226, 3), camera: 370x1226
timestamp:  0.0
 WARNING: could not set nFeatures in detector:  SUPERPOINT
 WARNING: could not set nFeatures in descriptor:  SUPERPOINT
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-benbp'
/home/benbp/.python/venvs/pyslam/lib/python3.8/site-packages/torch/nn/functional.py:4316: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  warnings.warn(
detector: SUPERPOINT, #features:  1053 , frame res:  (370, 1226)
detector: SUPERPOINT , #features: 1053
descriptor: SUPERPOINT , #features: 1053
detector: SUPERPOINT , descriptor: SUPERPOINT , #features: 1053  (#ref: 4000 ), [kp-filter: SAT ]
..................................
image: 1, timestamp: 0.1044989, duration: 0.10329459999999999
 @tracking MONOCULAR, img id: 1, frame id: 1, state: NOT_INITIALIZED
img.shape: (370, 1226, 3), camera: 370x1226
timestamp:  0.1044989
 WARNING: could not set nFeatures in detector:  SUPERPOINT
 WARNING: could not set nFeatures in descriptor:  SUPERPOINT
detector: SUPERPOINT, #features:  974 , frame res:  (370, 1226)
detector: SUPERPOINT , #features: 974
descriptor: SUPERPOINT , #features: 974
detector: SUPERPOINT , descriptor: SUPERPOINT , #features: 974  (#ref: 4000 ), [kp-filter: SAT ]
|------------
Initializer: processing frames: 1, 0
 Inializer: ko - init frames are too close!
..................................
image: 2, timestamp: 0.2077935, duration: 0.10384079999999998
 @tracking MONOCULAR, img id: 2, frame id: 2, state: NOT_INITIALIZED
img.shape: (370, 1226, 3), camera: 370x1226
timestamp:  0.2077935
 WARNING: could not set nFeatures in detector:  SUPERPOINT
 WARNING: could not set nFeatures in descriptor:  SUPERPOINT
detector: SUPERPOINT, #features:  947 , frame res:  (370, 1226)
detector: SUPERPOINT , #features: 947
descriptor: SUPERPOINT , #features: 947
detector: SUPERPOINT , descriptor: SUPERPOINT , #features: 947  (#ref: 4000 ), [kp-filter: SAT ]
|------------
Initializer: processing frames: 2, 0
Traceback (most recent call last):
  File "./main_slam.py", line 177, in <module>
    slam.track(img, img_right, depth, img_id, timestamp)  # main SLAM function 
  File "/home/benbp/pyslam/slam.py", line 158, in track
    return self.tracking.track(img, img_right, depth, img_id, timestamp)
  File "/home/benbp/pyslam/tracking.py", line 854, in track
    initializer_output, intializer_is_ok = self.intializer.initialize(f_cur, img)
  File "/home/benbp/pyslam/initializer.py", line 129, in initialize
    return self.initialize_simple(f_cur, img_cur)     # try to inizialize current frame with reference frame: reference position is adjusted in the buffer
  File "/home/benbp/pyslam/initializer.py", line 145, in initialize_simple
    return self.process_frames(f_cur, img_cur, f_ref)
  File "/home/benbp/pyslam/initializer.py", line 191, in process_frames
    matching_result = match_frames(f_cur, f_ref, kInitializerFeatureMatchRatioTest)      
  File "/home/benbp/pyslam/frame.py", line 1025, in match_frames
    matching_result = FrameShared.feature_matcher.match(f1.img, f2.img, f1.des, f2.des, ratio_test)
  File "/home/benbp/pyslam/feature_matcher.py", line 342, in match
    kps1 = np.array([x.pt for x in kps1], dtype=np.float32)
TypeError: 'float' object is not iterable

How can I fix it?
Thanks in advanced!!

[luigifreda]

Hi,
Thank you for pointing out the error.
I’ve just pushed a fix.

Please note that LIGHTGLUE cannot be used with the default DBOW3 loop-closing configuration. Instead, you can select one of the *_INDEPENDENT or vocabulary-free methods. For more details, refer to the main README and loop_closing/loop_detector_configs.py.

Best regards,
Luigi

[sentry0501]

Hello again, @luigifreda
I've successfull run LightGlue+OBINDEX2 with mono cam but when run on stereo cam, i keep getting this error, try on both kitti_06 and euroc mh01 dataset with LIGHTGLUE+SAD/DBOW2_INDEPENDENT:

2024-12-26 13:50:43.800985: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-12-26 13:50:43.802143: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-12-26 13:50:43.825252: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-12-26 13:50:44.216330: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
importing pygame
pygame 2.6.0 (SDL 2.28.4, Python 3.8.10)
Hello from the pygame community. https://www.pygame.org/contribute.html
initialising pygame
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM default
 dataset_factory - sensor_type: STEREO
Processing KITTI Sequence of lenght:  1101
dataset in color!
using groundtruth:  kitti
 WARNING: Using stereo camera, images are automatically rectified, and DistCoef is set to [0,0,0,0,0]
Using depth_threshold = 21.486026
 feature_tracker_config:  {'num_features': 2000, 'num_levels': 1, 'scale_factor': 1.2, 'detector_type': FeatureDetectorTypes.SUPERPOINT, 'descriptor_type': FeatureDescriptorTypes.SUPERPOINT, 'sigma_level0': 1.0, 'match_ratio_test': 1.0, 'tracker_type': FeatureTrackerTypes.LIGHTGLUE}
 loop_detection_config:  {'global_descriptor_type': GlobalDescriptorType.SAD, 'local_feature_manager_config': None, 'local_descriptor_aggregation_type': LocalDescriptorAggregationType.NONE, 'vocabulary_data': None}
num_levels:  1
using opencv  4.10.0
SuperPoint weights: /home/benbp/pyslam/thirdparty/superpoint/superpoint_v1.pth
SuperPoint using  cuda
<feature_superpoint.SuperPointOptions object at 0x7d4c5b1672e0>
SuperPointFeature2D
==> Loading pre-trained network.
==> Successfully loaded pre-trained network.
 using same detector and descriptor object:  SUPERPOINT
 WARNING: using NON-ORIENTED features:  SUPERPOINT - SUPERPOINT  (i.e. kp.angle=0)
num_levels:  1
device:  cuda
 matcher: LightGlueFeatureMatcher
set_rlimit(): Current soft limit: 1048576, hard limit: 1048576
set_rlimit(): Updated soft limit: 4096, hard limit: 1048576
MultiprocessingManager: start method: fork
MultiprocessingManager: start method: fork
MultiprocessingManager: start method: fork
groundtruth shape: (1099, 3)
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM default
MultiprocessingManager: start method: fork
Mplot2d: starting the process on figure: 1
MultiprocessingManager: start method: fork
Mplot2d: starting the process on figure: 2
MultiprocessingManager: start method: fork
Mplot2d: starting the process on figure: 3
MultiprocessingManager: start method: fork
Mplot2d: starting the process on figure: 4
..................................
[KittiDataset] getImageRight: 0
image: 0, timestamp: 0.0, duration: 0.1044989
 @tracking STEREO, img id: 0, frame id: 0, state: NO_IMAGES_YET
img.shape: (370, 1226, 3), camera: 370x1226
img_right.shape:  (370, 1226, 3)
timestamp:  0.0
 WARNING: could not set nFeatures in detector:  SUPERPOINT
 WARNING: could not set nFeatures in descriptor:  SUPERPOINT
/home/benbp/.python/venvs/pyslam/lib/python3.8/site-packages/torch/nn/functional.py:4316: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  warnings.warn(
detector: SUPERPOINT , #features: 930
descriptor: SUPERPOINT , #features: 930
detector: SUPERPOINT , descriptor: SUPERPOINT , #features: 930  (#ref: 4000 ), [kp-filter: SAT ]
detector: SUPERPOINT , #features: 950
descriptor: SUPERPOINT , #features: 950
detector: SUPERPOINT , descriptor: SUPERPOINT , #features: 950  (#ref: 4000 ), [kp-filter: SAT ]
Traceback (most recent call last):
  File "./main_slam.py", line 177, in <module>
    slam.track(img, img_right, depth, img_id, timestamp)  # main SLAM function 
  File "/home/benbp/pyslam/slam/slam.py", line 158, in track
    return self.tracking.track(img, img_right, depth, img_id, timestamp)
  File "/home/benbp/pyslam/slam/tracking.py", line 837, in track
    f_cur = Frame(self.camera, img, img_right=img_right, depth=depth, timestamp=timestamp, img_id=img_id) 
  File "/home/benbp/pyslam/slam/frame.py", line 406, in __init__
    self.compute_stereo_matches(img, img_right)
  File "/home/benbp/pyslam/slam/frame.py", line 712, in compute_stereo_matches
    matched_kps_r = np.array(self.kps_r[stereo_matching_result.idxs2])         
IndexError: index 930 is out of bounds for axis 0 with size 930

[sentry0501]

Hello again, @luigifreda I've successfull run LightGlue+OBINDEX2 with mono cam but when run on stereo cam, i keep getting this error, try on both kitti_06 and euroc mh01 dataset with LIGHTGLUE+SAD/DBOW2_INDEPENDENT:

2024-12-26 13:50:43.800985: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-12-26 13:50:43.802143: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2024-12-26 13:50:43.825252: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-12-26 13:50:44.216330: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
importing pygame
pygame 2.6.0 (SDL 2.28.4, Python 3.8.10)
Hello from the pygame community. https://www.pygame.org/contribute.html
initialising pygame
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM default
 dataset_factory - sensor_type: STEREO
Processing KITTI Sequence of lenght:  1101
dataset in color!
using groundtruth:  kitti
 WARNING: Using stereo camera, images are automatically rectified, and DistCoef is set to [0,0,0,0,0]
Using depth_threshold = 21.486026
 feature_tracker_config:  {'num_features': 2000, 'num_levels': 1, 'scale_factor': 1.2, 'detector_type': FeatureDetectorTypes.SUPERPOINT, 'descriptor_type': FeatureDescriptorTypes.SUPERPOINT, 'sigma_level0': 1.0, 'match_ratio_test': 1.0, 'tracker_type': FeatureTrackerTypes.LIGHTGLUE}
 loop_detection_config:  {'global_descriptor_type': GlobalDescriptorType.SAD, 'local_feature_manager_config': None, 'local_descriptor_aggregation_type': LocalDescriptorAggregationType.NONE, 'vocabulary_data': None}
num_levels:  1
using opencv  4.10.0
SuperPoint weights: /home/benbp/pyslam/thirdparty/superpoint/superpoint_v1.pth
SuperPoint using  cuda
<feature_superpoint.SuperPointOptions object at 0x7d4c5b1672e0>
SuperPointFeature2D
==> Loading pre-trained network.
==> Successfully loaded pre-trained network.
 using same detector and descriptor object:  SUPERPOINT
 WARNING: using NON-ORIENTED features:  SUPERPOINT - SUPERPOINT  (i.e. kp.angle=0)
num_levels:  1
device:  cuda
 matcher: LightGlueFeatureMatcher
set_rlimit(): Current soft limit: 1048576, hard limit: 1048576
set_rlimit(): Updated soft limit: 4096, hard limit: 1048576
MultiprocessingManager: start method: fork
MultiprocessingManager: start method: fork
MultiprocessingManager: start method: fork
groundtruth shape: (1099, 3)
ALSA lib confmisc.c:767:(parse_card) cannot find card '0'
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory
ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory
ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name
ALSA lib conf.c:4732:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory
ALSA lib conf.c:5220:(snd_config_expand) Evaluate error: No such file or directory
ALSA lib pcm.c:2642:(snd_pcm_open_noupdate) Unknown PCM default
MultiprocessingManager: start method: fork
Mplot2d: starting the process on figure: 1
MultiprocessingManager: start method: fork
Mplot2d: starting the process on figure: 2
MultiprocessingManager: start method: fork
Mplot2d: starting the process on figure: 3
MultiprocessingManager: start method: fork
Mplot2d: starting the process on figure: 4
..................................
[KittiDataset] getImageRight: 0
image: 0, timestamp: 0.0, duration: 0.1044989
 @tracking STEREO, img id: 0, frame id: 0, state: NO_IMAGES_YET
img.shape: (370, 1226, 3), camera: 370x1226
img_right.shape:  (370, 1226, 3)
timestamp:  0.0
 WARNING: could not set nFeatures in detector:  SUPERPOINT
 WARNING: could not set nFeatures in descriptor:  SUPERPOINT
/home/benbp/.python/venvs/pyslam/lib/python3.8/site-packages/torch/nn/functional.py:4316: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  warnings.warn(
detector: SUPERPOINT , #features: 930
descriptor: SUPERPOINT , #features: 930
detector: SUPERPOINT , descriptor: SUPERPOINT , #features: 930  (#ref: 4000 ), [kp-filter: SAT ]
detector: SUPERPOINT , #features: 950
descriptor: SUPERPOINT , #features: 950
detector: SUPERPOINT , descriptor: SUPERPOINT , #features: 950  (#ref: 4000 ), [kp-filter: SAT ]
Traceback (most recent call last):
  File "./main_slam.py", line 177, in <module>
    slam.track(img, img_right, depth, img_id, timestamp)  # main SLAM function 
  File "/home/benbp/pyslam/slam/slam.py", line 158, in track
    return self.tracking.track(img, img_right, depth, img_id, timestamp)
  File "/home/benbp/pyslam/slam/tracking.py", line 837, in track
    f_cur = Frame(self.camera, img, img_right=img_right, depth=depth, timestamp=timestamp, img_id=img_id) 
  File "/home/benbp/pyslam/slam/frame.py", line 406, in __init__
    self.compute_stereo_matches(img, img_right)
  File "/home/benbp/pyslam/slam/frame.py", line 712, in compute_stereo_matches
    matched_kps_r = np.array(self.kps_r[stereo_matching_result.idxs2])         
IndexError: index 930 is out of bounds for axis 0 with size 930

Fixed by replacing return np.array(out_idxs1), np.array(out_idxs1) with return np.array(out_idxs1), np.array(out_idxs2) in here

[luigifreda]

Thanks for pointing out the issue.
I've pushed the fix.