Zhaoyi Wang · Shengyu Huang · Jemil Avers Butt · Yuanzhou Cai · Matej Varga · Andreas Wieser
This repository provides the official Pytorch implementation of the paper: Cross-Modal Feature Fusion for Robust Point Cloud Registration with Ambiguous Geometry, published in the ISPRS Journal.
📚 Table of Contents
[Abstract (click to expand)]
Point cloud registration has seen significant advancements with the application of deep learning techniques. However, existing approaches often overlook the potential of integrating radiometric information from RGB images. This limitation reduces their effectiveness in aligning point clouds pairs, especially in regions where geometric data alone is insufficient. When used effectively, radiometric information can enhance the registration process by providing context that is missing from purely geometric data. In this paper, we propose CoFF, a novel Cross-modal Feature Fusion method that utilizes both point cloud geometry and RGB images for pairwise point cloud registration. Assuming that the co-registration between point clouds and RGB images is available, CoFF explicitly addresses the challenges where geometric information alone is unclear, such as in regions with symmetric similarity or planar structures, through a two-stage fusion of 3D point cloud features and 2D image features. It incorporates a cross-modal feature fusion module that assigns pixel-wise image features to 3D input point clouds to enhance learned 3D point features, and integrates patch-wise image features with superpoint features to improve the quality of coarse matching. This is followed by a coarse-to-fine matching module that accurately establishes correspondences using the fused features. We extensively evaluate CoFF on four common datasets: 3DMatch, 3DLoMatch, IndoorLRS, and the recently released ScanNet++ datasets. In addition, we assess CoFF on specific subset datasets containing geometrically ambiguous cases. Our experimental results demonstrate that CoFF achieves state-of-the-art registration performance across all benchmarks, including remarkable registration recalls of 95.9% and 81.6% on the widely-used 3DMatch and 3DLoMatch datasets, respectively. CoFF is particularly effective in scenarios with challenging geometric features, provided that RGB images are available and that the overlapping regions exhibit sufficient texture in the RGB images.
Our code builds upon GeoTransformer framework, a 3D geometry-based point cloud registration method. The following key extensions are introduced:
- 🔀 Integration of 2D RGB and 3D geometric features at both point-pixel and patch-patch levels;
- 🧩 Modules for image preprocessing, offline image feature extraction, and feature lifting;
- 🔍 Evaluation scripts for additional datasets: IndoorLRS and ScanNet++.
We recommend users refer to GeoTransformer for more details on the core architecture.
| Method \ RR | 3DM | 3DLM | IndoorLRS | ScanNet++ | 3DM_P | 3DLM_P | IndoorLRS_P | ScanNet++_P |
|---|---|---|---|---|---|---|---|---|
| GeoTrans | 91.5 | 74.0 | 96.2 | 73.4 | 76.4 | 59.5 | 91.2 | 37.6 |
| CoFF (ours) | 95.9 | 81.6 | 97.1 | 78.7 | 90.5 | 70.4 | 94.2 | 56.0 |
Note: 3DM = 3DMatch, 3DLM = 3DLoMatch;
_Pindicates the 'Planar' subset of each dataset.
To run the code, please follow these steps:
- Clone the repository:
# Clone the CoFF repository
git clone git@github.com:zhaoyiww/CoFF.git
cd CoFF
# Create and activate a new Conda environment with Python 3.8
conda create -n coff python=3.8 -y
conda activate coff
# Install PyTorch and its dependencies (CUDA 11.8 version)
pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu118
# Install other required packages
pip install -r requirements.txt
python setup.py build developCoFF is evaluated on the following datasets:
- 3DMatch and 3DLoMatch
- IndoorLRS
- ScanNet++
Please follow the instructions for data preprocessing to prepare the data and metadata.
We provide our pre-trained model weight for direct testing and evaluation, as well as weights for baseline methods. All weights can be downloaded here. Note that all models are trained on the 3DMatch training set and evaluated on the 3DMatch test set, 3DLoMatch, IndoorLRS, and ScanNet++ datasets.
To use CoFF with the pre-trained model, place the downloaded weights in the ./weights/ folder.
-
Modify the config file for the target dataset under the
./configs/test/folder. Set the appropriate paths and parameters. Default configurations for all four datasets are provided. -
To test the model and extract features from the test dataset, run:
bash test.shThis will extract features from the test dataset. To evaluate the registration performance, run:
bash eval.shFor IndoorLRS and ScanNet++ datasets, additionally run benchmark_indoorlrs.py and benchmark_scannetpp.py in the './scripts/' folder to complete the evaluation.
To train your own model, modify the config file under the './configs/train/' folder and set the appropriate paths and parameters. The default configurations for the 3DMatch training set is provided.
Then, run the following command to train the model:
python trainval.py --config ./configs/train/3DMatch.yaml- Upload data preprocessing scripts.
- Upload complete training and evaluation code.
- Upload our pretrained model weight.
- Upload our preprocessed data.
We sincerely thank the following works and use parts of their official implementations:
- GeoTransformer for the 3D backbone (our code is also heavily borrowed from it);
- LCD for the 2D patch-based backbone;
- Pri3D and PCR-CG for the 2D pixel-based backbone;
- 3DMatch for the 3DMatch dataset;
- Predator for the 3DLoMatch dataset;
- Color-ICP for the IndoorLRS dataset;
- ScanNet++ for the ScanNet++ dataset.
If you find our code or paper useful, please cite:
@article{Wang2025CoFF,
title = {Cross-modal feature fusion for robust point cloud registration with ambiguous geometry},
author = {Wang, Zhaoyi and Huang, Shengyu and Butt, Jemil Avers and Cai, Yuanzhou and Varga, Matej and Wieser, Andreas},
journal = {ISPRS Journal of Photogrammetry and Remote Sensing},
volume = {227},
pages = {31--47},
year = {2025},
}