Chao Chen, Xinhao Liu, Xuchu Xu, Yiming Li, Li Ding, Ruoyu Wang, Chen Feng
"A Novel self-supervised VPR model capable of retrieving positives from various orientations."
Visual place recognition (VPR) using deep networks has achieved state-of-the-art performance. However, most of the related approaches require a training set with ground truth sensor poses to obtain the positive and negative samples of each observation's spatial neighborhoods. When such knowledge is unknown, the temporal neighborhoods from a sequentially collected data stream could be exploited for self-supervision, although with suboptimal performance. Inspired by noisy label learning, we propose a novel self-supervised VPR framework that uses both the temporal neighborhoods and the learnable feature neighborhoods to discover the unknown spatial neighborhoods. Our method follows an iterative training paradigm which alternates between: (1) representation learning with data augmentation, (2) positive set expansion to include the current feature space neighbors, and (3) positive set contraction via geometric verification. We conduct comprehensive experiments on both simulated and real datasets, with input of both images and point clouds. The results demonstrate that our method outperforms the baselines in both recall rate, robustness, and a novel metric we proposed for VPR, the orientation diversity.
Download links:
- For Pointcloud: Please refer to DeepMapping paper, https://github.com/ai4ce/PointCloudSimulator
- For NYU-VPR-360 dataset: https://drive.google.com/drive/u/0/folders/1ErXzIx0je5aGSRFbo5jP7oR8gPrdersO
You could find more detailed documents on our website!
Data_folder
├── 000001.pcd or 000001.RGB # Dataset
├── 000002.pcd or 000002.RGB # Dataset
| ...
├── 00nnnn.pcd or 00nnnn.RGB # Dataset
├── gt_pose.mat # ground truth mat file contains the geographical information for evaluation
TF-VPR follows the same file structure as the PointNetVLAD:
TF-VPR
├── loss # loss function
├── models # network model
| ├── PointNetVlad.py # PointNetVLAD network model
| ├── ImageNetVLAD.py # NetVLAD network model
| ├── resnet_mod.py # ResNet network model
| ├── Verification_PCL.py # Verification for PCL data
| ├── Verification_RGB.py # Verification for RGB data
| ├── Verification_RGB_real.py # Verification for RGB_real data
| ...
├── generating_queries # Preprocess the data, initial the label, and generate Pickle file
| ├── generate_test_PCL_baseline_sets.py # Generate the test pickle file for PCL baseline
| ├── generate_test_PCL_ours_sets.py # Generate the test pickle file for PCL TF-VPR
| ├── generate_test_PCL_supervise_sets.py # Generate the test pickle file for PCL supervise
| ├── generate_test_RGB_baseline_sets.py # Generate the test pickle file for Habitat-sim baseline
| ├── generate_test_RGB_ours_sets.py # Generate the test pickle file for Habitat-sim TF-VPR
| ├── generate_test_RGB_supervise_sets.py # Generate the test pickle file for Habitat-sim supervise
| ├── generate_test_RGB_real_baseline_sets.py # Generate the test pickle file for NYU-VPR-360 baseline
| ├── generate_test_RGB_real_ours_sets.py # Generate the test pickle file for NYU-VPR-360 TF-VPR
| ├── generate_test_RGB_real_supervise_sets.py # Generate the test pickle file for NYU-VPR-360 supervise
| ├── generate_training_tuples_PCL_baseline.py # Generate the train pickle file for PCL baseline
| ├── generate_training_tuples_PCL_ours.py # Generate the train pickle file for PCL TF-VPR
| ├── generate_training_tuples_PCL_supervise.py # Generate the train pickle file for PCL supervise
| ├── generate_training_tuples_RGB_baseline.py # Generate the train pickle file for Habitat-sim baseline
| ├── generate_training_tuples_RGB_ours.py # Generate the train pickle file for Habitat-sim TF-VPR
| ├── generate_training_tuples_RGB_supervise.py # Generate the train pickle file for Habitat-sim supervise
| ├── generate_training_tuples_RGB_supervise.py # Generate the train pickle file for NYU-VPR-360 baseline
| ├── generate_training_tuples_RGB_real_ours.py # Generate the train pickle file for NYU-VPR-360 TF-VPR
| ├── generate_training_tuples_RGB_real_supervise.py # Generate the train pickle file for NYU-VPR-360 supervise
| ...
├── results # Results are saved here
├── config.py # Config file
├── evaluate.py # evaluate file
├── loading_pointcloud.py # file loading script
├── train_pointnetvlad_PCL_baseline.py # Main file to train PCL baseline
├── train_pointnetvlad_PCL_ours.py # Main file to train PCL TF-VPR
├── train_pointnetvlad_PCL_supervise.py # Main file to train PCL supervise
├── train_netvlad_RGB_baseline.py # Main file to train Hatbitat-sim baseline
├── train_netvlad_RGB_ours.py # Main file to train Hatbitat-sim TF-VPR
├── train_netvlad_RGB_supervise.py # Main file to train Hatbitat-sim supervise
├── train_netvlad_RGB_real_baseline.py # Main file to train NYU-VPR-360 baseline
├── train_netvlad_RGB_real_ours.py # Main file to train NYU-VPR-360 TF-VPR
├── train_netvlad_RGB_real_supervise.py # Main file to train NYU-VPR-360 supervise
| ...
Point cloud TF-VPR result:
RGB TF-VPR result:
NYU-VPR-360 TF-VPR result:
I kept almost everything not related to tensorflow as the original implementation. The main differences are:
- Multi-GPU support
- Configuration file (config.py)
- Evaluation on the eval dataset after every epochs
- Python 3.6
- PyTorch >=1.10.0
- tensorboardX
- open3d-python 0.4
- scipy
- matplotlib
- numpy
- pandas
- scikit-learn
- pickle5
- torchvision
- opencv-contrib-python
cd generating_queries/
### For Pointcloud data
# To create pickle file for PCL baseline method
python generate_training_tuples_PCL_baseline.py
# To create pickle file for PCL TF-VPR method
python generate_training_tuples_PCL_ours.py
# To create pickle file for PCL supervise method
python generate_training_tuples_PCL_supervise.py
# To create pickle file for PCL baseline evaluation pickle
python generate_test_PCL_baseline_sets.py
# To create pickle file for PCL TF-VPR evaluation pickle
python generate_test_PCL_ours_sets.py
# To create pickle file for PCL supervise evaluation pickle
python generate_test_PCL_supervise_sets.py
cd generating_queries/
### For Habitat-sim data
# For training tuples in our RGB baseline network
python generate_training_tuples_RGB_baseline.py
# For training tuples in our RGB baseline network
python generate_training_tuples_RGB_ours.py
# For training tuples in our RGB baseline network
python generate_training_tuples_RGB_supervise.py
# For RGB network evaluation
python generate_test_RGB_baseline_sets.py
# For RGB network evaluation
python generate_test_RGB_ours_sets.py
# For RGB network evaluation
python generate_test_RGB_supervise_sets.py
cd generating_queries/
### For NYU-VPR-360 data
# For training tuples in our RGB baseline network
python generate_training_tuples_RGB_real_baseline.py
# For training tuples in our RGB baseline network
python generate_training_tuples_RGB_real_ours.py
# For training tuples in our RGB baseline network
python generate_training_tuples_RGB_real_supervise.py
# For RGB network evaluation
python generate_test_RGB_baseline_real_sets.py
# For RGB network evaluation
python generate_test_RGB_real_ours_sets.py
# For RGB network evaluation
python generate_test_RGB_real_supervise_sets.py
### For point cloud
python Verification_PCL.py # you can create max and min threshold using this command. For RGB, we calculate the threshold on Verification stage, no need to precalculate
### For point cloud
python train_pointnetvlad_PCL_baseline.py # Train baseline for PCL data
python train_pointnetvlad_PCL_ours.py # Train ours for PCL data
python train_pointnetvlad_PCL_supervise.py # Train supervise for PCL data
### For Habitat-sim
python train_pointnetvlad_RGBL_baseline.py # Train baseline for Habitat-sim data
python train_pointnetvlad_RGB_ours.py # Train ours for Habitat-sim data
python train_pointnetvlad_RGB_supervise.py # Train supervise for Habitat-sim data
### For point cloud
python train_pointnetvlad_RGB_real_baseline.py # Train baseline for NYU-VPR-360 data
python train_pointnetvlad_RGB_real_ours.py # Train baseline for NYU-VPR-360 data
python train_pointnetvlad_RGB_real_supervise.py # Train baseline for NYU-VPR-360 data
Evaluate (You don't need to run it separately. For every epoch, evaluation will be run automatically)
python evaluate.py
Take a look at train_pointnetvlad.py and evaluate.py for more parameters
We implement SPTM, TF-VPR, and supervise version, please check the other branches for reference
If you find TF-VPR useful in your research, please cite:
@article{Chen2022arxiv,
author = {Chen, Chao and Liu, Xinhao and Xu, Xuchu and Li, Yiming and Ding, Li and Wang, Ruoyu and Feng, Chen},
title = {Self-Supervised Visual Place Recognition by Mining Temporal and Feature Neighborhoods},
journal = {arxiv},
year = {2022}
}Chen Feng is the corresponding author. The research is supported by NSF Future Manufacturing program under CMMI-1932187, CNS-2121391, and EEC-2036870. Chao Chen gratefully thanks the help from Bhargav Makwana.