This repository contains the official PyTorch implementation of the paper: Yichao Zhou, Haozhi Qi, Jingwei Huang, Yi Ma. "NeurVPS: Neural Vanishing Point Scanning via Conic Convolution". NeurIPS 2019.
NeurVPS is an end-to-end trainable deep network with geometry-inspired convolutional operators for detecting vanishing points in images. With the power of data-driven approaches and geometrical priors, NeurVPS is able to outperform the previous state-of-the-art vanishing point detection methods such as LSD/J-Linkage and Contour (TMM17).
| SceneCity Urban 3D (SU3) | Natural Scene (TMM17) | ScanNet |
|---|---|---|
 |
 |
 |
Random sampled results can be found in the supplementary material of the paper.
| SceneCity Urban 3D (SU3) | Natural Scene (TMM17) | ScanNet |
|---|---|---|
 |
 |
 |
Here, the x-axis represents the angle accuracy of the detected vanishing points and the y-axis represents the percentage of the results whose error is less than that. Our conic convolutional networks outperform all the baseline methods and previous state-of-the-art vanishing point detection approaches, while naive CNN implementations might under-perform those traditional methods, especially in the high-accuracy regions.
Below is a quick overview of the function of each file.
########################### Data ###########################
data/ # default folder for placing the data
su3/ # folder for SU3 dataset
tmm17/ # folder for TMM17 dataset
scannet-vp/ # folder for ScanNet dataset
logs/ # default folder for storing the output during training
########################### Code ###########################
config/ # neural network hyper-parameters and configurations
su3.yaml # default parameters for SU3 dataset
tmm17.yaml # default parameters for TMM17 dataset
scannet.yaml # default parameters for scannet dataset
dataset/ # all scripts related to data generation
su3.py # script for pre-processing the SU3 dataset to npz
misc/ # misc scripts that are not important
find-radius.py # script for generating figure grids
neurvps/ # neurvps module so you can "import neurvps" in other scripts
models/ # neural network architectures
cpp/ # CUDA kernel for deformable convolution
deformable.py # python wrapper for deformable convolution layers
conic.py # conic convolution layers
hourglass_pose.py # backbone network
vanishing_net.py # main network
datasets.py # reading the training data
trainer.py # trainer
config.py # global variables for configuration
utils.py # misc functions
train.py # script for training the neural network
eval.py # script for evaluating a dataset from a checkpointFor the ease of reproducibility, you are suggested to install miniconda (or anaconda if you prefer) before following executing the following commands.
git clone https://github.com/zhou13/neurvps
cd neurvps
conda create -y -n neurvps
source activate neurvps
# Replace cudatoolkit=10.1 with your CUDA version: https://pytorch.org/get-started/
conda install -y pytorch cudatoolkit=10.1 -c pytorch
conda install -y tensorboardx gdown -c conda-forge
conda install -y pyyaml docopt matplotlib scikit-image opencv tqdm
mkdir data logsMake sure curl is installed on your system and execute
cd data
wget https://huggingface.co/yichaozhou/neurvps/resolve/main/Data/scannet-vp/scannet-vp.z0{1..7}
wget https://huggingface.co/yichaozhou/neurvps/resolve/main/Data/scannet-vp/scannet-vp.zip
zip -FF scannet-vp.zip --out scannet-vp-full.zip
unzip -FF scannet-vp-full.zip
wget https://huggingface.co/yichaozhou/neurvps/resolve/main/Data/su3.tar.xz
wget https://huggingface.co/yichaozhou/neurvps/resolve/main/Data/tmm17.tar.xz
tar xf su3.tar.xz
tar xf tmm17.tar.xz
rm *.tar.xz *.z*
cd ..Alternatively, you can download the processed datasets manually from our Hugging Face repo and proceed accordingly.
Execute the following commands to train the neural networks from scratch on 2 GPUs (GPU 0 and GPU 1, specified by -d 0,1) with the default parameters:
python ./train.py -d 0,1 --identifier su3 config/su3.yaml
python ./train.py -d 0,1 --identifier tmm17 config/tmm17.yaml
python ./train.py -d 0,1 --identifier scannet config/scannet.yamlThe checkpoints and logs will be written to logs/ accordingly.
Note: For TMM17 dataset, due to its small size the model is more senstive to the initialization. You may need to train it multiple times to reach the same performance as the pre-trained model. For SU3, it has been reported that it is possible to achieve higher performance with 4-GPU training than the reported one in the paper, though the training process is more volatile.
You can download our reference pre-trained models from our Hugging Face repo. Those pre-trained models should be able to reproduce the numbers in our paper.
Execute the following commands to compute and plot the angular accuracy (AA) curves with trained network checkpoints:
python eval.py -d 0 logs/YOUR_LOG/config.yaml logs/YOUR_LOG/checkpoint_best.pth.tarA: I have a blog post on Vanishing Points and “Gaussian Sphere Representation”.
A: Uncomment these lines or these lines to visualize vanishing points overlaid with 2D images.
A: The focal length in our implementation is in the unit of 2/w pixel (w is the image width. only a square image is supported). This follows the convention of the OpenGL projection matrix so that to make it resolution invariant. The focal length is used for uniform sampling of the position of vanishing points. If it is not known, you can set it to some common focal length for your categories of images, as we do in config/tmm17.yaml.
You can also check the function to_label and to_pixel, which use the focal length to convert the 3D line direction from and to a 2D vanishing point.
A: You can post an issue on Github, which might help other people that have the same question. You can also send me an email if you think that is more appropriate.
We thank Yikai Li from SJTU and Jiajun Wu from MIT for pointing out a bug in the data augmentation for the TMM17 Natural Scene dataset. This work is supported by a research grant from Sony Research.
If you find NeurVPS useful in your research, please consider citing:
@inproceedings{zhou2019neurvps,
author={Zhou, Yichao and Qi, Haozhi and Huang, Jingwei and Ma, Yi},
title={{NeurVPS}: Neural Vanishing Point Scanning via Conic Convolution},
booktitle={{NeurIPS}},
year={2019}
}