Skip to content

Commit

Permalink
first commit
Browse files Browse the repository at this point in the history
  • Loading branch information
@apchenstu
apchenstu committed Jul 2, 2021
1 parent ff5b106 commit d3ce599
Show file tree
Hide file tree
Showing 23 changed files with 5,310 additions and 1 deletion.
166 changes: 165 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
@@ -1 +1,165 @@
# mvsnerf
# MVSNeRF
## [Project page](https://apchenstu.github.io/mvsnerf/) | [Paper](https://arxiv.org/abs/2103.15595)
This repository contains a pytorch lightning implementation for the paper: [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://arxiv.org/abs/1911.11530). Our work present a novel neural rendering approach that can efficiently reconstruct
geometric and neural radiance fields for view synthesis, Moreover, if dense images are captured, our estimated radiance field representation can be easily fine-tuned; this leads to fast per-scene reconstruction.<br><br>

![Pipeline](configs/pipeline.png)

## Installation

#### Tested on Ubuntu 16.04 + Pytorch 1.8 + Pytorch Lignting 1.3.5

Install environment:
```
pip install pytorch-lightning, inplace_abn
pip install imageio, pillow, scikit-image, opencv-python, config-argparse, lpips
```


## Training
Please see each subsection for training on different datasets. Available training datasets:

* [DTU](#dtu)
* [Blender](#blender) (Realistic Synthetic)
* [LLFF](#llff) (Real Forward-Facing)
* [Your own data](#your-own-data) (images/intrinsic/extrinsic/nearfar boundles)

### DTU dataset

#### Data download

Download the preprocessed [DTU training data](https://drive.google.com/file/d/1eDjh-_bxKKnEuz5h-HXS7EDJn59clx6V/view)
and [Depth_raw](https://virutalbuy-public.oss-cn-hangzhou.aliyuncs.com/share/cascade-stereo/CasMVSNet/dtu_data/dtu_train_hr/Depths_raw.zip) from original [MVSNet repo](https://github.com/YoYo000/MVSNet)
and unzip. We provide a [DTU example](https://1drv.ms/u/s!AjyDwSVHuwr8zhAAXh7x5We9czKj?e=oStQ48), please
follow with the example's folder structure.

#### Training model

Run
```
CUDA_VISIBLE_DEVICES=$cuda python train_mvs_nerf_pl.py \
--expname $exp_name
--num_epochs 6
--use_viewdirs \
--dataset_name dtu \
--datadir $DTU_DIR
```
More options refer to the `opt.py`, training command example:
```
CUDA_VISIBLE_DEVICES=0 python train_mvs_nerf_pl.py
--with_depth --imgScale_test 1.0 \
--expname mvs-nerf-is-all-your-need \
--num_epochs 6 --N_samples 128 --use_viewdirs --batch_size 1024 \
--dataset_name dtu \
--datadir path/to/dtu/data \
--N_vis 6
```

You may need to add `--with_depth` if you want to quantity depth during training. `--N_vis` denotes the validation frequency.
`--imgScale_test` is the downsample ratio during validation, like 0.5. The training process takes about 30h on single RTX 2080Ti
for 6 epochs.

*Important*: please always set batch_size to 1 when you are trining a genelize model, you can enarge it when fine-tuning.

*Checkpoint*: a pre-trained checkpint is included in `ckpts/mvsnerf-v0.tar`.

*Evaluation*: We also provide a rendering and quantity scipt in `renderer.ipynb`,
and you also can use the run_batch.py if you want to testing or finetuning on different dataset.

Rendering from the trained model should have result like this:

![no-finetuned](https://user-images.githubusercontent.com/16453770/124207949-210b8300-db19-11eb-9ab9-610eff35395e.gif)

## Finetuning
### Blender
<details>
<summary>Steps</summary>

#### Data download

Download `nerf_synthetic.zip` from [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1)

```
CUDA_VISIBLE_DEVICES=0 python train_mvs_nerf_finetuning_pl.py \
--dataset_name blender --datadir /path/to/nerf_synthetic/lego \
--expname lego-ft --with_rgb_loss --batch_size 1024 \
--num_epochs 1 --imgScale_test 1.0 --white_bkgd --pad 0 \
--ckpt ./ckpts/mvsnerf-v0.tar --N_vis 1
```

</details>

### LLFF
<details>
<summary>Steps</summary>


#### Data download

Download `nerf_llff_data.zip` from [here](https://drive.google.com/drive/folders/128yBriW1IG_3NJ5Rp7APSTZsJqdJdfc1)

```
CUDA_VISIBLE_DEVICES=0 python train_mvs_nerf_finetuning_pl.py \
--dataset_name llff --datadir /path/to/nerf_llff_data/{scene_name} \
--expname horns-ft --with_rgb_loss --batch_size 1024 \
--num_epochs 1 --imgScale_test 1.0 --pad 24 \
--ckpt ./ckpts/mvsnerf-v0.tar --N_vis 1
```

</details>

### DTU
<details>
<summary>Steps</summary>

```
CUDA_VISIBLE_DEVICES=0 python train_mvs_nerf_finetuning_pl.py \
--dataset_name dtu_ft --datadir /path/to/DTU/mvs_training/dtu/scan1 \
--expname scan1-ft --with_rgb_loss --batch_size 1024 \
--num_epochs 1 --imgScale_test 1.0 --pad 24 \
--ckpt ./ckpts/mvsnerf-v0.tar --N_vis 1
```

</details>

## Rendering
After training or finetuning, you can render free-viewpoint videos
with the `renderer-video.ipynb`. if you want to use your own data,
please follow the prepare the data (intrinsic, nearfar and extrinsic either with
camera to world or world to camera in opencv format) and modify the rendering scipts.

After 10k iterations (~ 15min), you should have videos like this:

![finetuned](https://user-images.githubusercontent.com/16453770/124207013-15b75800-db17-11eb-8d96-e8dbe4181c98.gif)


## Citation
If you find our code or paper helps, please consider citing:
```
@article{chen2021mvsnerf,
title={MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo},
author={Chen, Anpei and Xu, Zexiang and Zhao, Fuqiang and Zhang, Xiaoshuai and Xiang, Fanbo and Yu, Jingyi and Su, Hao},
journal={arXiv preprint arXiv:2103.15595},
year={2021}
}
}
```

Big thanks to [**CasMVSNet_pl**](https://github.com/kwea123/CasMVSNet_pl), our code are partially
borrow from them.

## Relevant Works
[**MVSNet: Depth Inference for Unstructured Multi-view Stereo (ECCV 2018)**](https://arxiv.org/abs/1804.02505)<br>
Justus Thies, Michael Zollhöfer, Matthias Nießner

[**Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching (CVPR 2020)**](https://arxiv.org/abs/1912.06378)<br>
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, Michael Zollhöfer

[**NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020)**](http://www.matthewtancik.com/nerf)<br>
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng

[**IBRNet: Learning Multi-View Image-Based Rendering (CVPR 2021)**](https://ibrnet.github.io/)<br>
Konstantinos Rematas, Vittorio Ferrari

[**PixelNeRF: Neural Radiance Fields from One or Few Images (CVPR 2021)**](https://alexyu.net/pixelnerf/)<br>
Michael Oechsle, Michael Niemeyer, Lars Mescheder, Thilo Strauss, Andreas Geiger
Binary file added ckpts/mvsnerf-v0.tar
View file
Binary file not shown.
99 changes: 99 additions & 0 deletions configs/dtu_pairs.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
49
0
10 10 2346.41 1 2036.53 9 1243.89 12 1052.87 11 1000.84 13 703.583 2 604.456 8 439.759 14 327.419 27 249.278
1
10 9 2850.87 10 2583.94 2 2105.59 0 2052.84 8 1868.24 13 1184.23 14 1017.51 12 961.966 7 670.208 15 657.218
2
10 8 2501.24 1 2106.88 7 1856.5 9 1782.34 3 1141.77 15 1061.76 14 815.457 16 762.153 6 709.789 10 699.921
3
10 7 1294.39 6 1159.13 2 1134.27 4 905.717 8 687.32 5 600.015 17 496.958 16 481.969 1 379.011 15 307.45
4
10 5 1333.74 6 1145.15 3 895.254 7 486.504 18 446.42 2 418.517 17 326.528 8 161.115 16 149.154 1 103.626
5
10 6 1676.06 18 1555.06 4 1335.55 17 868.416 3 593.755 7 467.816 20 440.579 19 428.255 16 242.327 21 210.253
6
10 17 2332.35 7 1848.24 18 1812.74 5 1696.07 16 1273 3 1157.99 4 1155.41 20 771.624 21 744.945 2 700.368
7
10 16 2709.46 8 2439.7 15 2078.21 6 1864.16 2 1846.6 17 1791.71 3 1296.86 22 957.793 9 879.088 21 782.277
8
10 15 3124.01 9 3099.92 14 2756.29 2 2501.22 7 2449.32 1 1875.94 16 1726.04 13 1325.76 23 1177.09 24 1108.82
9
10 13 3355.62 14 3226.07 8 3098.8 10 3097.07 1 2861.42 12 1873.63 2 1785.98 15 1753.32 25 1365.45 0 1261.59
10
10 12 3750.7 9 3085.87 13 3028.39 1 2590.55 0 2369.79 11 2266.67 14 1524.16 26 1448.15 27 1293.6 8 1041.84
11
10 12 3543.76 27 3056.05 10 2248.07 26 1524.28 28 1273.33 13 1265.9 29 1129.55 0 998.164 9 591.176 30 572.919
12
10 27 3889.87 10 3754.54 13 3745.21 11 3584.26 26 3574.56 25 1877.11 9 1866.34 29 1482.72 30 1418.51 14 1341.86
13
10 12 3773.14 26 3699.28 25 3657.17 14 3652.04 9 3356.29 10 3049.27 24 2098.91 27 1900.96 31 1460.96 30 1349.62
14
10 13 3663.52 24 3610.69 9 3232.55 25 3216.4 15 3128.84 8 2758.04 23 2219.91 26 1567.45 10 1536.6 32 1419.33
15
10 23 3194.92 14 3126 8 3120.43 16 2897.02 24 2562.49 7 2084.05 22 2041.63 9 1752.08 33 1232.29 13 1137.55
16
10 15 2884.14 7 2713.88 22 2708.57 17 2448.5 21 2173.3 23 1908.03 8 1718.79 6 1281.96 35 1047.38 34 980.064
17
10 21 2632.48 16 2428 6 2343.57 18 2250.23 20 2149.75 7 1779.42 22 1380.25 36 957.046 5 878.398 15 789.068
18
10 17 2219.15 20 2173.02 6 1802.39 19 1575.77 5 1564.81 21 1160.13 37 827.951 16 660.317 7 589.484 36 559.983
19
10 20 1828.97 18 1564.63 37 1474.35 17 685.249 38 620.304 36 613.42 21 572.77 39 499.123 5 427.597 6 368.651
20
10 37 2569.8 21 2569.79 36 2258.33 18 2186.71 17 2130.67 19 1865.06 39 1049.03 35 996.122 16 799.808 40 778.721
21
10 36 2704.59 35 2639.69 17 2638.19 20 2605.43 22 2604.26 16 2158.25 37 1446.49 34 1239.25 18 1178.24 40 1128.57
22
10 23 3232.68 34 3175.15 35 2831.09 16 2712.51 21 2632.19 15 2033.39 33 1712.67 17 1393.86 36 1290.96 24 1195.33
23
10 24 3710.9 33 3603.07 22 3244.2 15 3190.62 34 3086.49 14 2220.11 32 2100 16 1917.1 35 1359.79 25 1356.71
24
10 25 3844.6 32 3750.75 23 3710.6 14 3609.09 33 3091.04 15 2559.24 31 2423.71 13 2109.36 26 1440.58 34 1410.03
25
10 26 3951.74 31 3888.57 24 3833.07 13 3667.35 14 3208.21 32 2993.46 30 2681.52 12 1900.23 45 1484.03 27 1462.88
26
10 30 4033.35 27 3970.47 25 3925.25 13 3686.34 12 3595.59 29 2943.87 31 2917 14 1556.34 11 1554.75 46 1503.84
27
10 29 4027.84 26 3929.94 12 3875.58 11 3085.03 28 2908.6 30 2792.67 13 1878.42 25 1438.55 47 1425.2 10 1290.25
28
10 29 3687.02 48 3209.13 27 2872.86 47 2014.53 30 1361.95 11 1273.6 26 1062.85 12 840.841 46 672.985 31 271.952
29
10 27 4029.43 30 3909.55 28 3739.93 47 3695.23 48 3135.87 26 2910.97 46 2229.55 12 1479.16 31 1430.26 11 1144.56
30
10 26 4029.86 29 3953.72 31 3811.12 46 3630.46 47 3105.96 27 2824.43 25 2657.89 45 2347.75 32 1459.11 12 1429.62
31
10 25 3882.21 30 3841.88 32 3808.5 45 3649.82 46 3000.67 26 2939.94 24 2409.93 44 2381.3 13 1467.59 29 1459.56
32
10 31 3826.5 24 3744.14 33 3613.24 44 3552.04 25 3004.6 45 2884.59 43 2393.34 23 2095.27 30 1478.6 14 1420.78
33
10 32 3618.11 23 3598.1 34 3530.53 43 3462.37 24 3091.53 44 2608.08 42 2426 22 1717.94 31 1407.65 25 1324.78
34
10 33 3523.37 42 3356.55 35 3210.34 22 3178.85 23 3079.03 43 2396.45 41 2386.86 24 1408.02 32 1301.34 21 1256.45
35
10 34 3187.88 41 3106.44 36 2866.04 22 2817.74 21 2654.87 40 2416.98 42 2137.81 23 1346.86 33 1150.33 16 1044.66
36
10 40 2910.7 35 2832.66 21 2689.96 37 2641.43 39 2349.53 20 2280.46 41 1787.97 22 1268.49 34 981.636 17 954.229
37
10 39 2678.55 36 2602.5 20 2558.22 38 1854.56 40 1611.7 19 1498.88 21 1419.51 35 902.641 18 826.803 17 680.253
38
10 39 2189.15 37 1834.05 40 824.669 36 771.589 19 622.648 20 590.632 21 190.621 41 157.673 35 155.716 18 134.943
39
10 40 2741.73 37 2690.66 36 2322.38 38 2228 20 1046.1 41 983.275 35 883.261 21 693.084 19 509.504 42 193.016
40
10 36 2918.14 41 2852.62 39 2782.6 35 2392.96 37 1641.45 21 1124.3 42 1056.48 34 877.946 38 853.944 20 788.701
41
10 35 3111.05 42 3049.71 40 2885.36 34 2371.02 36 1813.69 43 1164.71 22 1126.9 39 1011.26 21 906.536 33 903.238
42
10 34 3356.98 43 3183 41 3070.54 33 2421.77 35 2155.08 44 1278.41 23 1183.52 22 1147.07 40 1077.08 32 899.646
43
10 33 3461.24 44 3380.74 42 3188.7 34 2400.6 32 2399.09 45 1359.37 23 1314.08 41 1176.12 24 1159.62 31 901.556
44
10 32 3550.81 45 3510.16 43 3373.11 33 2602.33 31 2395.93 24 1410.43 46 1386.31 42 1279 25 1095.24 34 968.44
45
10 31 3650.09 46 3555.09 44 3491.15 32 2868.39 30 2373.59 25 1485.37 47 1405.28 43 1349.54 33 1104.77 26 1046.81
46
10 30 3635.64 47 3562.17 45 3524.17 31 2976.82 29 2264.04 26 1508.87 44 1367.41 48 1352.1 32 1211.24 25 1102.17
47
10 29 3705.31 46 3519.76 48 3450.48 30 3074.77 28 2054.63 27 1434.57 45 1377.34 31 1268.23 26 1223.83 25 471.111
48
10 47 3401.95 28 3224.84 29 3101.16 46 1317.1 30 1306.7 27 1235.07 26 537.731 31 291.919 45 276.869 11 258.856
31 changes: 31 additions & 0 deletions configs/lists/dtu_test_all.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
scan29
scan1
scan2
scan7
scan8
scan21
scan30
scan31
scan34
scan38
scan39
scan40
scan41
scan45
scan51
scan55
scan56
scan57
scan58
scan63
scan82
scan83
scan103
scan110
scan111
scan112
scan113
scan114
scan115
scan116
scan117
88 changes: 88 additions & 0 deletions configs/lists/dtu_train_all.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
scan3
scan4
scan5
scan6
scan9
scan10
scan11
scan12
scan13
scan14
scan15
scan16
scan17
scan18
scan19
scan20
scan22
scan23
scan24
scan28
scan32
scan33
scan35
scan36
scan37
scan42
scan43
scan44
scan46
scan47
scan48
scan49
scan50
scan52
scan53
scan59
scan60
scan61
scan62
scan64
scan65
scan66
scan67
scan68
scan69
scan70
scan71
scan72
scan74
scan75
scan76
scan77
scan84
scan85
scan86
scan87
scan88
scan89
scan90
scan91
scan92
scan93
scan94
scan95
scan96
scan97
scan98
scan99
scan100
scan101
scan102
scan104
scan105
scan106
scan107
scan108
scan109
scan118
scan119
scan120
scan121
scan122
scan123
scan124
scan125
scan126
scan127
scan128
Loading

0 comments on commit d3ce599

Please sign in to comment.