easyvolcap: add basic structure for core design docs

docs/design/camera.md

@@ -1,120 +1,19 @@
-# Using EasyVolcap For Camera Parameters Optimization (Finetuning)
+# Camera Parameters
 
+## Camera Pose Structure
 
-Test scripts:
+### Disk Storage
 
-```shell
-# Train static ngpt with mask
-evc -c configs/exps/ngpt/ngpt_my_313_static_mask.yaml runner_cfg.eval_ep=1 runner_cfg.resume=False exp_name=ngpt_my_313_static_mask_$(git rev-parse --short=6 HEAD)
+### Memory Storage
 
-# Train static mhet without mask
-evc -c configs/exps/mhet/mhet_sport1_static.yaml runner_cfg.eval_ep=1 runner_cfg.resume=False exp_name=mhet_sport1_static_$(git rev-parse --short=6 HEAD)
+### Viewer Camera
 
-# Train static ngpt without mask
-evc -c configs/exps/ngpt/ngpt_sport1_static.yaml runner_cfg.eval_ep=1 runner_cfg.resume=False exp_name=ngpt_sport1_static_$(git rev-parse --short=6 HEAD)
+### Other Formats
 
-# Render static ngpt with mask using the default dataset
-evc -t test -c configs/exps/ngpt/ngpt_my_313_static_mask.yaml exp_name=ngpt_my_313_static_mask_$(git rev-parse --short=6 HEAD)
+### Camera Alignments
 
-# Render static ngpt with mask using oribt cameras
-evc -t test -c configs/exps/ngpt/ngpt_my_313_static_mask.yaml exp_name=ngpt_my_313_static_mask_$(git rev-parse --short=6 HEAD) configs=configs/specs/orbit.yaml
+## Using ***EasyVolcap*** for Camera Pose Finetuning
 
-# Render IBR from enerf
-evc -t test -c configs/exps/enerf/enerf_my_313.yaml runner_cfg.eval_ep=1 exp_name=enerf_my_313_$(git rev-parse --short=6 HEAD) configs=configs/specs/orbit.yaml,configs/specs/ibr.yaml val_dataloader_cfg.dataset_cfg.frame_sample=0,1,1
-```
+## Optimizable Cameras
 
-Some examples:
-
-```shell
-evc -c configs/exps/mhet/mhet_basketball_static.yaml runner_cfg.eval_ep=1 runner_cfg.resume=False # train mhe on first frame of enerf_outdoor dataset
-evc -t test -c configs/exps/mhet/mhet_basketball_static.yaml configs=configs/specs/orbit.yaml runner_cfg.visualizer_cfg.types=RENDER,DEPTH, val_dataloader_cfg.dataset_cfg.render_size=1024,1024 # render basketball from NHR in FHD
-
-evc -c configs/exps/mhet/mhet_actor1_static.yaml runner_cfg.eval_ep=1 runner_cfg.resume=False # train mhe on first frame of enerf_outdoor dataset
-evc -t test -c configs/exps/mhet/mhet_actor1_static.yaml configs=configs/specs/interp.yaml runner_cfg.visualizer_cfg.types=RENDER,DEPTH, val_dataloader_cfg.dataset_cfg.render_size=1024,1024 # render actor1 from enerf_outdoor in FHD
-
-evc -c configs/exps/mhet/mhet_ip412_blurry.yaml runner_cfg.eval_ep=1 runner_cfg.resume=False # train mhe on 412
-evc -t test -c configs/exps/mhet/mhet_ip412_blurry.yaml configs=configs/specs/orbit.yaml runner_cfg.visualizer_cfg.types=RENDER,DEPTH, val_dataloader_cfg.dataset_cfg.render_size=1024,1024 # render mhe on iphone_412 scene in FHD
-
-evc -c configs/exps/enerf/enerf_actor1.yaml runner_cfg.eval_ep=1 # train enerf, render images every epoch
-evc -c configs/exps/enerf/enerf_actor1_static.yaml runner_cfg.eval_ep=1 runner_cfg.resume=False # train enerf on one frame, render images every epoch, start from scratch
-evc-prof -t test -c configs/exps/enerf/enerf_actor1.yaml # perform profling on testing of enerf
-
-# GUI without any control
-evc -t gui  -c configs/exps/enerf/enerf_actor1.yaml exp_name=enerf_actor1_caad45 val_dataloader_cfg.dataset_cfg.ratio=0.8 configs=configs/specs/cubic.yaml,configs/specs/ibr.yaml val_dataloader_cfg.dataset_cfg.view_sample=7,6,5,2,15,13,17,8,16,14,12,11,10,1,9,0,3,4 runner_cfg.visualizer_cfg.types=RENDER,DEPTH val_dataloader_cfg.dataset_cfg.n_render_views=600 val_dataloader_cfg.dataset_cfg.frame_sample=0,600,1 runner_cfg.visualizer_cfg.types="[]" viewer_cfg.window_size=800,1184
-
-evc -t gui -c configs/exps/enerf/enerf_actor1.yaml exp_name=enerf_actor1_caad45 val_dataloader_cfg.dataset_cfg.ratio=0.8 configs=configs/specs/cubic.yaml,configs/specs/ibr.yaml val_dataloader_cfg.dataset_cfg.view_sample=7,6,5,2,15,13,17,8,16,14,12,11,10,1,9,0,3,4 runner_cfg.visualizer_cfg.types=RENDER,DEPTH val_dataloader_cfg.dataset_cfg.n_render_views=600 val_dataloader_cfg.dataset_cfg.frame_sample=0,1,1 runner_cfg.visualizer_cfg.types="[]" viewer_cfg.window_size=800,1184 # for faster testing
-
-evc -t gui -c configs/exps/enerf/enerf_actor1.yaml exp_name=enerf_actor1_caad45 val_dataloader_cfg.dataset_cfg.ratio=0.4 configs=configs/specs/cubic.yaml,configs/specs/ibr.yaml val_dataloader_cfg.dataset_cfg.view_sample=7,6,5,2,15,13,17,8,16,14,12,11,10,1,9,0,3,4 runner_cfg.visualizer_cfg.types=RENDER,DEPTH val_dataloader_cfg.dataset_cfg.n_render_views=600 val_dataloader_cfg.batch_sampler_cfg.n_srcs_list=3, val_dataloader_cfg.dataset_cfg.frame_sample=0,1,1 runner_cfg.visualizer_cfg.types="[]" viewer_cfg.window_size=400,592 # for faster rendering of small images
-
-# GUI with some control, slow
-evc -t gui -c configs/exps/enerf/enerf_actor1.yaml exp_name=enerf_actor1_ce1fdd val_dataloader_cfg.dataset_cfg.ratio=0.8 val_dataloader_cfg.dataset_cfg.frame_sample=0,1,1 runner_cfg.visualizer_cfg.types="[]"
-
-# Rendering test views for ENeRF
-evc -t test -c configs/exps/enerf/enerf_actor1.yaml exp_name=enerf_actor1_$(git rev-parse --short=6 HEAD) val_dataloader_cfg.dataset_cfg.ratio=0.8 configs=configs/specs/cubic.yaml,configs/specs/ibr.yaml val_dataloader_cfg.dataset_cfg.view_sample=7,6,5,2,15,13,17,8,16,14,12,11,10,1,9,0,3,4 runner_cfg.visualizer_cfg.types=RENDER,DEPTH val_dataloader_cfg.dataset_cfg.n_render_views=600 val_dataloader_cfg.dataset_cfg.frame_sample=0,600,1 # for faster testing
-
-# The commands before are obsolete now
-evc -t gui -c configs/exps/enerf/enerf_my_313.yaml exp_name=enerf_my_313_0d7f34 val_dataloader_cfg.dataset_cfg.frame_sample=0,1,1 viewer_cfg.window_size=512,512 # for faster rendering of small images
-
-# GUI for ngp
-evc -t gui -c configs/exps/ngpt/ngpt_my_313_static_mask.yaml exp_name=ngpt_my_313_static_mask_a68c25 val_dataloader_cfg.dataset_cfg.ratio=0.5 val_dataloader_cfg.dataset_cfg.frame_sample=0,1,1 runner_cfg.visualizer_cfg.types="[]" viewer_cfg.window_size=256,256 model_cfg.renderer_cfg.bg_brightness=0.0 # for faster rendering of small images
-```
-
-We use similar strategies for optimizing camera parameters as in Instant-NGP.
-For now, only extrinsic parameters are taken care of.
-A script for extracting the optimized camera can be found at: `scripts/tools/extract_optimized_cameras.py`
-
-```shell
-
-# Optimized actor1_4
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_actor1_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_actor2_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_actor3_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_actor1_4_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_actor2_3_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_actor5_6_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_actor1_complex_static.yaml
-
-# Optimized RenBody
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0007_01_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0008_01_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0008_03_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0008_05_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0011_11_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0012_01_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0012_03_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0012_11_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0013_01_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0013_03_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0013_06_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0013_09_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0013_11_static.yaml
-# python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0014_09_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0019_08_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0019_09_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0021_01_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0021_02_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0021_03_static.yaml
-# python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0021_08_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0022_04_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0022_07_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0023_01_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0023_04_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_0023_06_static.yaml
-
-# Optimized ZJU-MoCap
-# python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_my_313_static.yaml
-# python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_my_315_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_my_377_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_my_386_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_my_387_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_my_390_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_my_392_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_my_393_static.yaml
-
-# Optimized MobileStage
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_dance3_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_xuzhen_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_purple_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_black_static.yaml
-python scripts/tools/extract_optimized_cameras.py -- -c configs/exps/l3mhet/l3mhet_white_static.yaml
-```
+## Extract Optimized Cameras

docs/design/dataset.md

@@ -0,0 +1,17 @@
+# Multi-View Dataset Management System
+
+NOTE: 
+
+## Core Features
+
+### Parallel Loading
+
+### Efficient Memory Usage
+
+### Fast I/O
+
+#### Precomputation
+
+#### Decoding & Sampling
+
+### Sharding

docs/design/install.md

@@ -86,6 +86,8 @@ export CUDA_HOME="/usr/local/cuda"
 export CUDA_DEVICE_ORDER=PCI_BUS_ID # OPTIONAL: defaults to capability order, might be different for GL and CUDA
 ```
 
+Let's go through the compilation process with the `tinycudann` package.
+
 To retain the compiled objects without starting over in case anything fails, we recommend first cloning `tinycudann` then perform the compilation and installation manually:
 
 ```shell
@@ -115,4 +117,14 @@ If you still encounter `cannot find -lcuda: No such file or directory` error aft
 ln -s /usr/lib/x86_64-linux-gnu/libcuda.so ~/miniconda3/envs/easyvolcap/lib/libcuda.so
 ```
 
-and then install `tiny-cuda-nn` with: `pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch`.
+and then install `tiny-cuda-nn` with: `pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch`.
+
+Sometimes (`diff-gaussian-rasterization`), on some systems, the import order of CUDA-compiled extension and PyTorch matters.
+On some of our system, if you import PyTorch before the CUDA-compiled extension, it will fail to find some Python symbols.
+Try changing the orders if that happens.
+
+```shell
+python -c "from diff_gaussian_rasterization import _C" # works
+python -c "import torch; from diff_gaussian_rasterization import _C" # might fail
+python -c "from diff_gaussian_rasterization import _C; import torch" # should work
+```

docs/design/logging.md

@@ -1,13 +1,13 @@
-### Logging & Printing System
-
-We'd like to make this repository a knowledge center of generations of researchers at ZJU3DV.
-
-- [ ] TODO: Add example of implementing new functionalities
-  - [ ] Like a new type of visualization (creating reusable parts of the framework)
-  - [ ] Or a whole new network (lots of components are reused)
-  - [ ] Or a whole new type of model (only dataloading and visualizations are reused)
-- [ ] TODO: Add docs about the CUDA semantics and pytorch intricacies
-- [ ] TODO: Add docs about the dataloading and pytorch intricacies
-- [ ] TODO: Add readme about the design and usage of logging system
-- [ ] TODO: Add readme about timer system and torch_tb_profiler tools
-- [ ] TODO: Add docs and examples about distributed training system 
+# Logging System
+
+## Core Features
+
+### Colored & Unified Logging & Printing
+
+### Loss Printing
+
+### Progress Bar
+
+### Exception Logging & Debugging
+
+## Using ***EasyVolcap***'s Logging Elsewhere

docs/design/main.md

@@ -2,8 +2,8 @@
 
 ## Core Modules
 
-Here a directory tree of the core modules of EasyVolcap.
-I commended on the directory structure of the `easyvolcap` folder, which should give you a general sense about our module organization.
+Here's a directory tree of the core modules of EasyVolcap.
+I commended on the directory structure of the `easyvolcap` folder, which should give you a general sense about our codebase's organization.
 
 ```shell
 easyvolcap

docs/design/model.md

@@ -0,0 +1,25 @@
+# Modular and Extensible Pipeline
+
+## [Ray Sampling](../../easyvolcap/models/volumetric_video_model.py)
+
+## [Camera Finetuning](../../easyvolcap/models/cameras)
+
+## Network Regression
+
+### Embedders
+
+### Regressors
+
+### NeRF-Like Pipeline
+
+#### [Point Sampling](../../easyvolcap/models/samplers/importance_sampler.py)
+
+#### [Field Regression](../../easyvolcap/models/networks/volumetric_video_network.py)
+
+#### [Volume Rendering](../../easyvolcap/models/renderers/volume_rendere.py)
+
+### Custom Pipeline
+
+#### [Model Forwarding](../../easyvolcap/models/samplers/gaussiant_sampler.py)
+
+## [Loss Computation](../../easyvolcap/models/supervisors/volumetric_video_supervisor.py)

docs/design/runner.md

@@ -0,0 +1,9 @@
+# Training & Evaluation Loop
+
+## Training
+
+## Recording
+
+## Evaluation
+
+## Model Saving

docs/design/viewer.md

@@ -0,0 +1,9 @@
+# Real-Time Visualization
+
+## ImGUI-Based Viewer
+
+### CUDA-GL Interop
+
+## Unity Socket Viewer
+
+## WebSocket Viewer