Dream2DGS

Dream2DGS is a tool for text/image to 3D generation based on 2D Gaussian Splatting and DreamGaussian.

comparison.mp4

Update log

Click to expand

• [2024/05/13]: Fixed segmentation bug.
• [2024/05/07]: Released the initial code.

Gallery

Here are some rendered gs results from Dream2DGS. From left to right are rendered rgb, rendered normal, normal from depth.

out.mp4

a.campfire.mp4

an.ice.cream.mp4

a.potted.cactus.plant.mp4

Install

conda create -n dream2dgs python=3.9 

# torch2.3.0+cu12.1 
pip3 install torch torchvision torchaudio

# xformers  
pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu121
 
pip install -r requirements.txt
  
# mvdream 
pip install git+https://github.com/bytedance/MVDream

# nvdiffrast
pip install git+https://github.com/NVlabs/nvdiffrast/
 
pip install -e submodules/diff-surfel-rasterization
pip install -e submodules/simple-knn
pip install -e submodules/generative_models

# download pretriained models  
mkdir checkpoints && cd checkpoints 
wget https://huggingface.co/stabilityai/sv3d/blob/main/sv3d_p.safetensors  # sv3d_p.safetensors
gdown '1wNxVO4vVbDEMEpnAi_jwQObf2MFodcBR&confirm=t' # omnidata_dpt_normal_v2.ckpt
cd ..

Usage

# image-to-3d (sv3d)  
python main.py --config configs/image_sv3d.yaml input=data/zelda_rgba.png save_path=zelda

# gui
python main.py --config configs/image_sv3d.yaml input=data/zelda_rgba.png save_path=zelda gui=True

# text-to-3d (sd2.1 + sv3d)  
python main.py --config configs/text_mv.yaml prompt="an ice cream" save_path="an ice cream" 

# extract mesh  
python convert.py --model_path logs/zelda/zelda_gs.ply --optimize_texture 
 
# using normal supervision set dpt=True 
python main.py --config configs/image_sv3d.yaml input=data/zelda_rgba.png save_path=zelda dpt=True 

Acknowledgement

Special thanks to the projects and their contributors:

• 2D Gaussian Splatting

• DreamGaussian

• SV3D

Citation

If you find Dream2DGS helpful, please consider citing it:

@misc{Dream2DGS,
  author = {Tingting Liao},
  title = {Dream2DGS: Text/Image to 3D Generation Tool},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/TingtingLiao/dream2dgs}}
}