- Setup conda environment.
conda create -n gs-dynamics python=3.10
conda activate gs-dynamics
conda install pytorch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 pytorch-cuda=12.4 -c pytorch -c nvidia
pip install -r requirements.txt
conda install -c dglteam/label/th24_cu124 dgl # install dgl with cuda 12.4
- Install Gaussian Rasterization module.
mkdir third-party
cd third-party
git clone [email protected]:JonathonLuiten/diff-gaussian-rasterization-w-depth.git
cd diff-gaussian-rasterization-w-depth
python setup.py install
cd ..
- Install Grounded SAM.
pip install git+https://github.com/facebookresearch/segment-anything.git
git clone [email protected]:IDEA-Research/GroundingDINO.git
cd GroundingDINO
python setup.py install
cd ../../weights
gdown 1pc195KITtCCLteHkxD7mr1w-cG8XAvTs # download DINO+SAM weights
gdown 1X-Et3-5TdSWuEUfq9gs_Ng-V-hypmLdB
gdown 1HR3O-rMv9qJoTrG6GNxi-4ZVREQzZ1Rf
We prepared an interactive demo without requiring a robot setup. Try it now!
- Download checkpoints here. Unzip and put it into the
logfolder like this:
/xxx/gs-dynamics/log/rope/checkpoints/latest.pth
/xxx/gs-dynamics/log/sloth/checkpoints/latest.pth
...
- Run the demo!
cd src
python demo.py
- Calibrate the cameras using the ChArUco calibration board.
cd src/real_world
python calibrate.py --calibrate # calibrate realsense cameras
- Put object in workspace, run an interactive interface to simulate interaction with the object with GS-Dynamics.
python gs_sim_real_gradio.py --config config/rope.py
Data should be stored in the folder {base_path}/data (you can define your own base path). Download our data here, unzip, and put it in the data folder like this:
{base_path}/data/episodes_rope
{base_path}/data/episodes_dog
...
- Prepare data to obtain the mask, initial point cloud, and metadata for the object.
cd src/tracking
python utils/obtain_mask.py --text_prompt "rope" --data_path $data_path # obtain object mask
python utils/merge_masks.py --data_path $data_path # obtain the foreground images of the object
python utils/init_pcd.py --data_path $data_path # obtain the initial point cloud
python utils/metadata.py --data_path $data_path # obtain metadata for training
- Run the optimization.
python train_gs.py --sequence $episode --exp_name $exp_name --weight_im $weight_im --weight_rigid $weight_rigid --weight_seg $weight_seg --weight_soft_col_cons $weight_soft_col_cons --weight_bg $weight_bg --weight_iso $weight_iso --weight_rot $weight_rot --num_knn $num_knn --metadata_path $metadata_path --init_pt_cld_path=$init_pt_cld_path --scale_scene_radius=$scale_scene_radius
We provide a short description of the data captured from real world and the different configurations for various objects in assets/datasets.md.
- Prepare the data to parse unit actions, saved in
{base_path}/preprocessed. We useconfig/rope.yamlas an example.
cd src
python preprocess.py --config config/rope.yaml # preprocesses training data.
- Train the dynamics model.
python train.py --config config/rope.yaml
- Evaluate model prediction.
cd src
python predict.py --config config/rope.yaml --epoch latest # evaluation
We thank the authors of the following projects for making their code open source:
@inproceedings{zhang2024dynamics,
title={Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling},
author={Zhang, Mingtong and Zhang, Kaifeng and Li, Yunzhu},
booktitle={8th Annual Conference on Robot Learning},
year={2024}
}