This is the official implementation of our CVPR2024 paper, HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions.

Here is our presentation video: YouTube

We release following key parts, but there are still some issues need to be fixed caused by transfering code from server to local environment.

• Novel condition creator
• Hand-object image synthesis
• Boosted hand-mesh reconstruction baselines
• Fix small issues

• Environment

  conda create -n handbooster python=3.8
  conda activate handbooster
  
• Requirements

  pip install -r requirements.txt
  

We evaluate different models on the DexYCB and HO3D datasets. The pre-processed ground truths are from HandOccNet. Please follow its instruction to prepare the data and ground truths like this,

|-- data  
|   |-- HO3D
|   |   |-- train
|   |   |   |-- ABF10
|   |   |   |-- ......
|   |   |-- evaluation
|   |   |-- annotations
|   |   |   |-- HO3D_train_data.json
|   |   |   |-- HO3D_evaluation_data.json
|   |-- DEX_YCB
|   |   |-- 20200709-subject-01
|   |   |-- ......
|   |   |-- annotations
|   |   |   |--DEX_YCB_s0_train_data.json
|   |   |   |--DEX_YCB_s0_test_data.json

Also, you need to download the MANO models (MANO_LEFT.pkl and MANO_RIGHT.pkl) from its official website and put it into NovelConditionCreator/DexGraspNet/grasp_generation/mano, NovelConditionCreator/dexycb/manopth/mano/models, NovelConditionCreator/ho3d/manopth/mano/models, HandObjectImageSynthesizer/hand_recon/common/utils/manopth/mano/models, HandReconstruction/common/utils/manopth/mano/models. You can change the model path if you don't want to copy them several times.

We follow the DexGraspNet to similuate grasping poses. We remove the table penalty. Check NovelConditionCreator/DexGraspNet/grasp_generation/main.py for the pose generation and NovelConditionCreator/DexGraspNet/grasp_generation/pose_validation.py for the pose validation.

We preprocess the simulated grasping poses to the required image condition in the next step. Check NovelConditionCreator/dexycb/dexycb_preprocess.py and NovelConditionCreator/ho3d/ho3d_preprocess.py for the DexYCB and HO3D datasets, respectively.

We use the conditional diffsuion model to generate RGB images. We provide our model weights,

Here are the training and inference scripts,

# Change the dataset_name to one of [dexycb_s0, dexycb_s1, ho3d]
# Train
accelerate launch HandObjectImageSynthesizer/train.py --model_dir experiment/{dataset_name}
# Inference
accelerate launch HandObjectImageSynthesizer/inference_{dataset_name}.py --model_dir experiment/{dataset_name}--resume experiment/{dataset_name}/model/checkpoint.pt

Given the generated data, we mixed it with the original data to train different baselines. Cause the generated data is too large, we provide the boosted model weights,

Here is the evaluation script,

# Change the dataset_name to one of [dexycb_s0, dexycb_s1, ho3d] and the baseline_name to one of [handoccnet, mobrecon, h2onet]
accelerate launch HandReconstruction/test.py --model_dir experiment/{dataset_name}/{baseline_name} --resume experiment/{dataset_name}/{baseline_name}/test_model_best.pth

@InProceedings{Xu_2024_CVPR,
    author    = {Xu, Hao and Li, Haipeng and Wang, Yinqiao and Liu, Shuaicheng and Fu, Chi-Wing},
    title     = {HandBooster: Boosting 3D Hand-Mesh Reconstruction by Conditional Synthesis and Sampling of Hand-Object Interactions},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {10159-10169}
}

In this project we use (parts of) the official implementations of the following works:

• DexGraspNet (Grasping pose simulation)
• DDPM (Image generation)
• MobRecon (Baseline)
• HandOccNet (Baseline)
• H2ONet (Baseline)

We thank the respective authors for open sourcing their methods.