This repo is used for optimzation and visualization for MMVP dataset:

MMVP: A Multimodal MoCap Dataset with Vision and Pressure Sensors
He Zhang, Shenghao Ren, Haolei Yuan, Jianhui Zhao, Fan Li, Shuangpeng Sun, Zhenghao Liang, Tao Yu, Qiu Shen, Xun Cao

• [25/03/24]: Code for optimzation are released!

Demo for visulization released.

Instruction for optimization:

1. get essential files for optimizing here, then put it in code root.
2. Fill out this form to request authorization to use MMVP for non-commercial purposes. It may take one week to get reply. Or Contact Tao Yu at ([email protected]). More details about MMVP Dataset here.
3. We use RTM-pose for 2d keypoints detection and Cliff for pose initialization. Before run fitting, 2d keypoints should be processed and set as below:

images
└── 20230422
    ├── S01
    ├── ...
    └── S12
        ├── MoCap_20230422_145333
        ├── ...
        └── MoCap_20230422_150723
            ├── color
            ├── depth
            ├── depth_mask
            ├── insole
            ├── calibration.npy
            └── keypoints
                    ├── 000.npy
                    ├── ...  
                    └── 100.npy  

please note: the frame idx in keypoints/ should be consistent to those in insole/. 4. for one sequence, the results of RTM-pose should be saved under input/sub_ids/seq_name/. 5. Our optimization includes 3 stages: init_shape, init_pose, tracking. init_shape stage is applied for shape parameter initialization. init_pose stage is applied for pose estimation for the first frame in one sequence. tracking stage is applied for pose estimation for other frames in one sequence. Users should run optimization follow the order: init_shape, init_pose by changing the parameter fitting stage in configs/fit_smpl_rgbd.yaml. Stage tracking will run automatically if you choose init_pose in the yaml file. 6. For init_pose, we use Cliff to get pose initial value for depth alignment. The pose initial value will be saved under init_data_dir/. 7. Users could make optimization for MMVP Dataset with the command below:

sh run.sh

You could change frame info in configs/visualizing.yaml.
basdir is the root where you put MMVP Dataset.
dataset is fixed as 20230422.
sub_ids could be S01...S12
seq_name represents the seq under the sub_ids you select.
essential_root represents the path you put the essential files for optimizing.
init_data_dir represents the path you put the CLIFF result for pose initialization.

Noticing that frame_idx may not cover all frames and could be found in annotations/smpl_pose.

If you find our work useful in your research, please cite our paper MMVP:

@inproceedings{Zhang2024MMVP,
    title={MMVP: A Multimodal MoCap Dataset with Vision and Pressure Sensors},
    author={He Zhang, Shenghao Ren, Haolei Yuan, Jianhui Zhao, Fan Li, Shuangpeng Sun, Zhenghao Liang, Tao Yu, Qiu Shen, Xun Cao},
    journal={CVPR},
    year={2024}
}

• Tao Yu ([email protected])