Skip to content

Latest commit

 

History

History

mvpose

README.md

MVPose (Single frame)

Introduction

We provide the config files for MVPose (Single frame): Fast and robust multi-person 3d pose estimation from multiple views.

Official Implementation

@inproceedings{dong2019fast,
  title={Fast and robust multi-person 3d pose estimation from multiple views},
  author={Dong, Junting and Jiang, Wen and Huang, Qixing and Bao, Hujun and Zhou, Xiaowei},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={7792--7801},
  year={2019}
}

Prepare models and datasets

  • Prepare models:
sh scripts/download_weight.sh

You could find perception models in weight file.

  • Download body model

Please refer to Body Model Preparation.

  • Prepare the datasets:

You could download Shelf, Campus or CMU-Panoptic datasets, and convert original dataset to our unified meta-data. Considering that it takes long to run a converter, we have done it for you. Please download compressed zip file for converted meta-data from here, and place meta-data under ROOT/xrmocap_data/DATASET.

The final file structure would be like:

xrmocap
├── xrmocap
├── docs
├── tools
├── configs
├── weight
|   ├── mvpose
|   |   └── resnet50_reid_camstyle-98d61e41_20220921.pth
|   ├── ...
|   └── tracktor_reid_r50_iter25245-a452f51f.pth
└── xrmocap_data
    ├── body_models
    |   ├── gmm_08.pkl
    |   ├── smpl_mean_params.npz
    |   └── smpl
    |       ├── SMPL_FEMALE.pkl
    |       ├── SMPL_MALE.pkl
    |       └── SMPL_NEUTRAL.pkl
    |
    ├── CampusSeq1
    ├── Shelf
    |   ├── Camera0
    |   ├── ...
    |   ├── Camera4
    |   ├── xrmocap_meta_testset_fasterrcnn
    |   └── xrmocap_meta_testset
    └── Panoptic
        ├── xrmocap_meta_ian5
        |   ├── hd_00_03
        |   ├── ...
        |   ├── hd_00_23
        |   ├── camera_parameters
        |   ├── keypoints3d_GT.npz
        |   └── perception_2d.npz
        ├── xrmocap_meta_pizza1
        ├── xrmocap_meta_band4
        └── xrmocap_meta_haggling1

You can download just one dataset of Shelf, Campus and CMU-Panoptic.

Results

We evaluate MVPose (Single frame) on 3 popular benchmarks, report the Percentage of Correct Parts (PCP), Mean Per Joint Position Error (MPJPE), MPJPE with Procrustes Analysis (PA) as PA-MPJPE and Probability of Correct Keypoint (PCK) on Campus, Shelf and CMU Panoptic dataset.

To be more fair in evaluation, some modifications are made compared to the evaluations in the original work. For PCP, instead of by body parts, we evaluate by the limbs defined in selected_limbs_names and additional_limbs_names. We remove the root alignment in MPJPE and provide PA-MPJPE instead. Thresholds for outliers are removed as well.

You can find the recommended configs in configs/mvpose/*/eval_keypoints3d.py, where __bbox_thr__ is the threshold of bbox2d, you can set a high threshold to ignore incorrect 2D perception data, and we recommen setting it to 0.8~0.9. n_cam_min is the amount of views required for triangulation, which defaults to 2.

Campus

The 2D perception data we use is generated by fasterrcnn, and you can download it from here. What's more, we set __bbox_thr__=0.9 and n_cam_min=2.

Config PCP MPJPE(mm) PA-MPJPE(mm) PCK@50 PCK@100 Download
eval_keypoints3d.py 92.87 84.54 62.87 42.76 86.62 log

The PCP for each actor is as follows:

Actor 0 Actor 1 Actor 2 Average
93.52 86.88 98.20 92.87

Shelf

The 2D perception data we use is generated by fasterrcnn, and you can download it from here. What's more, we set __bbox_thr__=0.9 and n_cam_min=3.

Config PCP MPJPE(mm) PA-MPJPE(mm) PCK@50 PCK@100 Download
eval_keypoints3d.py 95.82 55.30 43.22 69.11 98.01 log

The PCP for each actor is as follows:

Actor 0 Actor 1 Actor 2 Average
97.97 91.89 97.58 95.82

CMU Panoptic

The 2D perception data we use is generated by mmpose, and you can download it from here. The selection principle of the camera is to cover as much information as possible about the human body, so we selected cameras 3, 6, 12, 13 and 23.

The CMU Panoptic dataset contains four sequences that share the same config file. You can find more details in config files. For different sequences, you need to change the __meta_path__. In addition, we set __bbox_thr__=0.85 and n_cam_min=2.

  • 160906_band4
Config PCP MPJPE(mm) PA-MPJPE(mm) PCK@50 PCK@100 Download
eval_keypoints3d.py 93.54 60.06 57.42 64.52 87.60 log
  • 160906_ian5
Config PCP MPJPE(mm) PA-MPJPE(mm) PCK@50 PCK@100 Download
eval_keypoints3d.py 83.30 105.63 86.35 76.86 87.11 log
  • 160906_pizza1
Config PCP MPJPE(mm) PA-MPJPE(mm) PCK@50 PCK@100 Download
eval_keypoints3d.py 91.06 81.59 50.93 71.10 90.49 log
  • 160422_haggling1
Config PCP MPJPE(mm) PA-MPJPE(mm) PCK@50 PCK@100 Download
eval_keypoints3d.py 93.49 77.38 56.69 79.30 92.87 log