This is the reference implementation for the MLPerf automotive 3D detection benchmark. The reference uses Pytorch as a backend. Additionally we provide an implementation using ONNX.
model | accuracy | dataset | model source | precision | notes |
---|---|---|---|---|---|
PointPainting | 0.5425 mAP | Waymo Open Dataset | https://github.com/rod409/pp | fp32 | Single-Stream 99.9 percentile |
Please see the new docs site for an automated way to run this benchmark across different available implementations and do an end-to-end submission with or without docker.
You can also do pip install mlc-scripts
and then use mlcr
commands for downloading the model and datasets using the commands given in the later sections.
Note
By default, the waymo dataset is downloaded from the mlcommons official drive. One has to accept the MLCommons Waymo Open Dataset EULA to access the dataset files.
mlcr get,ml-model,pointpainting --outdirname=<path_to_download> -j
Note
By default, the waymo dataset is downloaded from the mlcommons official drive. One has to accept the MLCommons Waymo Open Dataset EULA to access the dataset files.
mlcr get,dataset,waymo --outdirname=<path_to_download> -j
Contact MLCommons support for accessing the Waymo Open Dataset along with the model checkpoints for the reference implementation. You will need to accept a license agreement and will be given directions to download the data with rclone. You will need to place the kitti_format folder under a directory named waymo. There are four total checkpoints 2 for pytorch and 2 for onnx.
After downloading, the structure of the data should look like below:
├── waymo
│ ├── best_deeplabv3plus_resnet50_waymo_os16.pth
│ ├── deeplabv3+.onnx
│ ├── kitti_format
│ │ ├── ImageSets
│ │ ├── painted_waymo_infos_test.pkl
│ │ ├── painted_waymo_infos_train.pkl
│ │ ├── painted_waymo_infos_trainval.pkl
│ │ ├── painted_waymo_infos_val.pkl
│ │ ├── testing
│ │ ├── testing_3d_camera_only_detection
│ │ ├── training
│ │ ├── waymo_infos_test.pkl
│ │ ├── waymo_infos_train.pkl
│ │ ├── waymo_infos_trainval.pkl
│ │ └── waymo_infos_val.pkl
│ ├── pp_ep36.pth
│ ├── pp.onnx
Within the training folder is the validation data used for the dataset. Extract all the compressed files in the training folder.
cd <your path to waymo>/waymo/kitti_format/training
for f in *.tar.gz; do tar -xzvf "$f"; done
Build the container and mount the inference repo and Waymo dataset directory. Build the container and mount the inference repo and Waymo dataset directory.
docker build -t auto_inference -f dockerfile.gpu .
docker run --gpus=all -it -v <directory to inference repo>/inference/:/inference -v <path to waymo dataset>/waymo:/waymo --rm auto_inference
cd /inference/automotive/3d-object-detection
python main.py --dataset waymo --dataset-path /waymo/kitti_format/ --lidar-path <checkpoint_path>/pp_ep36.pth --segmentor-path <checkpoint_path>/best_deeplabv3plus_resnet50_waymo_os16.pth
python main.py --dataset waymo --dataset-path /waymo/kitti_format/ --lidar-path <checkpoint_path>/pp.onnx --segmentor-path <checkpoint_path>/deeplabv3+.onnx --mlperf_conf /inference/mlperf.conf --backend onnx
Note
The minimum number of queries needed to be run is 6636. To get the best latency, users are encouraged to run a larger number of samples.
Add the accuracy flag
cd /inference/automotive/3d-object-detection
python main.py --dataset waymo --dataset-path /waymo/kitti_format/ --lidar-path <checkpoint_path>/pp_ep36.pth --segmentor-path <checkpoint_path>/best_deeplabv3plus_resnet50_waymo_os16.pth --accuracy
mlcr process,mlperf,accuracy,_waymo --result_dir=<Path to directory where files are generated after the benchmark run>
Please click here to view the Python script for evaluating accuracy for the Waymo dataset.
python accuracy_waymo.py --mlperf-accuracy-file <path to accuracy file>/mlperf_log_accuracy.json --waymo-dir /waymo/kitti_format/