Please see the new docs site for an automated way to run this benchmark across different available implementations and do an end-to-end submission with or without docker.
You can also do pip install mlc-scripts
and then use mlcr
commands for downloading the model and datasets using the commands given in the later sections.
model | accuracy | dataset | model source | precision | notes |
---|---|---|---|---|---|
Stable Diffusion XL 1.0 | - | Coco2014 | Hugging Face | fp32 | NCHW |
Data | Description |
---|---|
Coco-2014 | We use a subset of 5000 images and captions of the coco 2014 validation dataset, so that there is exaclty one caption per image. The model takes as input the caption of the image and generates an image from it. The original images and the generated images are used to compute FID score. The caption and the generated images are used to compute the CLIP score. We provide a script to automatically download the dataset |
Coco-2014 (calibration) | We use a subset of 500 captions and images of the coco 2014 training dataset, so that there is exaclty one caption per image. The subset was generated using this script. We provide the caption ids and a script to download them. |
Set the following helper variables
export ROOT=$PWD/inference
export SD_FOLDER=$PWD/inference/text_to_image
export LOADGEN_FOLDER=$PWD/inference/loadgen
export MODEL_PATH=$PWD/inference/text_to_image/model/
git clone --recurse-submodules https://github.com/mlcommons/inference --depth 1
Install requirements:
cd $SD_FOLDER
pip install -r requirements.txt
Install loadgen:
cd $LOADGEN_FOLDER
CFLAGS="-std=c++14" python setup.py install
We host two checkpoints (fp32 and fp16) that are a snapshot of the Hugging Face pipeline at the time of the release of the benchmark. Download them and move them to your model path.
The following MLCommons MLC commands can be used to programmatically download the model checkpoints.
mlcr get,ml-model,sdxl,_fp16,_rclone --outdirname=$MODEL_PATH
mlcr get,ml-model,sdxl,_fp32,_rclone --outdirname-$MODEL_PATH
The above command automatically runs a set of Rclone commands to download the data from a Cloudflare R2 bucket. However, if you'd like to run the Rclone commands manually, you can do so as follows:
To run Rclone on Windows, you can download the executable here. To install Rclone on Linux/macOS/BSD systems, run:
sudo -v ; curl https://rclone.org/install.sh | sudo bash
Once Rclone is installed, run the following command to authenticate with the bucket:
rclone config create mlc-inference s3 provider=Cloudflare access_key_id=f65ba5eef400db161ea49967de89f47b secret_access_key=fbea333914c292b854f14d3fe232bad6c5407bf0ab1bebf78833c2b359bdfd2b endpoint=https://c2686074cb2caf5cbaf6d134bdba8b47.r2.cloudflarestorage.com
You can then navigate in the terminal to your desired download directory and run the following commands to download the checkpoints:
cd $MODEL_PATH
fp32
rclone copy mlc-inference:mlcommons-inference-wg-public/stable_diffusion_fp32 $MODEL_PATH -P
fp16
rclone copy mlc-inference:mlcommons-inference-wg-public/stable_diffusion_fp16 $MODEL_PATH -P
The following MLCommons MLC commands can be used to programmatically download the validation dataset.
mlcr get,dataset,coco2014,_validation,_full --outdirname=coco2014
For debugging you can download only a part of all the images in the dataset
mlcr get,dataset,coco2014,_validation,_size.50 --outdirname=coco2014
cd $SD_FOLDER/tools
./download-coco-2014.sh -n <number_of_workers>
For debugging you can download only a part of all the images in the dataset
cd $SD_FOLDER/tools
./download-coco-2014.sh -m <max_number_of_images>
If the file captions.tsv can be found in the script, it will be used to download the target dataset subset, otherwise it will be generated. We recommend you to have this file for consistency.
The following MLCommons MLC commands can be used to programmatically download the calibration dataset.
mlcr get,dataset,coco2014,_calibration --outdirname=coco2014
We provide a script to download the calibration captions and images. To download only the captions:
cd $SD_FOLDER/tools
./download-coco-2014-calibration.sh -n <number_of_workers>
To download both the captions and images:
cd $SD_FOLDER/tools
./download-coco-2014-calibration.sh -i -n <number_of_workers>
# Go to the benchmark folder
cd $SD_FOLDER
# Run the benchmark
python3 main.py --dataset "coco-1024" --dataset-path coco2014 --profile stable-diffusion-xl-pytorch --model-path model/ [--dtype <fp32, fp16 or bf16>] [--device <cuda or cpu>] [--time <time>] [--scenario <SingleStream, MultiStream, Server or Offline>]
# Go to the benchmark folder
cd $SD_FOLDER
# Build the container
docker build . -t sd_mlperf_inference
# Run the container
docker run --rm -it --gpus=all -v $SD_FOLDER:/workspace sd_mlperf_inference bash
Inside the container run the following:
python3 main.py --dataset "coco-1024" --dataset-path coco2014 --profile stable-diffusion-xl-pytorch --model-path model/ [--dtype <fp32, fp16 or bf16>] [--device <cuda or cpu>] [--time <time>] [--scenario <SingleStream, MultiStream, Server or Offline>]
Add the --accuracy
to the command to run the benchmark
python3 main.py --dataset "coco-1024" --dataset-path coco2014 --profile stable-diffusion-xl-pytorch --accuracy --model-path model/ [--dtype <fp32, fp16 or bf16>] [--device <cuda or cpu>] [--time <time>] [--scenario <SingleStream, MultiStream, Server or Offline>]