The mCLIPEval (multilingual evaluation for CLIP) tookit provides generic evaluation for pretrained vision-language models (such as CLIP, Contrastive Language–Image Pre-training). More precisely, mCLIPEval provides evaluations:

• On multilingual (12 languages) datasets and monolingual (English/Chinese) datasets, for tasks like zeroshot image/video classification, zeroshot image-to-text and text-to image retrieval and zeroshot visio-linguistic compositionality.

• Adapted to various open-source pretrained models, like FlagAI pretrained models (AltCLIP, EVA-CLIP), OpenCLIP pretrained models, Chinese CLIP models, Multilingual CLIP models, Taiyi Series pretrained models. Customized models can be also supported with model scripts.

mCLIPEval provides APIs to quickly download and preparation of public datasets from torchvision, huggingface, kaggle, as well as download and use those open-source pretrained models on these datasets.

mCLIPEval provides visualization of evaluation results through streamlit web app, to see the comparsions of performances on specific languages, tasks, or model parameters.

Below is the scatter plot to show the performances of some open-source models.

You can use the online demo to experience mCLIPEval.

1. Easy-to-compare capabilities of open-source models

   • on specific datasets, languages, tasks, model parameters
   • through interactive inferfaces to switch different configuration settings
   • without computational resources needs to support inference and evaluation.

2. Easy-to-evaluate pretrained models using checkpoint files

   • trained through various frameworks, like FlagAI, OpenCLIP, Transformers
   • or any customized frameworks with model scripts
   • on specific datasets, languages, tasks.

3. Easy-to-build a evaluation framework from scratch

   • including download and preparation of datasets and models,
   • evaluation with various configuration settings
   • and visualization of the evaluation results.

• Requiremnets Installation
• How to Use
  • Structure
  • Quick Tour
  • Advanced Usage Examples
• Datasets and Models
  • Supported Datasets
  • Built-in Models
• Reference
• Credits
• License

To use mCLIPEval, you need:

• Pytorch version >= 1.8.0
• Python version >= 3.8
• For evaluating models on GPUs, you'll also need install CUDA and NCCL

[Recommended] For complete usage, you are recommended to install the required packages through:

pip install -r requirements.txt

• [Optional] To use datasets from huggingface (imagenet1k in any languages, winoground), you need to:

  • 1. generate huggingface API TOKEN (select the role "read") from huggingface following the instructions;
  • 2. run the command and add the generated token as git credential: huggingface-cli login or modify the download/constants.py file with generated token >>> _HUGGINGFACE_AUTH_TOKEN = "hf_..."
  • 3. click the Agree and access repository button on dataset pages (imagenet-1k and winoground) to accept the license agreements of the datasets.

    

• [Optional] To use datasets from kaggle (fer2013, flickr30k, flickr30k_cn, multi30k), you need to:

  • 1. generate API token from kaggle following the instructions.
  • 2. install unzip command, for Debian/Ubuntu Linux, use sudo apt-get install unzip; for CentOS/RHEL Linux, use yum install unzip; for macOS, use brew install unzip.

[Partial] For partial usage, you need to install required packages:

• [Necessary] for basic evaluation: pip install torch,torchvision,sklearn

• [Optional] for data preparation: pip install -r requirements_dataset.txt

• [Optional] for usage of pretrained models: pip install -r requirements_models.txt

• [Optional] for visualization: pip install -r requirements_visual.txt

The complete use of mCLIPEval contains three standalone modules: data preparation, evaluation and visualization.

• To immediately see the comparison results of built-in open-source models, we provide outputs as early-run evaluation results. You just need to run:

  streamlit run visual.py -- --json="outputs/*.json"
  

• To evaluate a pretrained model with checkpoint files, you need to:

  • specify the model script, for example models/altclip.py
  • choose the evaluation datasets (for example cifar10 is a classical image classification dataset)
  • download and prepare the datasets with:

    python download.py --datasets=cifar10
    
  • evaluate the pretrained model in the directory [MODEL_DIR]

    python evaluate.py --model_name=[MODEL_NAME] --model_dir=[MODEL_DIR] --model_script=models.altclip --datasets=cifar10 --output=[MODEL_NAME].json
    
  • the evaluation results are saved in [MODEL_NAME].json file
  • [Tips] if the parameter --datasets is not specified, all supported datasets are chosen (the process of data preparation and evaluation would take a long time).

Thanks for your interest in contributing! Apart from regular commits, we also welcome contributions to resources (datasets, models, tasks).

• Thanks to CLIP Benchmark authors, the zeroshot classification and retrieval code, and the dataset building code are adpated from there.

• Thanks to CLIP, Lit, SLIP, imagenet_classes_chinese, japanese_clip, clip-italian authors, as the orginal zeroshot templates and multilingual classnames.

• Thanks to Winoground, ImageNet-Sketch, ImageNet-A, ImageNet-R authors to provide original datasets.

• Thanks to COCO-CN and Flickr30k CN authors to provide the original datasets.

The majority of mCLIPEval is licensed under the Apache 2.0 license, however portions of the project are available under separate license terms:

• The usage of CLIP_benchmark is licensed under the MIT license
• The usage of ImageNet1k datasets in under the huggingface datasets license and ImageNet licenese