Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
ssundaram21 authored Jul 30, 2024
1 parent 9ae211b commit 9fd48bf
Showing 1 changed file with 5 additions and 5 deletions.
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@ Current metrics for perceptual image similarity operate at the level of pixels a
DreamSim is a new metric for perceptual image similarity that bridges the gap between "low-level" metrics (e.g. LPIPS, PSNR, SSIM) and "high-level" measures (e.g. CLIP). Our model was trained by concatenating CLIP, OpenCLIP, and DINO embeddings, and then finetuning on human perceptual judgements. We gathered these judgements on a dataset of ~20k image triplets, generated by diffusion models. Our model achieves better alignment with human similarity judgements than existing metrics, and can be used for downstream applications such as image retrieval.

## 🕰️ Coming soon
* JND Dataset release
* ✅ JND Dataset release
* ✅ Compatibility with the most recent version of PEFT
* Distilled DreamSim models (i.e. smaller models distilled from the main ensemble)
* DreamSim variants trained for higher resolutions
* Compatibility with the most recent version of PEFT

## 🚀 Newest Updates
**X/XX/24:** Released new versions of the ensemble and single-branch DreamSim models compatible with `peft>=0.2.0`.
Expand All @@ -36,7 +36,7 @@ Here's how they perform on the NIGHTS validation set:
* [Setup](#setup)
* [Usage](#usage)
* [Quickstart](#quickstart-perceptual-similarity-metric)
* [Single-branch models](#new-single-branch-models)
* [Single-branch models](#single-branch-models)
* [Feature extraction](#feature-extraction)
* [Image retrieval](#image-retrieval)
* [Perceptual loss function](#perceptual-loss-function)
Expand Down Expand Up @@ -95,9 +95,9 @@ distance = model(img1, img2) # The model takes an RGB image from [0, 1], size ba
To run on example images, run `demo.py`. The script should produce distances (0.424, 0.34).

### Single-branch models
By default, DreamSim uses an ensemble of CLIP, DINO, and OpenCLIP (all ViT-B/16). If you need a lighter-weight model you can use *single-branch* versions of DreamSim where only a single backbone is finetuned. The available options are OpenCLIP-ViTB/32, DINO-ViTB/16, CLIP-ViTB/32, in order of performance.
By default, DreamSim uses an ensemble of CLIP, DINO, and OpenCLIP (all ViT-B/16). If you need a lighter-weight model you can use *single-branch* versions of DreamSim where only a single backbone is finetuned. **The single-branch models provide a ~3x speedup over the ensemble.**

To load a single-branch model, use the `dreamsim_type` argument. For example:
The available options are OpenCLIP-ViTB/32, DINO-ViTB/16, CLIP-ViTB/32, in order of performance. To load a single-branch model, use the `dreamsim_type` argument. For example:
```
dreamsim_dino_model, preprocess = dreamsim(pretrained=True, dreamsim_type="dino_vitb16")
```
Expand Down

0 comments on commit 9fd48bf

Please sign in to comment.