Skip to content

jankolf/QUD

Repository files navigation

Official repository of
 
[ /kjuːt/ ]  

QUD: Unsupervised Knowledge Distillation
for Deep Face Recognition

Jan Niklas Kolf 1,2Naser Damer 1,2Fadi Boutros 1

1 Fraunhofer IGD   2 Technische Universität Darmstadt

Accepted at BMVC 2024

Overview 🔎

Overview of the proposed unsupervised KD method QUD. Using contrastive loss, student S is trained so that the distance of its feature f = S(x) to teacher T’s positive feature f+ = T(x) of the same input sample x is smaller than the distance between f and a set of negative features f-f- stored in a queue. The queue is filled with features of T from previous iterations. After each iteration, the current features of T are enqueued and an equal amount of the oldest features are dequeued from the queue. Only S’s parameters are updated.

Video 📼

qud_bmvc_video.mp4

Abstract 🤏

We present in this paper an unsupervised knowledge distillation (KD) approach, namely QUD, for face recognition. The proposed QUD approach utilizes a queue of features within a contrastive learning setup to guide the student model to learn a feature representation similar to its counterpart obtained from the teacher and dissimilar from the ones that are stored in a queue. This queue is updated by pushing a batch of feature representations obtained from the teacher into the queue and dequeuing the oldest ones from the queue in each training iteration. We additionally incorporate a temperature into the contrastive loss to control how sensitive contrastive learning is to samples considered negative in the queue. The proposed unsupervised QUD approach does not require accessing the same dataset used to train the teacher model or even for the data to have identity labels. The effectiveness of the proposed approach is demonstrated through several sensitivity studies on different teacher architectures and using different datasets for student training in the KD framework. Additionally, the achieved results on mainstream benchmarks by our unsupervised QUD are compared to state-of-the-art (SOTA), achieving very competitive performances and even outperforming SOTA on several benchmarks.

Citation ✒

If you found this work helpful for your research, please cite the article with the following bibtex entry:

@inproceedings{DBLP:conf/bmvc/KolfQUD,
  author       = {Jan Niklas Kolf and
                  Naser Damer and
                  Fadi Boutros},
  title        = {{QUD:} Unsupervised Knowledge Distillation for Deep Face Recognition},
  booktitle    = {35th British Machine Vision Conference 2024, {BMVC} 2024, Glasgow,
                  UK, November 25-28, 2024},
  publisher    = {{BMVA} Press},
  year         = {2024},
}

License

This project is licensed under the terms of the Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license.

Copyright (c) 2024 Fraunhofer Institute for Computer Graphics Research IGD, Darmstadt.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published