This repository contains the code for the following paper:

• Long Chen, Hanwang Zhang, Jun Xiao, Wei Liu, Shih-Fu Chang, Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Networks. In CVPR, 2018. [PDF]

Note: This repository is adaptived from DeepSim, tensorflow-resnet, Thanks a lot to Jiaxin Shi from Tsinghua University, and Yongqin Xian to release his dataset split and code.

• Python 2.7 (Anaconda recommended)
• TensorFlow
• Caffe (Optional, In my original experiment, I load pretrained ImageNet CNN model weigths in caffemodel format and save it to npy format for tensorflow. But we can direct download the npy format CNN pretrained weights)

• Download the dataset (CUB/AWA/aPY/SUN), and save it into directory /SP-AEN-HOME/data/, and change the corresponding path of dataset and split file in /SP-AEN-HOME/cfg.py.
• Download pretrained CNN model alexnet, caffenet, resnet, and save the pretrained model in /SP-AEN-HOME/model/pretrained/imagenet. And download pretrained model caffenet/fc6 and save to /SP-AEN-HOME/model/pretrained/generator/caffenet/fc6. Use /SP-AEN-HOME/model/caffe2tf.py to convert the caffemodel format weights to npy format.

Extract the resnet-101 CNN featuer and preprocessing the dataset directly use:

python /SP-AEN-HOME/convert_data.py -dataset cub (for CUB)
python /SP-AEN-HOME/convert_data.py -dataset apy (for aPY)
python /SP-AEN-HOME/convert_data.py -dataset awa2 (for AWA2)
python /SP-AEN-HOME/convert_data.py -dataset sun (for SUN)

dataset can select from ['cub', 'apy', 'awa2', 'sun'], for dataset CUB:

step1:
python run.py --dataset=cub --mode=trainf --summary_path=.. --alpha_map_rank=.. --alpha_rank_dis=.. --alpha_rank_gen=..
step2:
python run.py --dataset=cub --mode=traine --retrain_model=True --train_checkpoint=.. --summary_path=..

If you find this code useful, please cite the following paper:

@inproceedings{chen2018zero,
  title={Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Networks},
  author={Chen, Long and Zhang, Hanwang and Xiao, Jun and Liu, Wei and Chang, Shih-Fu},
  booktitle={CVPR},
  year={2018}
}