Zero-shot image captioning (IC) without well-paired image-text data can be divided into two categories, training-free and text-only-training. The main difference between them is whether using a textual corpus to train the LM. Though achieving attractive performance w.r.t. some metrics, existing methods often exhibit some common drawbacks. Training-free methods tend to produce hallucinations, while text-only-training often lose generalization capability. To move forward, in this paper, we propose a novel Memory-Augmented zero-shot image Captioning framework (MeaCap). Specifically, equipped with a textual memory, we introduce a retrieve-then-filter module to get key concepts that are highly related to the image. By deploying our proposed memory-augmented visual-related fusion score in a keywords-to-sentence LM, MeaCap can generate concept-centered captions that keep high consistency with the image with fewer hallucinations and more world-knowledge.

@article{zeng2024meacap,
  title={MeaCap: Memory-Augmented Zero-shot Image Captioning},
  author={Zeng, Zequn and Xie, Yan and Zhang, Hao and Chen, Chiyu and Wang, Zhengjue and Chen, Bo},
  journal={arXiv preprint arXiv:2403.03715},
  year={2024}
}

pip install -r requirements.txt

data
└── memory
    ├── cc3m
    │   ├── memory_captions.json
    │   ├── memory_clip_embeddings.pt
    │   └── memory_wte_embeddings.pt
    ├── coco
    │   ├── memory_captions.json
    │   ├── memory_clip_embeddings.pt
    │   └── memory_wte_embeddings.pt
    └── ...    

python prepare_embedding.py --memory_id coco --memory_path data/memory/coco/memory_captions.json

$~~~~$

$~~~~$

$~~~~$

$~~~~$

$~~~~$

$~~~~$

$MeaCap_{TF}$

$MeaCap_{ToT}$

$MeaCap_{ToT}$

$MeaCap_{ToT}$

$MeaCap_{ToT}$

$MeaCap_{InvLM}$

$MeaCap_{TF}$

python inference.py --use_prompt  --memory_id cc3m --img_path ./image_example --lm_model_path ./checkpoints/CBART_one_billion 

$MeaCap_{ToT}$

python inference.py --memory_id coco --img_path ./image_example --lm_model_path ./checkpoints/CBART_COCO 

$MeaCap_{InvLM}$

python viecap_inference.py --memory_id coco --image_path "*.jpg" --weight_path "checkpoints/train_coco/coco_prefix-0014.pt"

$MeaCap_{TF}$

$MeaCap_{ToT}$

$MeaCap_{TF}$

$MeaCap_{ToT}$

$\Rightarrow$

$\Rightarrow$

$MeaCap_{TF}$

$MeaCap_{ToT}$

$~~~~$

$~~~~$

$~~~~$

$~~~~$

$~~~~$

$MeaCap_{InvLM}$