unofficial ALBEF tutorial, with MIMIC-CXR data.

The following image from the ALBEF paper(https://arxiv.org/abs/2107.07651) illustrates the overall model architecture and training objectives of ALBEF. You can find more description of the detailed code in the comments in pretrain_albef.py.

Data are not included in this repository. You need to prepare jsonl file(contains labels, etc.) and images.

jsonl file: Visit https://github.com/SuperSupermoon/MedViLL, and from their /data/mimic, take Train.jsonl, Valid.jsonl, Test.jsonl and move them into your depository's ./data/MIMIC_CXR/. This jsonl file contains the image name(as a path), english prescription, and diagnosis labels.

images: Visit https://github.com/SuperSupermoon/MedViLL, and from their README.md, download MIMIC-CXR datasets to get mimic_dset.tar.gz. Move this into your depository and unzip it. This will create ./re_512_3ch/ folder with photos.

※The image download link above is now blocked, since redistributing the MIMIC-CXR data is forbidden. Please visit their website https://physionet.org/content/mimic-cxr/2.0.0/ and follow the official method to access the data.

Arguments can be passed with commands, or edited manually in the running code. Default values are already good to go, but I recommend you to check the arguments and hyperparameters in the code. You can read the code in pretrain_albef.py to understand how ALBEF works.

-Pretrain

python train.py

-Downstream task: multi-label classification

python classification.py --checkpoint ./save/file/path

-Retrieval [*pre-trained ALBEF would already work well for retrieval of its pre-train data, so no additional training.]

python retrieval.py --checkpoint ./save/file/path