This is evaluation code for our recent paper Explaining Answers with Entailment Trees, EMNLP 2021 (https://arxiv.org/abs/2104.08661)
Dataset available at https://allenai.org/data/entailmentbank
EntailmentBank annotation tool can be found at https://github.com/cognitiveailab/entailmentbank-tree-annotation-tool
Our T5-large/T5-11B models for Task1/Task2 are available in HuggingFace PyTorch format in Google Cloud storage at: https://console.cloud.google.com/storage/browser/ai2-bhavanad/entailment_bank/emnlp_2021_models/t5-models-hf
For additional instructions about using these models and sample commands please refer to https://github.com/allenai/entailment_bank/blob/main/README_models.md
- conda create -n entbank python=3.7
- conda activate entbank
- pip install -r requirements.txt
- Download the bleurt-large-512 model from https://github.com/google-research/bleurt/blob/master/checkpoints.md
Task1:
python eval/run_scorer.py \
--task "task_1" \
--split test \
--prediction_file PREDICTION-TSV-PATH \
--output_dir OUTPUT-PATH \
--bleurt_checkpoint "PATH to bleurt-large-512 model"
Task2:
python eval/run_scorer.py \
--task "task_2" \
--split test \
--prediction_file PREDICTION-TSV-PATH \
--output_dir OUTPUT-PATH \
--bleurt_checkpoint "PATH to bleurt-large-512 model"
Task3:
example command coming up soon
Prediction file (PREDICTION-TSV-PATH) is a single column TSV file with datapoints in the same order as the public dataset jsonl file. Value of each line is the predicted proof in the DOT format. e.g.
$proof$ = sent2 & sent3 -> int1: the northern hemisphere is a kind of place; int1 & sent1 -> hypothesis;
@article{entalmentbank2021,
title={Explaining Answers with Entailment Trees},
author={Dalvi, Bhavana and Jansen, Peter and Tafjord, Oyvind and Xie, Zhengnan and Smith, Hannah and Pipatanangkura, Leighanna and Clark, Peter},
journal={EMNLP},
year={2021}
}