Implementation of the object-based transformer model from "Object-based attention for spatio-temporal reasoning" [1].

This package includes source code for the transformer model, pre-trained model parameters for the CLEVRER task, and MONet [2] latent variables for all videos in the training and validation sets. It does not include the model training code. See Section 2 of [1] for details.

[1] David Ding, Felix Hill, Adam Santoro, Matt Botvinick. Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures. arXiv preprint arXiv:2012.08508, 2020.

[2] Chris P. Burgess, Loic Matthey, Nick Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner MONet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11390, 2019.

Note: This code depends on Tensorflow 1 and Sonnet 1. Tensorflow 1 is only available on PYPI for Python 3.7 and earlier.

To run this code, execute the following commands from the deepmind_research/ directory:

# Download checkpoints and MONet latents
wget https://storage.googleapis.com/object-attention-for-reasoning/checkpoints_and_latents.zip
unzip checkpoints_and_latents.zip
python3.7 -m venv object_based_attention_venv
source object_based_attention_venv/bin/activate
pip install --upgrade setuptools wheel
pip install -r requirements.txt
python -m object_attention_for_reasoning.run_model

If the code runs correctly, you should see the model's predicted answer to two CLEVRER questions (a descriptive one and a multiple choice one), and both answers should be correct.

If you find the provided code useful, please cite this paper:

@article{objectattention2020,
  title={Object-based attention for spatio-temporal reasoning: Outperforming
         neuro-symbolic models with flexible distributed architectures},
  author={David Ding and Felix Hill and Adam Santoro and Matt Botvinick},
  journal={arXiv preprint arXiv:2012.08508},
  year={2020}
}