A platform for quick and easy development of deep learning networks for recognition and detection in videos. Includes popular models like C3D and SSD.

Check out our wiki!

fw: framewise weighting, obj: object interaction

Please cite ViP when releasing any work that used this platform: https://arxiv.org/abs/1910.02793

@article{ganesh2019vip,
  title={ViP: Video Platform for PyTorch},
  author={Ganesh, Madan Ravi and Hofesmann, Eric and Louis, Nathan and Corso, Jason},
  journal={arXiv preprint arXiv:1910.02793},
  year={2019}
}

• Datasets
• Models
• Requirements
• Installation
• Quick Start
  • Testing
  • Training
• Development
  • Add a Model
  • Add a Dataset
• FAQ

• Python 3.6
• Cuda 9.0
• (Suggested) Virtualenv

# Set up Python3 virtual environment
virtualenv -p python3.6 --no-site-packages vip
source vip/bin/activate

# Clone ViP repository
git clone https://github.com/MichiganCOG/ViP
cd ViP

# Install requirements and model weights
./install.sh

Run train.py and eval.py to train or test any implemented model. The parameters of every experiment is specified in its config.yaml file.

Use the --cfg_file command line argument to point to a different config yaml file. Additionally, all config parameters can be overriden with a command line argument.

Run eval.py with the argument --cfg_file pointing to the desired model config yaml file.

Ex: From the root directory of ViP, evaluate the action recognition network C3D on HMDB51

python eval.py --cfg_file models/c3d/config_test.yaml

Run train.py with the argument --cfg_file pointing to the desired model config yaml file.

Ex: From the root directory of ViP, train the action recognition network C3D on HMDB51

python train.py --cfg_file models/c3d/config_train.yaml

Additional examples can be found on our wiki.

New models and datasets can be added without needing to rewrite any training, evaluation, or data loading code.

To add a new model:

1. Create a new folder ViP/models/custom_model_name
2. Create a model class in ViP/models/custom_model_name/custom_model_name.py
   • Complete __init__, forward, and (optional) __load_pretrained_weights functions
3. Add PreprocessTrain and PreprocessEval classes within custom_model_name.py
4. Create config_train.yaml and config_test.yaml files for the new model

Examples of previously implemented models can be found here.

Additional information can be found on our wiki.

To add a new dataset:

1. Convert annotation data to our JSON format
   • The JSON skeleton templates can be found here
   • Existing scripts for datasets can be found here
2. Create a dataset class in ViP/datasets/custom_dataset_name.py.
   • Inherit DetectionDataset or RecognitionDataset from ViP/abstract_dataset.py
   • Complete __init__ and __getitem__ functions
   • Example skeleton dataset can be found here

Additional information can be found on our wiki.

A detailed FAQ can be found on our wiki.