Paper | Supplementary Material | Project Page

This repository hosts the datasets and the code for the SketchyScene Colorization system (SIGGRAPH Asia 2019). Please refer to our paper for more information: Language-based Colorization of Scene Sketches.

Our system supports two-mode interactive colorization for a given input scene sketch and text-based colorization instructions, using three models, namely, the instance matching model, foreground colorization model, and background colorization model.

• Requirements
• Preparations
  • Instance Matching
  • Foreground Instance Colorization
  • Background Colorization
• Colorizing With The Whole Pipeline
• Citation

• Python 3
• Tensorflow (>= 1.3.0)
• scipy
• PIL
• skimage

Please follow the instructions in the following three sections (Instance Matching, Foreground Instance Colorization, and Background Colorization) to download the dataset and pre-trained models and place them in the correct directories.

For the details of MATCHING dataset and the code, please refer to the Instance_Matching directory.

For the details of FOREGROUND dataset and the code, please refer to the Foreground_Instance_Colorization directory.

For the details of BACKGROUND dataset and the code, please refer to the Background_Colorization directory.

Our system allows users to colorize the sketches through language instructions. If the result is not satisfactory, users can also withdraw the last instruction.

🔥 We have provided some test examples in examples directory.

1. To colorize a sketch, run the command like:

   python3 sketchyscene_colorization_main.py --image_id 9996 \
                                             --instruction 'the bus is orange with gray windows'
   

   • Set image_id to the sketch you want.
   • Try other instructions by changing the instruction.

   You will see the results in outputs directory.

2. To withdraw the last instruction, run the command like:

   python3 sketchyscene_colorization_main.py --command 'withdraw' --image_id 9996
   

   See what happens in outputs directory :)

Please cite the corresponding paper if you found the datasets or code useful:

@article{zouSA2019sketchcolorization,
  title = {Language-based Colorization of Scene Sketches},
  author = {Zou, Changqing and Mo, Haoran and Gao, Chengying and Du, Ruofei and Fu, Hongbo},
  journal = {ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia 2019)},
  year = {2019},
  volume = 38,
  number = 6,
  pages = {233:1--233:16}
}

@inproceedings{zou2018sketchyscene,
  title={Sketchyscene: Richly-annotated scene sketches},
  author={Zou, Changqing and Yu, Qian and Du, Ruofei and Mo, Haoran and Song, Yi-Zhe and Xiang, Tao and Gao, Chengying and Chen, Baoquan and Zhang, Hao},
  booktitle={Proceedings of the european conference on computer vision (ECCV)},
  pages={421--436},
  year={2018}
}