Published in ACM Transactions on Graphics (Proc. of Siggraph 2021), 40(4): Article 46., 2021

Yiqian Wu, Yongliang Yang, Qinjie Xiao, Xiaogang Jin.

Paper Video Project Page

Abstract:

Facial structure editing of portrait images is challenging given the facial variety, the lack of ground-truth, the necessity of jointly adjusting color and shape, and the requirement of no visual artifacts. In this paper, we investigate how to perform chin editing as a case study of editing facial structures. We present a novel method that can automatically remove the double chin effect in portrait images. Our core idea is to train a fine classification boundary in the latent space of the portrait images. This can be used to edit the chin appearance by manipulating the latent code of the input portrait image while preserving the original portrait features. To achieve such a fine separation boundary, we employ a carefully designed training stage based on latent codes of paired synthetic images with and without a double chin. In the testing stage, our method can automatically handle portrait images with only a refinement to subtle misalignment before and after double chin editing. Our model enables alteration to the neck region of the input portrait image while keeping other regions unchanged, and guarantees the rationality of neck structure and the consistency of facial characteristics. To the best of our knowledge, this presents the first effort towards an effective application for editing double chins. We validate the efficacy and efficiency of our approach through extensive experiments and user studies.

1. Windows
2. Python 3.6
3. NVIDIA GPU + CUDA10.0 + CuDNN

Download the following pretrained models, put each of them to PATH:

Please install dependencies by

conda create -n Coarse2Fine python=3.6
activate Coarse2Fine
pip install -r requirements.txt

Pre-trained separation boundaries can be found at ./interface/boundaries:

Notice that psi-0.5 dataset and psi-0.8 dataset is images and latent codes we generated by stylegan2 with psi=0.5(faces are more stable ) and psi=0.8(faces are more diverse)

For real images, first find the matching latent vectors.

For aligned images original image {name}.jpg that placed in$DATA_PATH$/origin, the corresponding latent code (in WP(W+) latent space) {name}_wp.npy should be placed in $DATA_PATH$/code.

1. We recommend you use the projector of official stylegan2 to obtain the latent codes of real images, to correctly use the StyleGAN2 projector, please follow the Requirements in stylegan2 .
2. Or you can run Projector(pytorch version):

​ First, Align faces from input images and save aligned images to $DATA_PATH$/origin.

python align_images.py\
	--raw_dir $DATA_PATH$/raw\
    --aligned_dir $DATA_PATH$/origin

​ Then run projector.

cd styleGAN2_model/stylegan2_pytorch
python run_projector.py 
    project_real_images\ 
    --data_dir=$DATA_PATH$/origin\ 
    --num_images=THE_NUMBER_OF_IMAGE_TO_PROJECT \
    --network=../pretrain/Gs.pth\
    --output=$DATA_PATH$/code

For diffuse method (both coarse separation boundaries and fine separation boundaries work):

python main_diffuse.py\
--data_dir $DATA_PATH$\
--boundary_path ./interface/boundaries/fine/all\
--boundary_init_ratio -4.0\
--boundary_additional_ratio -1.0\
--latent_space_type WP

For warp method (need fine separation boundaries):

python main_warp.py\
--data_dir $DATA_PATH$\
--boundary_path ./interface/boundaries/fine/all\
--boundary_init_ratio -4.0\
--latent_space_type WP

1. Data generation:

   python generate_data_and_score.py\
   --output_dir $PATH_TO_DATASET$\
   --num 50000\
   --truncation_psi 0.8\

If you want to generate data from your own latent codes, please set --latent_codes_path as PATH_TO_LATENT_CODE, else the latent codes will be randomly generated.

If you only want to generate images that have double chin ,set --double_chin_only

​ 2.Coarse boundary training:

python train_coarse_boundary.py\
--output_dir PATH_TO_SAVE_BOUNDARY\
--latent_codes_path $PATH_TO_DATASET$/w.npy \
--scores_path $PATH_TO_DATASET$/double_chin_scores.npy\
--chosen_num_or_ratio 0.1\
--split_ratio 0.9 \

The boundary will be saved in PATH_TO_SAVE_BOUNDARY

First, prepare the data for diffusion

using coarse boundary to prepare data

python remove_double_chin_step1.py\
    --output_dir $TRAINING_DIR$\
    --boundary_path PATH_TO_LOAD_BOUNDARY\   
    --input_data_dir $PATH_TO_DATASET$

Then diffuse the prepared data:

python remove_double_chin_step2.py\
	--data_dir $TRAINING_DIR$

the data_dir should be the same as output_dir that you input in remove_double_chin_step1.py

the results of diffusion will be saved in data_dir

3.After diffuse, you can use the results of diffuse to train a fine boundary:

python train_fine_boundary.py\
--output_dir PATH_TO_SAVE_BOUNDARY\
--latent_codes_path $TRAINING_DIR$/codes\
--split_ratio 0.9 \

the fine boundary has better performance in preserving face identity.

the comparison between fine boundary(right) and coarse boundary(middle):

[email protected]

If you use this code for your research, please cite our paper:

@article{DBLP:journals/tog/WuYX021,
  author    = {Yiqian Wu and
               Yong{-}Liang Yang and
               Qinjie Xiao and
               Xiaogang Jin},
  title     = {Coarse-to-fine: facial structure editing of portrait images via latent
               space classifications},
  journal   = {{ACM} Trans. Graph.},
  volume    = {40},
  number    = {4},
  pages     = {46:1--46:13},
  year      = {2021}
}

We thanks the following works:

StyleGAN2

InterFaceGAN

StyleGAN2(pytorch-version)

face-alignment

idinvert

MaskGAN

@inproceedings{zhu2020indomain,
  title     = {In-domain GAN Inversion for Real Image Editing},
  author    = {Zhu, Jiapeng and Shen, Yujun and Zhao, Deli and Zhou, Bolei},
  booktitle = {Proceedings of European Conference on Computer Vision (ECCV)},
  year      = {2020}
}
@inproceedings{bulat2017far,
  title={How far are we from solving the 2D \& 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)},
  author={Bulat, Adrian and Tzimiropoulos, Georgios},
  booktitle={International Conference on Computer Vision},
  year={2017}
}
@inproceedings{shen2020interpreting,
  title     = {Interpreting the Latent Space of GANs for Semantic Face Editing},
  author    = {Shen, Yujun and Gu, Jinjin and Tang, Xiaoou and Zhou, Bolei},
  booktitle = {CVPR},
  year      = {2020}
}
@inproceedings{Karras2019stylegan2,
  title     = {Analyzing and Improving the Image Quality of {StyleGAN}},
  author    = {Tero Karras and Samuli Laine and Miika Aittala and Janne Hellsten and Jaakko Lehtinen and Timo Aila},
  booktitle = {Proc. CVPR},
  year      = {2020}
}
@inproceedings{CelebAMask-HQ,
  title={MaskGAN: Towards Diverse and Interactive Facial Image Manipulation},
  author={Lee, Cheng-Han and Liu, Ziwei and Wu, Lingyun and Luo, Ping},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2020}
}