Skip to content

catlab-team/latent3D_code

Repository files navigation

Text and Image Guided 3D Avatar Generation and Manipulation

Installation and Usage

  1. Install the dependencies in env.yml

    conda env create -f env.yml

    conda activate latent3d-env

  2. Install pytorch3d by running the following:

    import sys
    import torch
    import subprocess
    version_str="".join([
        f"py3{sys.version_info.minor}_cu",
        torch.version.cuda.replace(".",""),
        f"_pyt{torch.__version__.split("+")[0].replace(".", "")}"
    ])
    subprocess.run(f"conda install https://anaconda.org/pytorch3d/pytorch3d/0.6.2/download/linux-64/pytorch3d-0.6.2-{version_str}.tar.bz2", shell=True)
  3. Download the pre-trained TBGAN model from this link: https://ibug.doc.ic.ac.uk/resources/tbgan/.

  • Note: You will be required to fill the form for 'End User Licence Agreement'.
  1. Place the pre-trained TBGAN model snapshot in the '/models/snapshots' directory.

  2. Download the pre-trained ArcFace model

    wget https://www.dropbox.com/s/kzo52d9neybjxsb/model_ir_se50.pth?dl=0 -O model_ir_se50.pth

  3. Place the pre-trained ArcFace model in the '/models/snapshots' directory.

Text-Based Manipulations

Running the 'optimize.py' with the required parameters and list of text prompts for manipulation, the rendered manipulated 3D faces and their originals will be saved under the direcory './results'.

python3 optimize.py --seed [SEED] --mode [MODE] --text_list [TEXT_LIST] --num_epochs [NUM_EPOCHS] --lambda_id [ID_COEFF] --lambda_l2 [L2_COEFF] --learning_rate [LEARNING_RATE]

  • Example: python3 optimize.py --seed 1000 --mode text-based --text_list "happy human" "happy person" --num_epochs 100 --lambda_id 0.01 --lambda_l2 0.0 --learning_rate 0.01 --folder_title happy

Image-based Manipulations

Running the 'optimize.py' with given parameters and an image for manipulation, the rendered manipulated 3D faces and their originals will be saved under the direcory './results'.

python3 optimize.py --seed [SEED] --mode [MODE] --image_path [IMAGE_PATH] --num_epochs [NUM_EPOCHS] --lambda_id [ID_COEFF] --lambda_l2 [L2_COEFF] --learning_rate [LEARNING_RATE]

Generate Mesh from Intermediate Latent Vector

python3 sample.py --functionality latent_vector --result_subdir ./results/003-optimize

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages