Text-guided image generation has progressed rapidly in recent years, inspiring major breakthroughs in text-guided shape generation. Recently, it has been shown that using score distillation, one can successfully text-guide a NeRF model to generate a 3D object. We adapt the score distillation to the publicly available, and computationally efficient, Latent Diffusion Models, which apply the entire diffusion process in a compact latent space of a pretrained autoencoder. As NeRFs operate in image space, a naïve solution for guiding them with latent score distillation would require encoding to the latent space at each guidance step. Instead, we propose to bring the NeRF to the latent space, resulting in a Latent-NeRF. Analyzing our Latent-NeRF, we show that while Text-to-3D models can generate impressive results, they are inherently unconstrained and may lack the ability to guide or enforce a specific 3D structure. To assist and direct the 3D generation, we propose to guide our Latent-NeRF using a Sketch-Shape: an abstract geometry that defines the coarse structure of the desired object. Then, we present means to integrate such a constraint directly into a Latent-NeRF. This unique combination of text and shape guidance allows for increased control over the generation process. We also show that latent score distillation can be successfully applied directly on 3D meshes. This allows for generating high-quality textures on a given geometry. Our experiments validate the power of our different forms of guidance and the efficiency of using latent rendering.
Official Implementation for "Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures".
TL;DR - We explore different ways of introducing shape-guidance for Text-to-3D and present three models: a purely text-guided Latent-NeRF, Latent-NeRF with soft shape guidance for more exact control over the generated shape, and Latent-Paint for texture generation for explicit shapes.
-
27.11.2022
- Code release -
14.11.2022
- Created initial repo
In the Latent-Paint
application, a texture is generated for an explicit mesh directly on its texture map using stable-diffusion as a prior.
Here the geometry is used as a hard constraint where the generation process is tied to the given mesh and its parameterization.
Below we can see the progress of the generation process over the optimization process
To create such results, run the train_latent_paint
script. Parameters are handled using pyrallis and can be passed from a config file or the cmd.
python -m scripts.train_latent_paint --config_path demo_configs/latent_paint/goldfish.yaml
or alternatively
python -m scripts.train_latent_paint --log.exp_name 2022_11_22_goldfish --guide.text "A goldfish" --guide.shape_path /nfs/private/gal/meshes/blub.obj
Here we use a simple coarse geometry which we call a SketchShape
to guide the generation process.
A SketchShape
presents a soft constraint which guides the occupancy of a learned NeRF model but isn't constrained to its exact geometry.
A SketchShape
can come in many forms, here are some extruded ones.
To create such results, run the train_latent_nerf
script. Parameters are handled using pyrallis and can be passed from a config file or the cmd.
python -m scripts.train_latent_nerf --config_path demo_configs/latent_nerf/lego_man.yaml
Or alternatively
python -m scripts.train_latent_nerf --log.exp_name '2022_11_25_lego_man' --guide.text 'a lego man' --guide.shape_path shapes/teddy.obj --render.nerf_type latent
Here we apply a text-to-3D without any shape constraint similarly to dreamfusion and stable-dreamfusion.
We directly train the NeRF in latent space, so no encoding into the latent space is required during training.
To create such results, run the train_latent_nerf
script. Parameters are handled using pyrallis and can be passed from a config file or the cmd.
python -m scripts.train_latent_nerf --config_path demo_configs/latent_nerf/sand_castle.yaml
Or alternatively
python -m scripts.train_latent_nerf --log.exp_name 'sand_castle' --guide.text 'a highly detailed sand castle' --render.nerf_type latent
As our Latent-NeRF is supervised by Stable-Diffusion, we can also use Textual Inversion
tokens as part of the input text prompt. This allows conditioning the object generation on specific objects and styles, defined only by input images.
For Textual-Inversion results use the guide.concept_name
with a concept from the 🤗 concept library. For example --guide.concept_name=cat-toy
and then simply use the corresponding token in your --guide.text
Install the common dependencies from the requirements.txt
file
pip install -r requirements.txt
For Latent-NeRF
with shape-guidance, additionally install igl
conda install -c conda-forge igl
For Latent-Paint
, additionally install kaolin
pip install git+https://github.com/NVIDIAGameWorks/kaolin
Note that you also need a 🤗 token for StableDiffusion. First accept conditions for the model you want to use, default one is CompVis/stable-diffusion-v1-4
. Then, add a TOKEN file access token to the root folder of this project, or use the huggingface-cli login
command
Scripts for training are available in the scripts/
folder, see above or in the demo_configs/
for some actual examples.
Meshes for shape-guidance are available under shapes/
-
Check out the
vis/train
to see the actual rendering used during the optimization. You might want to play around with theguide.mesh_scale
if the object looks too small or too large. -
For
Latent-NeRF
with shape-guidance try changingguide.proximal_surface
andoptim.lambda_shape
to control the strictness of the guidance
The Latent-NeRF
code is heavily based on the stable-dreamfusion project, and the Latent-Paint
code borrows from text2mesh.
If you use this code for your research, please cite our paper Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures
@article{metzer2022latent,
title={Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures},
author={Metzer, Gal and Richardson, Elad and Patashnik, Or and Giryes, Raja and Cohen-Or, Daniel},
journal={arXiv preprint arXiv:2211.07600},
year={2022}
}