IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination
Xi Chen, Sida Peng, Dongchen Yang, Yuan Liu, Bowen Pan, Chengfei Lv, Xiaowei Zhou

• 2024-5-22: 🤗 Live demo released at https://huggingface.co/spaces/LittleFrog/IntrinsicAnything.
• 2024-5-31: Code release for single-view inference.

1. Installation

conda create -n anyintrinsic python=3.10
conda activate anyintrinsic
pip install -r requirements.txt

2. Download the pre-trained diffusion models from hugghing face as follow:

# albedo checkpoint
huggingface-cli download --repo-type space --cache-dir  weights/albedo/checkpoints/ LittleFrog/IntrinsicAnything weights/albedo/checkpoints/last.ckpt 

# specular shaing checkpoint
huggingface-cli download --repo-type space --cache-dir  weights/specular/checkpoints/ LittleFrog/IntrinsicAnything weights/specular/checkpoints/last.ckpt 

3. Run inference to get intrinsic images:

python inference.py \
--input_dir  examples  \
--model_dir  weights/albedo \
--output_dir out/albedo \
--ddim 100 \
--batch_size 4

Parameter explanation:

• --input_dir: Path to the folder containing test images. The foreground object mask can be supplied as either RGBA input or specify a mask directory through --mask_dir. The mask can be obtained interactively using tools like Segment-Anything. We also provide a script generate_masks.py to generate masks automatically based on Rembg. (Note: Scene-level images can be directly referenced without masks)

• --model_dir: Path to the pre-trained checkpoint. Set weights/albedo for the albedo generation and weights/specular for the specular shading generation.

• --output_dir: Output path for the generated samples.

• --batch_size: Inference batch size.

4. Optionally, you can generate high-resolution samples under the guidance of samples from step 3.

python inference.py \
--input_dir examples  \
--model_dir weights/albedo \
--output_dir out/albedo_high_res \
--ddim 200 \
--batch_size 4 \
--guidance_dir out/albedo \
--guidance 3  \
--splits_vertical 2 \
--splits_horizontal 2 \
--splits_overlap 1

Extra parameter explanation:

• --guidance_dir: Path to the low resolution output (step 3).
• --guidance: Guidance scale of low-resolution images. We impartially found a scale from 2 to 5 generally yields good results.
• --splits_vertical & --splits_horizontal: The number of splits in the vertical and horizontal direction.
• --splits_overlap: The number of overlaps evenly distributed between two adjacent patches. The final value of each pixel is the average of all overlapped patches to improve the patches' consistency.

Comming Soon.

@article{chen2024intrinsicanything,
    title     = {IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination},
    author    = {Xi, Chen and Sida, Peng and Dongchen, Yang and Yuan, Liu and Bowen, Pan and Chengfei, Lv and Xiaowei, Zhou.},
    journal   = {arxiv: 2404.11593},
    year      = {2024},
    }