Ported to Nuke

The files from the original repository were converted to be compatible with Nuke Cattery

1. It works very poorly for movement since it blinking too much.
2. Requires using a deflicker
3. The result varies depending on the resolution

Official implementation of the paper

Rethinking Inductive Biases for Surface Normal Estimation
CVPR 2024 [oral]
Gwangbin Bae and Andrew J. Davison
[paper.pdf] [arXiv] [youtube] [project page]

Despite the growing demand for accurate surface normal estimation models, existing methods use general-purpose dense prediction models, adopting the same inductive biases as other tasks. In this paper, we discuss the inductive biases needed for surface normal estimation and propose to (1) utilize the per-pixel ray direction and (2) encode the relationship between neighboring surface normals by learning their relative rotation. The proposed method can generate crisp — yet, piecewise smooth — predictions for challenging in-the-wild images of arbitrary resolution and aspect ratio. Compared to a recent ViT-based state-of-the-art model, our method shows a stronger generalization ability, despite being trained on an orders of magnitude smaller dataset.

1. Download and unzip the latest release from here.
2. Copy the extracted Cattery folder to .nuke or your plugins path.

# Linux
git clone https://github.com/vinavfx/DSINE-for-Nuke.git
cd ./DSINE-for-Nuke

conda create -n dsine python=3.9
conda activate dsine
pip install -r requirements.txt

wget https://huggingface.co/bdsqlsz/qinglong_controlnet-lllite/resolve/main/Annotators/dsine.pt
python ./dsine_nuke.py
# Convert with CatFileCreator.nk

If you find our work useful in your research please consider citing our paper:

@inproceedings{bae2024dsine,
    title     = {Rethinking Inductive Biases for Surface Normal Estimation},
    author    = {Gwangbin Bae and Andrew J. Davison},
    booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year      = {2024}
}

If you use the models that also estimate the uncertainty, please also cite the following paper, where we introduced the loss function:

@InProceedings{bae2021eesnu,
    title     = {Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation}
    author    = {Gwangbin Bae and Ignas Budvytis and Roberto Cipolla},
    booktitle = {International Conference on Computer Vision (ICCV)},
    year      = {2021}                         
}