A light-weight FaceRender is support for more than 100x faster rendering on Mac OS.

[vinthony]

Hi all,

I create a new pull request(#458) to include a new facerender for FAST inference.

CONV3D is not optimized in MacOS(device==mps), making the device of MacOS too slow to generate.

Here, I integrate the weights/model of https://github.com/RenYurui/PIRender as an alternation facerender for a generation.

This face-render is much faster than our previous one but with (a little bit) different qualities (I do not test the visual quality of different facerenders due to time limitation), for a 3 second audio,

Method	Speed
facevid2vid-cpu	~50min
pirender-cpu	~5min
pirender-mps	~5second

To use the new facerender,

1. checkout to the new branch

git fetch origin pull/458/head:pirender
git checkout pirender

2. Download the checkpoint from https://drive.google.com/file/d/1-0xOf6g58OmtKtEWJlU3VlnfRqPN9Uq7/view or the Github of PIRender and put the file of epoch_00190_iteration_000400000_checkpoint.pt directly in the checkpoints folder as before.

3. For the command line user, a new command line option --facerender is introduced with the choice of facevid2vid and pirender.

4. For WEBUI/Gradio users, a new button is added.

This facerender can also run very fast on GPU, I have not tested it yet but the conclusion is the same.

[MoroseYu]

fatal: couldn't find remote ref pull/458/head

[vinthony]

make sure the code is correct:

git fetch origin pull/458/head:pirender

[MoroseYu]

Thank you, the issue has been resolved after re-downloading the code.

[alphafa]

还是没有facerender选项

[MoroseYu]

I noticed a small issue. When using the "--facerender pirender --size 512" mode, the images are heavily distorted. However, when using size 256, the images appear normal.

[vinthony]

This render is only trained on 256x256 images.

[jjmlovesgit]

WOW! Impressive work... I am running windows on an AMD Ryzen 5000 series Laptop with small 2 GB Nvidia (weak laptop) and I can confirm this works 10X faster for the image rendering. It did take the GPU right up to 100%. I did a quick test on my standard benchmark 9 second file that showed 36 mins with facevid2vid and with pirender its came in under 4 mins...

Nice looking interface in Gradio too!

[RonyOficial]

I apologize for my lay position, I'm just starting to understand and learn, and it wasn't clear to me, what I have to do to implement the New faceRender... I didn't understand what I had to do in
1- checkout to the new branch

Can you help me ?

[vinthony]

It will be merged into the main repo. If you want to test it now, a basic option is to clone from my fork and run as the original sadtalker: https://github.com/vinthony/SadTalker

[RonyOficial]

OMG thank you! Everything worked, I'm going to test it now.

Thanks again.

[snakeninny]

Any updates? Your fork is not merged yet

[RonyOficial]

Hi Vinthony, how are you bro?

I need one more help, I've tried but I'm not getting big movements with the lips, the mouth is half open most of the time, can you help me with a better setup to fix this?

Thank you in advance once again.

[Girish35897]

Hi ! ,
Really am awesome work , thanks to the team. This is so fast in giving output. But I observed in the output there is no teeth for the person, is there any precautions or parameter to optimize or other methods to fix this problem ?

Image passed as input is closed lips , no teeth visible , same image passed to earlier model which used facevid2vid have good results with teeth

Thank you

[Metacowboy]

On a MacBook Pro Ventura 13.4 Python crash with following message

/AppleInternal/Library/BuildRoots/2acced82-df86-11ed-9b95-428477786501/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSCore/Types/MPSNDArray.mm:90: failed assertion [MPSNDArrayDescriptor sliceDimension:withSubrange:] error: the range subRange.start + subRange.length does not fit in dimension[2] (1411)'
zsh: abort sh webui.sh`

miniconda3/envs/sadtalker/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

[edwios]

Got the same error here on a M1 Max, MacOS 13.4.1, torch==2.1.0.dev20230710, torchvision==0.16.0.dev20230710. It always failed at around 70% with this error message.

[halcyonwalker]

same here, await for a solution. Appreciate the attention in advance!

[gasparitiago]

Any news about this? @Metacowboy @edwios have you fixed it?

[shisaq]

Any updates?

[edwios]

Finally got things sort of running on my M1 Max with MacOS 13.4.1, torch==2.1.0.dev20230710, torchvision==0.16.0.dev20230710.
However, only face model resolution of 256 works, for 512, the video looks like it's placed under flowing water. Still, it was awesome, @vinthony ! Thank you for bring this to us!

Both input image was 512x512

Using 256 model:
https://github.com/OpenTalker/SadTalker/assets/619004/1905d82d-de94-48cd-8d13-3c33b70a1be4

Using 512 model:
https://github.com/OpenTalker/SadTalker/assets/619004/7d258daa-6638-4648-a344-4f74d29b9e02

Additional changes made to the sauce

I have to do the following changes in additional to this PR:

predict.py: in class Predictor:
device = "cuda" => device = "mps"

src/utils/croper.py: in class Preprocesser:
def __init__(self, device='cuda'): => def __init__(self, device='mps'):

src/utils/face_enhancer.py: in function enhancer_generator_no_len(), add device='mps', to both the GFPGAN and RealESRGAN model loaders (line 82 and 103).

Change the file retinaface.py in the facexlib python module to use MPS:
facexlib/detection/retinaface.py (line 14)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') => device = torch.device('cuda' if torch.cuda.is_available() else 'mps' if torch.has_mps else 'cpu')

It was using MPS:

[retret66]

I try this on m2 sonoma and did not work, it doesnt use MPS , is it strictly dependent on torch versions? thanks

[freedom6xiaobai]

Very cool, generated in 5 seconds

[Edanaw]

Screen FaceRender is 00:00:25, you say 5 seconds?

[NBNew]

Dear @vinthony , thanks a ton for coming up with this awesome model for facerender enhancement, this is really a game changer for me being on mac m1 pro chip and huge thanks to @jjmlovesgit as well for introducing me to this page 🙏🏻

Currently, the facerender is much faster than before however now, the Face Enhancer step seems to be taking way too long to complete. Is there any specific model/weight or some steps you would recommend me to try ?

Any advise would be greatly appreciated !!

[kenichi-kyogoku]

I think SadTalker is of excellent quality. Thank you very much for releasing this wonderful library.

Currently I have the normal SadTalker Extension working in my local environment with Stabule diffusion web-ui, but the light-weight FaceRender is not working!
My environment is MacOS.
Is there anything else to do besides adding the following files to the checkpoits folder?

epoch_00190_iteration_000400000_checkpoint.pt

Sorry for the rudimentary question. Best regards.

[kingkong135]

You can access the repo https://github.com/RenYurui/PIRender to weight, or simple go https://drive.google.com/file/d/1-0xOf6g58OmtKtEWJlU3VlnfRqPN9Uq7/view

[alphafa]

具体如何操作

[lavanphat]

I am running windows on an Intel i5 12th series Laptop with small 4 GB Nvidia. I am getting CUDA out of memory error and It did not take the GPU right up to 100%. Can someone help me ?

[alphafa]

请确认您有正确的访问权限并且仓库存在。
(base) yuchangfa@yuchangfadeAir ~ % git fetch origin pull/458/head:pirender
致命错误：'origin' does not appear to be a git repository
致命错误：无法读取远程仓库。

请确认您有正确的访问权限并且仓库存在。
(base) yuchangfa@yuchangfadeAir ~ % git fetch origin pull/458/head:pirendergit checkout pirender
致命错误：'origin' does not appear to be a git repository
致命错误：无法读取远程仓库。

请确认您有正确的访问权限并且仓库存在。
(base) yuchangfa@yuchangfadeAir ~ % git fetch origin pull/458/head:pirender git checkout pirender
致命错误：'origin' does not appear to be a git repository
致命错误：无法读取远程仓库。

请确认您有正确的访问权限并且仓库存在。
(base) yuchangfa@yuchangfadeAir ~ %

[alphafa]

git fetch origin pull/458/head:pirender
git checkout pirender。使用该命令后出现提示

[fakerybakery]

Hi @vinthony, the license of PiRender is CC-BY-NC, will this cause issues with the Apache 2.0 license which this project is licensed under? Will this mean that on this mode this project will no longer be able to be used for commercial purposes?

[alphafa]

您知道如何进行安装该分支么，我一直报错

[genglinxiao]

As of today, the face render option just disappeared from the webui and rendering is extremely slow again. Git status confirmed that I'm on the pirender branch. What could be the problem?

[genglinxiao]

Hmm, interesting. After a lot of changes to the source code of SadTalker - mainly print function calls to figure out how to correctly switch to pirender - I decided to completely remove it from A1111 and reinstall the latest main. After that, for no particular reason, I migrated to pirender branch again and guess what. The options are back. I can continue use pirender on my M2 Macbook air now.
I guess the moral is, when something really wired happens, instead of spending time debugging, try reset first.

[CodeFromAnywhere]

hey I was wondering if there's a cost efficient api of this somewhere?

[nextosai]

Dear @vinthony, you've done an amazing job with SadTalker. I've been creating real-time avatar assistants since 2005 (Virtual Assistant Denise), and AI has improved everything. I also understand the complexity of a project such as this. As an Open Source project, yours is getting really close to the big players, and I'm sure they are following your progress. Please add to your wish list a very light, real-time version. This will be a game-changer for us! It's not by chance that they are charging USD 1,00 / minute to use their real-time API. As a note related to this pull, I've tested it on several Windows / NVidia computers with the same results. The face render using pirender is extremely fast, but for some reason, as already noticed here, it does not show the avatar teeth. When using the facevid2vid option with GFPGan, it takes longer compared to the current regular build, and the resulting video is blurry, as if we haven't used the face enhancer. I hope you find time to keep working on this project, as I'm sure there are many people like me wishing you luck.

[2shou]

can face-enhancer use gpu on mac m1? it run very slow. @vinthony

[xBurnsed]

Thsi is really good! I'd suggest changing the title and description tho. I am not sure why you added that this feature was meant only to speed up stuff in MacOS.

I just tested it on WSL (UBUNTU) under Windows, and it worked very very fast too.

[freedom6xiaobai]

mac Intel cpu, using mps, the effect is blurry, is there any good solution?

[jelly-liu]

same result, and the scarf of man does not match very well, a little bigger than source image.
but the OpenTalker/SadTalker render result works very well.

[Riotamus]

Thank you for the software improvement,
I have difficulties installing it on my PC because other applications are using different versions of Python, and I encountered various errors when trying to install it through Anaconda. So, I installed it through Automatic 1111, and it went smoothly even though I used a different version than the one supported by sadtalker.
I would like to emphasize that the application is only using the CPU, an AMD 7950x, without utilizing the GPU for rendering, a 3080 Nvidia with 10GB of memory.
Another detail that might be a bug is that after 3 to 5 minutes of video playback, it starts to become extremely slow, significantly reducing performance.
Currently, when using only the CPU, it works by 7 or 6 it/s, and near the end of the video, it drops to 1.7 it/s .
It's as if there's a cache that gets filled up. Making long videos is impossible; at most, I can create videos up to 3 minutes, and even then, as I approach the end, the performance deteriorates.

This is my usage experience, and I want to remind you that I'm using it as an extension on Automatic 1111 due to installation problems.

Again ty, i hope my experience helps you work.

[AIhasArrived]

@sahreen-haider,
does this have anything with what mentioned here ? (Rudrabha/Wav2Lip#584)
You said we have to use a new face detection model, and wondered if this topic was any relevant or not

[sahreen-haider]

Kinda the topic is if the face detection model is detecting the face from the frames efficiently and effectively, but using any other face detection model could also weigh on to the computational complexity, thus adding on to more running time.

[shisaq]

I'm using Apple Silicon M2 Pro and have an error showing this:

{'checkpoint': '/Users/shisaq/Coding/stable-diffusion-webui/extensions/SadTalker/checkpoints/SadTalker_V0.0.2_512.safetensors', 'dir_of_BFM_fitting': '/Users/shisaq/Coding/stable-diffusion-webui/extensions/SadTalker/src/config', 'audio2pose_yaml_path': '/Users/shisaq/Coding/stable-diffusion-webui/extensions/SadTalker/src/config/auido2pose.yaml', 'audio2exp_yaml_path': '/Users/shisaq/Coding/stable-diffusion-webui/extensions/SadTalker/src/config/auido2exp.yaml', 'pirender_yaml_path': '/Users/shisaq/Coding/stable-diffusion-webui/extensions/SadTalker/src/config/facerender_pirender.yaml', 'pirender_checkpoint': '/Users/shisaq/Coding/stable-diffusion-webui/extensions/SadTalker/checkpoints/epoch_00190_iteration_000400000_checkpoint.pt', 'use_safetensor': True, 'mappingnet_checkpoint': '/Users/shisaq/Coding/stable-diffusion-webui/extensions/SadTalker/checkpoints/mapping_00109-model.pth.tar', 'facerender_yaml': '/Users/shisaq/Coding/stable-diffusion-webui/extensions/SadTalker/src/config/facerender_still.yaml'}
load [net_G] and [net_G_ema] from /Users/shisaq/Coding/stable-diffusion-webui/extensions/SadTalker/checkpoints/epoch_00190_iteration_000400000_checkpoint.pt
/var/folders/t1/rhwc34s93xg6g89j3q92jdcm0000gn/T/gradio/tmpoqxbr4dr.png
landmark Det:: 100%|██████████████████████████████| 1/1 [00:00<00:00,  3.21it/s]
3DMM Extraction In Video:: 100%|██████████████████| 1/1 [00:00<00:00,  5.25it/s]
mel:: 100%|███████████████████████████████| 293/293 [00:00<00:00, 129483.83it/s]
audio2exp:: 100%|███████████████████████████████| 30/30 [00:00<00:00, 79.59it/s]
FaceRender::  75%|█████████████████████       | 221/293 [00:26<00:08,  8.25it/s]/AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSCore/Types/MPSNDArray.mm:126: failed assertion `[MPSNDArrayDescriptor sliceDimension:withSubrange:] error: the range subRange.start + subRange.length does not fit in dimension[2] (293)'
./webui.sh: line 256: 74502 Abort trap: 6           "${python_cmd}" -u "${LAUNCH_SCRIPT}" "$@"
/opt/homebrew/Cellar/[email protected]/3.10.10_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Nearly every time when FaceRender reaches 75%, it stops working.

The error was with using Stable Diffusion extension mode, but I succeeded in using command line mode. Any ideas?

[johnscode]

same on M2 Max running osx 13.6.4.

[ChebliAlamiAhmed]

any updates??

[olololoe110399]

We can check branch from Colab.
Example: https://colab.research.google.com/gist/olololoe110399/fa0adbfe47493592fc7148244c7583da/vid_api.ipynb
I think video generated not better more than main branch

[halahw]

same to you, it is much faster, but the effect is not as good as the master branch.

[pilsen]

This is awesome! Big thanks for the faster face render. You just made our AI project twice as fast ;) You rock!

[xzorix35]

hey guys, I am so new in coding an ai tools but I am learning. I am currently using sad talker as an extension of stable diffusion webui. I want to use pirenderer in sad talker but I could not make this installation. I couldn't do checkout to the new branch section, when I do it terminal says incurable. Also I could not do pirenderer installation section. I don't know which files should I extract or how to code it. My pc is Mac m3. If any of you succeed the installation please explain briefly. Thanks.

[N8wlk]

I'm stuck at the same point I think.

git checkout pirender
error: Your local changes to the following files would be overwritten by checkout:
app_sadtalker.py
Please commit your changes or stash them before you switch branches.
Aborting

last time i tried some approach, I had to reinstall all new. someone an idea what to do about this error? cant check out, so cant finish this.

[Riotamus]

Ok, i'm using the a1111, release 16 from august, with this release i can install sadtalker and get a web tab for sad talker. What i did with the rc 0.2 (new release of sadtalk) was, i backup sadtalker original on a1111, i replace sadtalk original with the new one. Just copy past and replace... ta da !!! work with all new improvements... when i'm going to create like a 1h audio i split the audio on 4 minutos maximum wav, and made a bat file to make all vídeos.
Exp: python inference.py --driven_audio 01.wav --source_image 08.png --preprocess resize --pose_style 1
You can personalize it with any more options like gfan or else. A1111 or sadtalker dont need to be active, in fact i turn off a1111 to gain more performance.
I noted that you can improve performance starting a1111 doing one avatar, and them turning off and starting the bat file. Long audios with sadtalker is horrible for me.
Another thing, NVIDIA release Audio2Face who do the same thing but more complex, and you will need loots of things but is perfect (https://www.youtube.com/watch?v=ScMb1f1M9yU)

[starchx]

First of all, this is awesome - so FAST, thank you!
However, for some reason, the video generated when using --facerender=pirender is not playable in the browser, it is playable using QuickTime though. Anyone knows a quick fix?

Example html page with the video generated:

<html>
    <body>
        <video src="video_generated_with_pirender.mp4" id="inputVideo" autoplay muted loop playsinline></video>
    </body>
</html>

[Danny0802]

Amazing speed, so fast!!!

[groovybits]

Is this able to be merged into the current version? I looked and seems like it may need to be re-merged? Also is this planned to be merged officially ever?

I really need this for my Mac and can't see the program as useful for me without it, since it is so slow otherwise on mps. The mps support would be very much appreciated.

[dgrosen]

I was able to run SadTalker using automatic1111 as an extension. However, it is only using CPU on my mac silicon M3.
I get fatal error when trying to checkout to the new branch in the terminal window. Is there a more detail explanation for a non-programmer? thanks

git fetch origin pull/458/head:pirender
git checkout pirender

fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git