Many new features are launching in SDWEBUI extension and standalong gradio demo

[vinthony]

Hi, everyone!

Thanks for your patience with the bugs and long-time updates in SadTalker.

We are releasing some new features in SadTalker for the WEBUI.

You can try to install the dev version for these new features:

git clone -b dev https://github.com/OpenTalker/SadTalker

OR in SDWEBUI:

cd extensions/SadTalker && git checkout dev

It will be merged into the main code after careful testing.

1. The IDLE mode as discussed in #280 (comment).

Now, you can simply generate the video with a simple image from SD or by yourself with the length of the video as input.
Where the head pose will be generated automatically as well as the eye blinking.

2. Ref Video mode.

It also allows a new video as reference pose and eye blink and FULLY MOTION!

• As for the reference pose and eye blink, the generated video still needs the audio from IDLE/AUDIO FILE/TTS.
• As for the full motion mode(all), the generated video will use the audio from the input video and generate the same motion as the original video.

3. Controllable Blinking

We set a new use blink checkbox in settings to control the eye blink explicitly. The generated video in FULL mode may work fine without an eye blink.

Have fun and let me know if you have a better idea or problem with these updates.

[james-s-tayler]

I pulled the latest version of the extension but I don't see these features? There's no UI for enabling the blink checkbox, and I can't see anything about reference video either? Am I missing something?

[vinthony]

to avoid bugs i have make a dev branch to test these new features. you may need to checkout the dev branch as below:

[james-s-tayler]

I already looked at the dev branch and saw it only contained a single commit with a single one line change, so I didn't bother checking it out because it only added a print statement. Have the changes actually been pushed to that branch?

[vinthony]

Sure, the code are here in dev branch:

SadTalker/app.py

Lines 45 to 100 in 4c38d1f

	with gr.Tabs(elem_id="sadtalker_driven_audio"):
	with gr.TabItem('Driving Methods'):
	gr.Markdown("Possible driving combinations: <br> 1. Audio only 2. Audio/IDLE Mode + Ref Video(pose, blink, pose+blink) 3. IDLE Mode only 4. Ref Video only (all) ")
	with gr.Row():
	driven_audio = gr.Audio(label="Input audio", source="upload", type="filepath")
	driven_audio_no = gr.Audio(label="Use IDLE mode, no audio is required", source="upload", type="filepath", visible=False)
	with gr.Column():
	use_idle_mode = gr.Checkbox(label="Use Idle Animation")
	length_of_audio = gr.Number(value=5, label="The length(seconds) of the generated video.")
	use_idle_mode.change(toggle_audio_file, inputs=use_idle_mode, outputs=[driven_audio, driven_audio_no]) # todo
	if sys.platform != 'win32' and not in_webui:
	with gr.Accordion('Generate Audio From TTS', open=False):
	from src.utils.text2speech import TTSTalker
	tts_talker = TTSTalker()
	with gr.Column(variant='panel'):
	input_text = gr.Textbox(label="Generating audio from text", lines=5, placeholder="please enter some text here, we genreate the audio from text using @Coqui.ai TTS.")
	tts = gr.Button('Generate audio',elem_id="sadtalker_audio_generate", variant='primary')
	tts.click(fn=tts_talker.test, inputs=[input_text], outputs=[driven_audio])
	with gr.Row():
	ref_video = gr.Video(label="Reference Video", source="upload", type="filepath", elem_id="vidref").style(width=512)
	with gr.Column():
	use_ref_video = gr.Checkbox(label="Use Reference Video")
	ref_info = gr.Radio(['pose', 'blink','pose+blink', 'all'], value='pose', label='Reference Video',info="How to borrow from reference Video?((fully transfer, aka, video driving mode))")
	ref_video.change(ref_video_fn, inputs=ref_video, outputs=[use_ref_video]) # todo
	with gr.Column(variant='panel'):
	with gr.Tabs(elem_id="sadtalker_checkbox"):
	with gr.TabItem('Settings'):
	gr.Markdown("need help? please visit our [[best practice page](https://github.com/OpenTalker/SadTalker/blob/main/docs/best_practice.md)] for more detials")
	with gr.Column(variant='panel'):
	# width = gr.Slider(minimum=64, elem_id="img2img_width", maximum=2048, step=8, label="Manually Crop Width", value=512) # img2img_width
	# height = gr.Slider(minimum=64, elem_id="img2img_height", maximum=2048, step=8, label="Manually Crop Height", value=512) # img2img_width
	with gr.Row():
	pose_style = gr.Slider(minimum=0, maximum=45, step=1, label="Pose style", value=0) #
	exp_weight = gr.Slider(minimum=0, maximum=3, step=0.1, label="expression scale", value=1) #
	blink_every = gr.Checkbox(label="use eye blink", value=True)
	with gr.Row():
	size_of_image = gr.Radio([256, 512], value=256, label='face model resolution', info="use 256/512 model?") #
	preprocess_type = gr.Radio(['crop', 'resize','full', 'extcrop', 'extfull'], value='crop', label='preprocess', info="How to handle input image?")
	with gr.Row():
	is_still_mode = gr.Checkbox(label="Still Mode (fewer head motion, works with preprocess `full`)")
	batch_size = gr.Slider(label="batch size in generation", step=1, maximum=10, value=1)
	enhancer = gr.Checkbox(label="GFPGAN as Face enhancer")
	submit = gr.Button('Generate', elem_id="sadtalker_generate", variant='primary')
	with gr.Tabs(elem_id="sadtalker_genearted"):

[jet3004]

I pulled the latest version of the extension but I don't see these features? There's no UI for enabling the blink checkbox, and I can't see anything about reference video either? Am I missing something?

I pulled the latest version of the extension but I don't see these features? There's no UI for enabling the blink checkbox, and I can't see anything about reference video either? Am I missing something?

I'm having the exact same thing, stoked for these features, did cd extensions/SadTalker && git checkout dev and all but no new interface, after restarts, relaunching, trashing, and reinstalling. Not sure what to do @vinthony...? Thanks for your work on this.

[vinthony]

Have you see this information when you switch the branch?

[jjmlovesgit]

Nice --- and of course I had just done one myself to let me add the pose videos and pose style :) Guess I will not be needing this Afterall!

I also was messing with the Yaw, Pitch, and Roll settings... I eventually landed on something I like for my use case where I use a virtual human to walk my code on my free Youtube. This guy is a great assistant as he never stumbles or mumbles during long code reviews... it definitely is good result off a single still image. An incredible job by the team involved here.

Here is my baseline I am trying to emulate (Note the watermark as this is paid generation):
https://github.com/OpenTalker/SadTalker/assets/47751509/4301911e-dc1d-4a1c-93a8-f94169af3661

Here is what I was able to generate (This is free SadTalker done locally):
https://github.com/OpenTalker/SadTalker/assets/47751509/8bbce1a9-44d3-433e-a9d3-4004161e4b7f

[ldavis9000aws]

If you're using yaw, pitch, and roll, it sounds like that is for the CLI version while most folks utilize the WebUi that doesn't have those options.

[G-force78]

Cool, good to see things get integrated and driving videos possible too like in video-retalking
I did get this error on first run

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 437, in run_predict
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1352, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1077, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/content/SadTalker/src/gradio_demo.py", line 107, in test
    ref_video_coeff_path, _, _ =  self.preprocess_model.generate(ref_video, ref_video_frame_dir, preprocess, source_image_flag=False)
  File "/content/SadTalker/src/utils/preprocess.py", line 124, in generate
    lm = self.propress.predictor.extract_keypoint(frames_pil, landmarks_path)
  File "/content/SadTalker/src/face3d/extract_kp_videos_safe.py", line 39, in extract_keypoint
    current_kp = self.extract_keypoint(image)
  File "/content/SadTalker/src/face3d/extract_kp_videos_safe.py", line 57, in extract_keypoint
    bboxes = bboxes[0]
IndexError: index 0 is out of bounds for axis 0 with size 0

Ok so i changed it from pose and blink to full , reduced pose scale and it started generating but then got this error

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 437, in run_predict
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1352, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1077, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/content/SadTalker/src/gradio_demo.py", line 139, in test
    return_path = self.animate_from_coeff.generate(data, save_dir,  pic_path, crop_info, enhancer='gfpgan' if use_enhancer else None, preprocess=preprocess, img_size=size)
  File "/content/SadTalker/src/facerender/animate.py", line 215, in generate
    sound = AudioSegment.from_file(audio_path)
  File "/usr/local/lib/python3.10/dist-packages/pydub/audio_segment.py", line 651, in from_file
    file, close_file = _fd_or_path_or_tempfile(file, 'rb', tempfile=False)
  File "/usr/local/lib/python3.10/dist-packages/pydub/utils.py", line 60, in _fd_or_path_or_tempfile
    fd = open(fd, mode=mode)
FileNotFoundError: [Errno 2] No such file or directory: './results/5ef6ef02-53f6-4e75-9263-4ca6d8c07299/Patrick Stewart is So Proud of Star Trek Picard Season 3 After Initial Worries - TheWrap Magazine_Clip 2.mp4.wav'

Here are my settings used

I have now been able to generate by using these settings. I did also add some code to enable gradio to setup tmp files, no idea if that did anything

def upload_files(input_file):
    # Create a temporary folder
    tmp_folder = tempfile.mkdtemp()

    # Save the uploaded file to the temporary folder
    input_file.save(os.path.join(tmp_folder, input_file.name))

    # Process the uploaded file
    # ...

    # Return the result
    return result

Settings used for successful result

[arezaei2002]

can i see your hole code ??

[G-force78]

These are great examples how did you do these? https://user-images.githubusercontent.com/4397546/224310754-665eb2dd-aadc-47dc-b1f9-2029a937b20a.mp4 What settings correspond to in the WebUI?

Also what is the setting extfull?

[qusonny]

Is the video in the link above made with Sadtalker? Why is it that when I make the video, the head and body always separate? Unless still mode is selected, but still mode head movement is too little for what I want.

[barbarabax]

Love it! Thank you

[G-force78]

Is the video in the link above made with Sadtalker? Why is it that when I make the video, the head and body always separate? Unless still mode is selected, but still mode head movement is too little for what I want.

Certainly not from scratch it looks like its just been lip synced using something like wav2lip

[MetaFomos]

Can I use any portrait images as a source image for sad talker?
Please let me know @vinthony
Regards.

[MetaFomos]

@vinthony , please answer above question.
Thanks.

[Ailen122]

为什么我生成的人物口型没有牙齿

[slavakurilyak]

When can we expect this dev code to be merged into main?

[Nate82]

For anyone else still getting the old UI when you have the latest version installed:

cd extensions/SadTalker && git checkout dev

The above line did not work for me, the old UI was still showing.

So I deleted every SadTalker folder/file and ran the following inside \stable-diffusion-webui\extensions

git clone -b dev https://github.com/OpenTalker/SadTalker

That downloaded the dev branch directly and created a SadTalker folder inside \stable-diffusion-webui\extensions

Then I created a checkpoints folder inside the SadTalker folder and downloaded the 4 files from the latest release on https://github.com/OpenTalker/SadTalker/releases

Now the new updated UI is showing in Automatic1111

[keith-maltman]

Hi there,

I'm sorry if this is in the wrong place. I'm VERY new to this. After 3 full days of trying to program to install and run in Ubuntu (Every single set of instructions on the internet AND YOUR OWN has AT LEAST one mistake. (your .sh script does not work in ubuntu BTW), and there is a path error in the manual instructions, (that still did not work once I fixed it?) Anyway I finally managed to work it out the hard way, and get it to install. Only to encounter this message after it attempted to run the first time. New as I am, this time I suspect it's not the install procedure, (that I painfully worked out myself). This time I suspect it something in the app.

stderr: ERROR: Exception:
Traceback (most recent call last):
File "/home/lucifer/anaconda3/envs/sadtalker/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
File "/home/lucifer/anaconda3/envs/sadtalker/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
return func(self, options, args)
File "/home/lucifer/anaconda3/envs/sadtalker/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 377, in run
requirement_set = resolver.resolve(
File "/home/lucifer/anaconda3/envs/sadtalker/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 95, in resolve
result = self._result = resolver.resolve(
File "/home/lucifer/anaconda3/envs/sadtalker/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
File "/home/lucifer/anaconda3/envs/sadtalker/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 457, in resolve
raise ResolutionTooDeep(max_rounds)
pip._vendor.resolvelib.resolvers.ResolutionTooDeep: 200000

Anyone know what this error is and how to fix it, I suspect it to do with TTS, but I have only been using Linux and python for a couple of weeks.

Keith

Sorry again if this is in the wrong place

[saurytang]

启动webuui的时候发生异常，你那个gradio用的那个版本呀？
发生异常: AttributeError
'Row' object has no attribute 'style'
File "D:\datahuman\SadTalker\app_sadtalker.py", line 35, in sadtalker_demo
with gr.Row().style(equal_height=False):
File "D:\datahuman\SadTalker\app_sadtalker.py", line 113, in
demo = sadtalker_demo()
AttributeError: 'Row' object has no attribute 'style'

[1joshw]

can anyone tell me how to fix

[saurytang]

pip install gradio==3.41.2 saury ***@***.***  

…

------------------ 原始邮件 ------------------ 发件人: "OpenTalker/SadTalker" ***@***.***>; 发送时间: 2024年5月12日(星期天) 凌晨1:02 ***@***.***>; ***@***.******@***.***>; 主题: Re: [OpenTalker/SadTalker] Many new features are launching in SDWEBUI extension and standalong gradio demo (Discussion #386) can anyone tell me how to fix ss.png (view on web) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[1joshw]

thanks, but now.

[saurytang]

saury ***@***.***  

…

------------------ 原始邮件 ------------------ 发件人: "OpenTalker/SadTalker" ***@***.***>; 发送时间: 2024年5月12日(星期天) 晚上11:39 ***@***.***>; ***@***.******@***.***>; 主题: Re: [OpenTalker/SadTalker] Many new features are launching in SDWEBUI extension and standalong gradio demo (Discussion #386) thanks, but now. ss.png (view on web) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>