Use FOMM offline with driving video by vassilispapadop · Pull Request #36 · sensity-ai/dot

vassilispapadop · 2022-07-20T11:45:19Z

Description

Allows DOT.FOMM model to be used offline (without camera) with a given driving video.

Changelog:

Added:

dot/commons/pose/head_pose.py. Estimates whether desired head pose is achieved given Pitch, Roll and Yaw angles.

Updated:

dot/commons/model_option.py
dot/fomm/option.py
dot/dot.py
dot/commons/video/video_utils.py
dot/commons/utils.py
dot/commons/model_option.py

Usage:

dot -c ./configs/fomm.yaml --target /path/to/driving/video --source /path/to/source/image.png --use_video --save_folder /output/folder/

ajndkr · 2022-08-04T10:29:28Z

@vassilispapadop please update PR title.

vassilispapadop · 2022-08-04T10:30:44Z

@vassilispapadop please update PR title.

i see that @giorgiop changed it yesterday

ajndkr · 2022-08-04T10:31:47Z

ah. I would still change it. we use PR titles to generate release notes.

ajndkr

can you please document the offline usage in one of the READMEs?

src/dot/commons/pose/head_pose.py

ajndkr · 2022-09-07T05:31:32Z

src/dot/commons/pose/head_pose.py

+                if (
+                    idx == 33
+                    or idx == 263
+                    or idx == 1
+                    or idx == 61
+                    or idx == 291
+                    or idx == 199
+                ):


you can define a list of these landmark indices and use if idx in landmark_list you can use an appropriate variable name here. I just used a random one as an example.

src/dot/commons/pose/head_pose.py

src/dot/commons/utils.py

src/dot/commons/video/video_utils.py

…ulator (#45) * Update config path * Update readme * Update Android emulator * Update CHANGELOG * Fix review issues

Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 3.20.1 to 3.20.2. - [Release notes](https://github.com/protocolbuffers/protobuf/releases) - [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/generate_changelog.py) - [Commits](protocolbuffers/protobuf@v3.20.1...v3.20.2) --- updated-dependencies: - dependency-name: protobuf dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…y-ai/dot into feature/avatarify-offline

vassilispapadop · 2022-11-29T14:08:41Z

Updated README for offline usage
2dfbeb2

README.md

ajndkr · 2022-12-05T09:35:17Z

src/dot/commons/pose/head_pose.py

+)
+mp_drawing = mp.solutions.drawing_utils
+
+# https://github.com/tensorflow/tfjs-models/blob/838611c02f51159afdd77469ce67f0e26b7bbb23/face-landmarks-detection/src/mediapipe-facemesh/keypoints.ts


is this link correct? I see no relation to HEAD_POSE_LANDMARKS.

@ajinkyaindulkar could you have a look here? google-ai-edge/mediapipe#1615
it doesn't seem to be well documented

can you add this link instead?

src/dot/commons/utils.py

ajndkr · 2022-12-05T09:37:44Z

src/dot/commons/video/video_utils.py

+mp_face = mp.solutions.face_detection.FaceDetection(
+    model_selection=0,  # model selection
+    min_detection_confidence=0.5,  # confidence threshold
+)


comment for later. we should find a better solution than global initialisation here.

vassilispapadop · 2022-12-08T12:04:39Z

FOMM example

vassilispapadop · 2022-12-08T12:07:20Z

@ajinkyaindulkar @Ghassen-Chaabouni could you review? Can someone test DOT with use_cam(not possible on my local)

ajndkr · 2022-12-08T16:35:48Z

@ajinkyaindulkar @Ghassen-Chaabouni could you review? Can someone test DOT with use_cam(not possible on my local)

Even I won't be able to test it.

Ghassen-Chaabouni · 2022-12-08T16:43:43Z

@ajinkyaindulkar @Ghassen-Chaabouni could you review? Can someone test DOT with use_cam(not possible on my local)

I will test it

Ghassen-Chaabouni · 2022-12-09T13:50:49Z

There is no option for use_cam, but fomm works fine with the cam. Just tested it

vassilispapadop added 3 commits July 20, 2022 11:12

use fomm offline

e3bf473

video_pipeline for fomm and simswap

3adbe40

unused var

f4fa12b

ajndkr assigned vassilispapadop Jul 27, 2022

ajndkr added the feature New feature or request label Jul 27, 2022

giorgiop changed the title ~~Feature/avatarify offline~~ Feature/FOMM offline Aug 3, 2022

vassilispapadop added 3 commits August 4, 2022 10:31

head pose estimation checks

97699dd

head pose mediapipe

1005fda

updated video_utils

c867951

vassilispapadop marked this pull request as ready for review August 4, 2022 08:48

merge

05ac2a5

vassilispapadop changed the title ~~Feature/FOMM offline~~ Use Avatarify offline with driving video Aug 4, 2022

giorgiop changed the title ~~Use Avatarify offline with driving video~~ Use FOMM offline with driving video Aug 26, 2022

ajndkr reviewed Sep 7, 2022

View reviewed changes

Ghassen-Chaabouni and others added 9 commits September 19, 2022 15:17

Update README and add instructions for running dot with an Android em…

d6f3039

…ulator (#45) * Update config path * Update readme * Update Android emulator * Update CHANGELOG * Fix review issues

Update .pre-commit-config.yaml

195dbe2

Merge branch 'feature/avatarify-offline' of https://github.com/sensit…

1f7d636

…y-ai/dot into feature/avatarify-offline

updated docstring and added reference

51f5b41

added HEAD_POSE_LANDMARKS

1ac970b

removed duplicate code

8ad1b13

value error message

b1a593d

README offline usage

2dfbeb2

vassilispapadop marked this pull request as draft November 29, 2022 14:54

ajndkr reviewed Dec 5, 2022

View reviewed changes

np.random.shuffle

fc291ba

video_pipeline: crop size optional

804b25b

vassilispapadop marked this pull request as ready for review December 8, 2022 11:07

vassilispapadop added 4 commits December 8, 2022 12:11

updated run without cam instructions

0dad44c

added use_gpu flag to docs

90d8aca

added head_pose flag: used only by FOMM

0d503d0

removed unnecessary code

e8cbc82

face landmarks ref/link

752be64

Ghassen-Chaabouni approved these changes Dec 9, 2022

View reviewed changes

vassilispapadop merged commit 8c67ee9 into main Dec 12, 2022

vassilispapadop deleted the feature/avatarify-offline branch December 12, 2022 08:09

Conversation

vassilispapadop commented Jul 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changelog:

Added:

Updated:

Usage:

Uh oh!

ajndkr commented Aug 4, 2022

Uh oh!

vassilispapadop commented Aug 4, 2022

Uh oh!

ajndkr commented Aug 4, 2022

Uh oh!

ajndkr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ajndkr Sep 7, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vassilispapadop commented Nov 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ajndkr Dec 5, 2022

Choose a reason for hiding this comment

Uh oh!

vassilispapadop Dec 8, 2022

Choose a reason for hiding this comment

Uh oh!

ajndkr Dec 8, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ajndkr Dec 5, 2022

Choose a reason for hiding this comment

Uh oh!

vassilispapadop commented Dec 8, 2022

Uh oh!

vassilispapadop commented Dec 8, 2022

Uh oh!

ajndkr commented Dec 8, 2022

Uh oh!

Ghassen-Chaabouni commented Dec 8, 2022

Uh oh!

Ghassen-Chaabouni commented Dec 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vassilispapadop commented Jul 20, 2022 •

edited

Loading

vassilispapadop commented Nov 29, 2022 •

edited

Loading