An Implementation of VTuber (Both 3D and Live2D) using Python and Unity. Supporting face movement tracking, eye blinking detection, iris detection and tracking and mouth movement tracking using CPU only.
Live2D Demo
UnityChan 3D Demo
- Facial landmarks detection and movement tracking supported by Facemesh by Mediapipe.
- Various facial expressions detection including eye blinking, iris and mouth movements.
- Running smooth 30 FPS with CPU only for the aformentioned features.
- Simple and clean UI for adjusting the sensibility of detection in Unity.
- Saveload mechanism to save and load your preferences in Unity.
- Including sample (Unity) projects for both 3D and Live2D models
- Detailed and thorough explanation videos (with EN and ZH subtitles) playlist
Due to Github file size limitation, most of the folders in the unity chan sample project are removed, yet the project can still run
| File | Description |
|---|---|
| main.py | The main program |
| facial_landmark.py | The module which is used to detect your face and generate the facial landmarks. |
| pose_estimator.py | The module which estimates your pose/ orientation of your head based on the landmarks. |
| stabilizer.py | Implementation of Kalman Filter to stabilize the values. |
| facial_features.py | Various facial features detection implementation, including blinking, iris detection and mouth movement. |
| model.txt | The points of the 3D Canonical model used in Mediapipe. Source file |
| UnityAssets | Whole Unity Projects (in packages) and Scripts for both 3D (UnityChan) and Live2D (Hiyori) models |
Using avatars for streaming, content creation and VR gaming has been gaining increasing popularity, especially the boom of Hololive and other related companies active apperances in social media platforms such as YouTube and Twitter. Curious about the technology behind, I create this project after multiple researches.
Existing projects rely on Dlib, which although providing reliable and accurate facial landmark detection, requires decent graphic cards to run. However, implemented with the recent FaceMesh model in Mediapipe, accurate detection and tracking can be run smoothly using CPU only, making running on computers with mediocre graphic cards or laptops with integrated graphic cards possible.
Clone this project into your directory
git clone https://github.com/mmmmmm44/VTuber-Python-Unity.git
cd "VTuber-Python-Unity"
Download the unity packages of sample projects of both live2D and unitychan 3D in the link next to: Dropbox
(last edit of the sample projects: 23-12-2021)
-
Create An empty Unity 3D Project
-
Import either the Live2D or UnityChan3D package to your project. The corresponding SDKs have been included already.
Links for both packages (in a zip file): Dropbox
last edit: 23-12-2021
-
Run the Scene.
-
Run the following code in terminal [content in the bracket is optional]
python main.py --connect [--debug]
- Enjoy
Video Walkthrough: Click Me!
-
Download the Cubism SDK For Unity from this website and the sample model used (桃瀬ひより) from this website
-
Create an empty Unity 3D project, and import the Cubism SDK. Unzip the model and drag the whole folder to the Project window of the Unity Project.
-
Drag the live2D model's prefab into the scene. Run the scene immediately to allow the model to be shown in Scene and Game window.
-
Adjust the camera's position, background and projection properties. If there are some werid projection problems of the model, changing the projection of the camera from Perspective and Orthographic works for me.
-
Drag the HiyoriController.cs to the Hiyori GameObject. Adjust the parameters in the inspector
-
Run the scene.
-
Run the following code in terminal [content in the bracket is optional]
python main.py --connect [--debug]
- Enjoy
Video Walkthrough: Click Me!
-
Download the UnityChan model from the website. Go to "Data Download", accept the terms and agreements, and select the first one. Unzip the file.
-
Create an empty Unity 3D Project. Drag the unzipped folder to the Project Window of the project.
-
Go to UnityChan\Prefabs and Drag the "unitychan" prefab into the scene.
-
Adjust the camera's position, background and field of view.
-
Drag the UnityChanControl.cs script onto the prefab. Change the update mode of the Animator attached to "Animate Physics" and the Controller to UnityChanLocomotions. (Crucial) Adjust the variables in the inspector. Disable other attached scripts except AutoBlink and UnityChanController. You may tick the box "is Auto Blink Active" in UnityChanContoller to enable auto blinking (enable AutoBlink script when ticked).
-
Run the scene first
-
Run the following code in terminal [content in the bracket is optional]
python main.py --connect [--debug]
- Enjoy
Make sure you run the Unity Scene first before running the python script
Both the python scripts and the unity project can be exported to .exe, which can be run on other computers.
-
Python 3.8.5
-
Numpy 1.19.2
-
OpenCV 4.5.1
-
Mediapipe 0.8.9.1
-
Unity 2020.3.12f1
(Later version should be supported as well)
(For Windows, it is recommended to run this project using Anaconda and create a virtual environment before installing such packages.)
The whole project is run on a laptop with Intel Core i5-8250U, with 16GB RAM and integrated graphic card only.
The same project is tested on a M1 Max device, running in rosetta.
Eye motion tracking - Opencv with Python - Pysoruce
| Project | Author | LICENSE |
|---|---|---|
| head-pose-estimation | Yin Guobing | LICENSE |
| VTuber_Unity | AI葵 | LICENSE |
| VTuber-MomoseHiyori | KennardWang | LICENSE |
Hiyori Momose's model
| Position | Creator |
|---|---|
| Illustration | Kani Biimu [Twitter @kani_biimu] |
| Modeling | Live2D Inc. |
MIT
The Unity Chan model in the Unity Packages provided is distributed under Unity-Chan License © Unity Technologies Japan/UCL. A seperate sets of that License is included in UnityAssets/Licenses/UCL2_0