Volume 5, Issue 9, September – 2020 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165

AI-Based Yoga Pose Estimation for Android

Application
Girija Gireesh Chiddarwar, Abhishek Ranjane, Mugdha Chindhe, Rachana Deodhar, Palash Gangamwar
Computer Department, SCOE, Vadgaon,
Pune-411041, India

Abstract:- The importance of yoga is renowned II. RELATED WORK

worldwide and its health benefits, which were preached
by ancient sages, have stood the test of time. Even though  Deep Pose
yoga is becoming preeminent, there are important DeepPose was the first major paper[1], published in
challenges faced while doing yoga such as performing it CVPR 2014 that applied Deep Learning to Human pose
with incorrect form, the classes being expensive and the estimation. It achieved SOTA performance and beat existing
shortage of time in our busy lives. Computer vision models back in the year 2014. The model has an AlexNet
techniques exhibit promising solutions for human pose backend and estimated pose in a holistic fashion, i.e certain
estimation. However, these techniques are seldom poses can be estimated even if some joints are hidden when
applied in the domain of health or exercise, with no the pose is reasoned holistically. The paper applies Deep
literature or projects cited specifically for yoga. This Learning (CNN) to pose estimation and kicked off research
paper surveys the various technologies that can be used in this direction. The model used regression for XY
for pose estimation and concludes the best method based locations for certain regions. This added complexity and
on the usability for an android application. The paper weakened generalization hence performing poorly.
then discusses the methodology that will be used to
deploy the yoga pose estimation on an android  Efficient Object Localization Using Convolutional
application, how the app is modeled and the working of Networks
each component is explained. This approach in this paper[2] uses heatmaps by sliding
through multiple subsets of the image (windows) in parallel
Keywords:- AI; Yoga; Deep Learning; PoseNet; Android. to simultaneously capture features at a variety of scales. A
heatmap predicts the probability of one of the essential
I. INTRODUCTION points. This model is very successful for pose detection and
is used in current models as well.
Deep learning techniques have proved their importance
for object detection tasks. The same models can be
effectively used to detect the various essential body parts and
estimate the pose of the user in real-time. The paper surveys
the different options that can be used for pose estimation
such as OpenPose, Posenet, DeepPose, and concludes which
technique should be used for deploying it for an android
application for doing yoga.

There are several methods that can be used for pose

estimation. We shall discuss some of the most important
ones. In this paper we shall discuss the evolution of human
pose estimation over the years and how Posenet is the most
suitable for our project (yoga pose estimation on android). Fig 1:- Working of Sliding window detector
The project provides real-time pose estimation for yoga pose
estimation and correction on the client-side. This is achieved Heatmaps work better than direct joint regression. This
with Tensorflow. The library allows the inference of models model, however, does not include structure modeling, that
on Android, which makes it faster and more privacy is, taking into consideration the human body structure, such
respecting. Posenet is an open-sourced technology that as body part proportion, symmetries, joint limits, among
allows us to extract the 17 essential points natively. A others.
skeleton of the human pose is drawn with the help of these
points, which is then used to derive angles between these  Convolutional Pose Machines
points thus enabling us to effectively correct the user’s yoga This interesting paper[3] proposes the serial usage of
poses. This methodology is used in our Android application stages to predict. a CPM (Convolutional Pose Machine)
along with Google’s text-to-speech and speech-to-text consists of an image feature computation module followed
modules for the user to do yoga very effectively. by a prediction module. The multiple stages can be trained

IJISRT20SEP704 www.ijisrt.com 1070

Volume 5, Issue 9, September – 2020 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
end to end. Stage 1 generates heatmaps and image evidence by one to form more stages and connect the multi-resolution
as input for stage 2. Stages > 2 are just repetitions of the subnetworks in parallel.
second stage.
Repeated multi-scale fusions are conducted by
exchanging information across parallel multi-resolution
subnetworks over and over through the whole process.
Another pro is that this architecture does not use
intermediate heatmap supervision. Heatmaps are regressed
using an MSE loss.

 Deep High-Resolution Representaion Learning for

Human Pose Estimation
The HRNet (High-Resolution Network) model [5]
maintains a high-resolution representation throughout the
whole process, instead of high → low → high-resolution
representation, and this works very well. The architecture
starts from a high-resolution subnetwork as the first stage,
and gradually adds high-to-low resolution subnetworks one
Fig 2:- Effective Receptive Field by one to form more stages and connect the multi-resolution
subnetworks in parallel.
 OpenPose
Realtime Multi-Person 2D Pose Estimation using Part Repeated multi-scale fusions are conducted by
Affinity Fields[4] proposes the detection of multiple people exchanging information across parallel multi-resolution
in an image. It uses an approach called non-parametric subnetworks over and over through the whole process.
representation, also known as Part Affinity Fields (PAFs). Another pro is that this architecture does not use intermediate
The architecture encodes a global context, allowing a greedy heatmap supervision. Heatmaps are regressed using an MSE
bottom-up parsing step that maintains high accuracy while loss.
achieving real-time performance, irrespective of the number
of people in the image. The method got SOTA performance  Deep High-Resolution Representaion Learning for
on the MPII Multi-Person benchmark. Human Pose Estimation
The paper [6] by George Papandreou et al. presented a
The bottom-up approach decouples runtime complexity box-free bottom-up approach that is used in PoseNet.
from the number of people in the image. It uses Part Affinity PoseNet is an open-sourced application that can be deployed
Fields(PAFs), a set of 2D vector fields that encode the using Tensorflow. PoseNet gives an output of poses, pose
location and orientation of limbs over the image domain. confidence scores, keypoint positions, and keypoint
confidence scores from the model outputs. It uses the best
practices from some of the previous papers and gives very
high accuracy for the minimal cost it requires.

Fig 3:- Architecture of the Two-branch Multi-stage CNN

The problem with OpenPose is that it requires special

hardware and can not perform well on mobile devices.

 Deep High-Resolution Representation for Human Pose Fig 4:- Architecture of the Two-branch Multi-stage CNN
Estimation.
The HRNet (High-Resolution Network) model [5] PoseNet can be configured with the output stride.
maintains a high-resolution representation throughout the PoseNet is invariant of the size of the image presented, by
whole process, instead of high → low → high-resolution downscaling the image by a factor of this output stride. The
representation, and this works very well. The architecture higher the output stride the lower the accuracy, but the faster
starts from a high-resolution subnetwork as the first stage, the speed, which is key in the android implementation.
and gradually adds high-to-low resolution subnetworks one Altrous convolution [7] is then used to enable the

IJISRT20SEP704 www.ijisrt.com 1071

Volume 5, Issue 9, September – 2020 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
convolution filters in the subsequent layers to have a wider
field of view. PoseNet reapplies the concept of Heatmaps
from Efficient Object Localization Using Convolutional
Networks [2]. It also uses offset vectors that correspond in
location to the heatmaps points, and are used to predict the
exact locations of the keypoints by traveling along the vector
from the corresponding heatmap point.

PoseNet offers both single and multiple pose

estimations. Both of these techniques used different
approaches. Single pose estimation is much faster and
efficient, thus making it suitable to use on an android device.
PoseNet is a 2D pose estimation technique, which only does
the job of giving the XY coordinates as opposed to XYZ
coordinates present in 3D methods. PoseNet also does not Fig 6:- Pre-processing part of the system.
require special hardware, unlike OpenPose[4]. Thus, we can
conclude that PoseNet is the most suitable method that can be In this module, the target values are extracted in the
used for pose estimation in a mobile application. following steps:
 We collect a single ideal image for getting the keypoints.
III. METHODOLOGY We store these locally in our machine.
 We use OpenCV to predict the 17 essential keypoints
The methodology will be discussed in 2 modules using a pretrained model, OpenPose [4], that uses the 2-
namely the pre-processed part if the system and the Native branch multi-stage CNN.
(Real-Time) part of the system. The purpose of preprocessing  The keypoints are noted to be later hard-coded as the
is to get target values for the poses that need to be performed. target values in the next phase.
The Native part deals with the programs that run on the
android device, without any external interaction. The task of  Native (Real-Time) application
the Native part is to predict the actual poses done by the user,
in real-time, and provide the user with interfaces to do yoga
efficiently.

Fig 7:- Native part of the system

The android app natively provides a GUI, takes the user

video, predicts the angles using PoseNet, calculates the
angles and displays the results after comparison. These
processes happen in the following steps:
 In the camera activity, the user is asked for permission to
Fig 5:- methodology of the System access the phones camera. The video is taken and every
fifth frame is used by converting it’s resolution to 224 x
Finally, by using techniques, later discussed in the 224.
paper, the keypoints given by Pre-processing part and the  The frames are passed into the Posenet Class, where
Native part are compared to get the measure of correctness of various hyper-parameters are set and the frames are
the yoga pose performed by the user. passed through the PoseNet model. The output consists of
the coordinates of the 17 essential keypoints of the body.
 Preprocessing Part of the system  The PoseNet model can be easily deployed on an android
The objective of this module is to get the target values, app by using TensorFlow-Lite. The trained model is
that is the ideal values, of each yoga pose to use as a metric stored in a “.tflite” format and can be inferred to get the
to determine the correctness of actual yoga pose later. 17 essential points.

IJISRT20SEP704 www.ijisrt.com 1072

Volume 5, Issue 9, September – 2020 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
 The skeleton of the user, based on the keypoints is easily implemented by using the TensorFlow-Lite
plotted. This is done to give the user a more intuitive framework. The pre-trained model can be inferred to get the
experience. 17 essential points. We use these points to calculate the
 The distance between each body part is calculated, using angles and compare them to the correct pose angles
Euclidean Distance. Since, the distance might vary from calculated by OpenCV. The skeleton and wrong angles are
person to person or depending on the phone-user distance, displayed. Thus, This paper substantially provided a glimpse
we calculate the angles for comparison. of how the methods of pose estimation have evolved over
 The angles are calculated using the cosine law of triangles the years, and how they can be effectively used in many
for both- the target values and the actual values. The applications, such as yoga pose correction.
mathematical functions are performed using the Math
module in android. ACKNOWLEDGMENT

We would also like to take this opportunity to thank out

guide, Mrs. G. G. Chiddarwar, for helping us throughout.
Also, to our review committee member Mrs. J. B. Kulkarni
for invaluable advice and critique. Furthermore, we would
like to express gratitude and earnestly acknowledge out
parents for supporting us and friends for encouraging us. To
Prof. M. P. Wankhade, our Head of Department and our
principal, Dr. S. D. Lokhande for backing us.

 For the GUI, we use a navigation drawer for quickly REFERENCES

navigating the app. We include an Home page, Instruction
page, and About Us page. [1]. Alexander Toshev and Christian Szegedy, “DeepPose:
 We use the ImageView Layout along with drawables to Human Pose Estimation via Deep Neural Networks,”
display the frames, skeleton, the angles and correctness. The IEEE Conference on Computer Vision and Pattern
 We have also added Google’s text-to-speech and speech- Recognition (CVPR), 2014, pp. 1653-1660
to-text modules for the user to be able to navigate the app [2]. Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann
with interruption. This Voice-controlled Yoga activity is LeCun, Christoph Bregler New York
enabled by KontinousSpeechRecognizer module that University,“Efficient Object Localization Using
allows us to get user sound input and process it Convolutional Networks,” The IEEE Conference on
continuously. Computer Vision and Pattern Recognition (CVPR),
 Thus, by using these modules, the user can say, ”start 2015, pp. 648-656
second pose” to start doing the second pose and will be [3]. Shih-En Wei, Varun Ramakrishna, Takeo Kanade,
prompted by “Pose correctly performed” when they hit Yaser Sheikh, “Convolutional Pose Machines,” The
the target values for the pose. IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2016, pp. 4724-4732.
IV. FUTURE DIRECTION [4]. Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh,
“Realtime Multi-Person 2D Pose Estimation Using Part
 Gesture recognition: This feature has not been Affinity Fields,” The IEEE Conference on Computer
implemented yet. However, since PoseNet already gives Vision and Pattern Recognition (CVPR), 2017, pp.
the coordinates of various body parts, it will be fairly easy 7291-7299.
to implement a gesture recognition system to navigate the [5]. Ke Sun, Bin Xiao, Dong Liu, Jingdong Wang, “Deep
app. The app would allow the user to navigate and control High-Resolution Representation Learning for Human
the app by using gesture performed by moving their head Pose Estimation,” The IEEE Conference on Computer
or hands. Vision and Pattern Recognition (CVPR), 2019, pp.
5693-5703
 Progress tracking: This feature will store the measures of
the correctness of the user’s yoga poses. Giving users the [6]. George Papandreou, Tyler Zhu, Liang-Chieh Chen,
Spyros Gidaris, Jonathan Tompson, Kevin Murphy,
ability to monitor their progress will not only be
“PersonLab: Person Pose Estimation and Instance
informative to them of their improvement but it will also
Segmentation with a Bottom-Up, Part-Based,
keep the user motivated to keep on doing yoga.
Geometric Embedding Model,” The European
Conference on Computer Vision (ECCV), 2018, pp.
V. CONCLUSION
269-286
[7]. Liang-Chieh Chen, Yukun Zhu, George Papandreou,
Deep Learning methods have proved to be extremely
Florian Schroff, Hartwig Adam,“Encoder-Decoder with
useful for pose estimation, compared to any other methods.
The diverse applications of pose estimation have provoked Atrous Separable Convolution for Semantic Image
many advancements in this field, both in terms of speed and Segmentation” The European Conference on Computer
accuracy. We have concluded that PoseNet is, by the current Vision (ECCV), 2018, pp. 801-818.
standards, the best technique for implementing mobile
applications, specifically for yoga. This method can be

IJISRT20SEP704 www.ijisrt.com 1073