Sign language Converter

Project Phase II Report

On

“Sign Language Converter”

Submitted for the requirement of

Project course

BACHELOR OF ENGINEERING

COMPUTER SCIENCE & ENGINEERING

Submitted to: Er. Gauri Shankar Submitted By:

Project Teacher (Supervisor): Abhishek Sharma(20BCS9162)
Er. Sumit Malhotra(E7822) Anant Tripathi(20BCS9167)
Aditya Raj(20BCS9168)
Abhishek Pandey(20BCS9188)

Co Supervisor Signature
Er. Gauri Shankar(E11266)

DEPARTMENT OF COMPUTER SCIENCE & ENGINEERING

CHANDIGARH UNIVERSITY, GHARUAN
June 2022

i | Page
 Sign language Converter

ABSTRACT

Considering the barriers faced by pupils of the speech impaired community we aim to introduce a
tool which will bridge the communication gap and aid better interaction. In situations where a
vocal individual is unskilled with sign language, the need for a sign-interpreter is inevitable in
order to establish an interplay of expounding’s. We propose a system that enables a two-way
conversation between the speech-impaired and other vocal individuals. In this paper we present a
prototype which is effective in two phases. In the first phase, the sign-language gestures are fed
into the system in real-time through computer vision capabilities of the device. These gestures are
then recognized with the help of our Deep Neural Network while the hand detection is crisped
upon with an edge detection algorithm interpreting it in the text as well as audio format. The second
phase accomplishes to convert audio into text and eventually displays relevant hand-gestures for
the same. The system is capable of recognizing over 300 words gestured by the Indian Sign
Language.

Keywords - Speech Recognition, Sign Language, Natural Language Processing, Computer Vision

2 | Page
 Sign language Converter

TABLE OF CONTENT

1. Introduction 04
2. Literature Review 05-08
3. Objective 09
4. Methodology 10-15
5. System Requirements 16
6. Results 17
7. Conclusion and Future works 17
8. References 18

3 | Page
 Sign language Converter

1. INTRODUCTION

Normal people will find it difficult to communicate with hearing-impaired people. To break
the barrier of communication between normal and hearing-impaired people, the Speech to
Sign Language translator is used. This translator makes the interaction simpler and faster
for normal people to convey their ideas to hearing-impaired people. The translator converts
speech or text to Sign Language using Natural Language Processing algorithms. Thus, the
system is used to overcome the hurdles faced by normal people to share their thoughts with
hearing impaired people and it will be an ear for the hearing-impaired.

A. Existing System:
The existing systems focus on the translation by each letter in a word and translation
to American Sign Language (ASL). It is difficult for the Indian deaf people to understand
other countries Sign Language like American Sign Language (ASL), British Sign
Language (BSL), New Zealand Sign Language, and many more.

B. Proposed System:
Our proposed system is designed to overcome the troubles faced by the Indian deaf
people. This system is designed to translate each word that is received as input into sign
language. This project translates the words based on Indian Sign Language.
• Natural Language Processing – The filler words such as ‘is’, ‘are’, ‘was’, ‘were’
etc., are words that hardly contribute to the context in sign language conversion. Therefore,
the system removes those filler words from the speech or sentence.
• Root Words – The words may be in gerund form, plural form, or adjective form.
The proposed system will remove these forms of the words and find the root word from
those words. These root words will be helpful in the effective conversion into sign
language.
• Dataset – The system has a large dataset of Indian sign language words to map
according to the text or text recognized from the speech. So, it will be helpful to all deaf
people in India. It makes the people understand most of the speech or text.

4 | Page
 Sign language Converter

2. LITERATURE REVIEW

The researches done in this field are mostly done using a glove-based system. In the glove-
based system, sensors such as potentiometer, accelerometers etc. are attached to each of
the finger. Based on their readings the corresponding alphabet is displayed. Christopher
Lee and Yangsheng Xu developed a glove-based gesture recognition system that was able
to recognize 14 of the letters from the hand alphabet, learn new gestures and able to update
the model of each gesture in the system in online mode. Over the years advanced glove
devices have been designed such as the Sayre Glove, Dexterous Hand Master and Power
Glove. The main problem faced by this gloved based system is that it has to be recalibrate
every time whenever a new user on the finger-tips so that the fingertips are identified by
the Image Processing unit. We are implementing our project by using Image Processing.
The main advantage of our project is that it is not restricted to be used with black
background. It can be used with any background. Also wearing of color bands is not
required in our system. By that, securities could authorize an individual’s identity
depending on who she is, and not what she has and what she could remember. Two main
classes can be found in biometrics:

A· Physiological It is associated with the body shape, includes all physical traits, iris, palm
print, facial features, Fingerprints, etc.

Behavioral Related to the behavioral characteristics of a person. A characteristic widely

used till today is signatures. Modern methods of behavioral studies are emerging such as
keystroke dynamics and voice analysis.

Deaf Mute Communication Interpreter- A Review [1]: This paper aims to cover the various
prevailing methods of deaf-mute communication interpreter system. The two broad
classification of the communication methodologies used by the deaf-mute people are –
Wearable Communication Device and Online Learning System. Under Wearable
communication method, there are Glove based system, Keypad method and Handicap
Touch-screen. All the above mentioned three sub- divided methods make use of various
sensors, accelerometer, a suitable micro-controller, a text to speech conversion module, a

5 | Page
 Sign language Converter

keypad and a touch-screen. The need for an external device to interpret the message
between a deaf mute and non-deaf-mute people can be overcome by the second method
i.e., online learning system. The Online Learning System has different methods. The five
subdivided methods are- SLIM module, TESSA, Wi-See Technology, SWI_PELE System
and Web- Sign Technology.

An Efficient Framework for Indian Sign Language Recognition Using Wavelet Transform
[2]: The proposed ISLR system is considered as a pattern recognition technique that has
two important modules: feature extraction and classification. The joint use of Discrete
Wavelet Transform (DWT) based feature extraction and nearest neighbor classifier is used
to recognize the sign language. The experimental results show that the proposed hand
gesture recognition system achieves maximum 99.23% classification accuracy while using
cosine distance classifier.

Hand Gesture Recognition Using PCA in [3]: In this paper authors presented a scheme
using a database driven hand gesture recognition based upon skin color model approach
and thresholding approach along with an effective template matching with can be
effectively used for human robotics applications and similar other application. Initially,
hand region is segmented by applying skin color model in YCbCr color space. In the next
stage thresholding is applied to separate foreground and background. Finally, template
based matching technique is developed using Principal Component Analysis (PCA) for
recognition.

Hand Gesture Recognition System for the Dumb People [4]: Authors presented the static
hand gesture recognition system using digital image processing. For hand gesture feature
vector SIFT algorithm is used. The SIFT features have been computed at the edges which
are invariant to scaling, rotation, addition of noise.

An Automated System for Indian Sign Language Recognition in [ 5]: In this paper a method
for automatic recognition of signs on the basis of shape-based features is presented. For
segmentation of hand region from the images, Otsus thresholding algorithm is used, that
chooses an optimal threshold to minimize the within-class variance of threshold black and

6 | Page
 Sign language Converter

white pixels. Features of segmented hand region are calculated using Hus invariant
moments that are fed to Artificial Neural Network for classification. Performance of the
system is evaluated on the basis of Accuracy, Sensitivity and Specificity.

Hand Gesture Recognition for Sign Language Recognition: A Review in [6]: Authors
presented various method of hand gesture and sign language recognition proposed in the
past by various researchers. For deaf and dumb people, Sign language is the only way of
communication. With the help of sign language, these physical impaired people express
their emotions and thoughts to other.

Design Issue and Proposed Implementation of Communication Aid for Deaf & Dumb
People in [7]: In this paper author proposed a system to aid communication of deaf and
dumb people communication using Indian sign language (ISL) with normal people where
hand gestures will be converted into appropriate text message. Main objective is to design
an algorithm to convert dynamic gesture to text at real time finally after testing is done the
system will be implemented on android platform and will be available as an application for
smart phone and tablet pc.

Real Time Detection and Recognition of Indian and American Sign Language Using Sift
In [ 8]: Author proposed a real time vision-based system for hand gesture recognition for
human computer interaction in many applications. The system can recognize 35 different
hand gestures given by Indian and American Sign Language or ISL and ASL at faster rate
with virtuous accuracy. RGB-to-GRAY segmentation technique was used to minimize the
chances of false detection. Authors proposed a method of improvised Scale Invariant
Feature Transform (SIFT) and same was used to extract features. The system is model
using MATLAB. To design and efficient user-friendly hand gesture recognition system, a
GUI model has been implemented.

A Review on Feature Extraction for Indian and American Sign Language in [9]: Paper
presented the recent research and development of sign language based on manual
communication and body language. Sign language recognition system typically elaborate
three steps preprocessing, feature extraction and classification. Classification methods used

7 | Page
 Sign language Converter

for recognition are Neural Network (NN), Support Vector Machine (SVM), Hidden
Markov Models (HMM), Scale Invariant Feature Transform (SIFT), etc.

Sign Pro-an Application Suite for Deaf and Dumb. in [10]: Author presented application
that helps the deaf and dumb person to communicate with the rest of the world using sign
language. The key feature in this system is the real time gesture to text conversion. The
processing steps include: gesture extraction, gesture matching and conversion to speech.
Gesture extraction involves use of various image processing techniques such as ISSN No.:
2454- 2024 (online) International Journal of Technical Research & Science pg. 433
www.ijtrs.com www.ijtrs.org Paper Id: IJTRS-V2-I7-005 Volume 2 Issue VII, August
2017 @2017, IJTRS All Right Reserved histogram matching, bounding box computation,
skin color segmentation and region growing. Techniques applicable for Gesture matching
include feature point matching and correlation-based matching. The other features in the
application include voicing out of text and text to gesture conversion.

Entries and input values at a given position. As we continue this process well create a 2-
Dimensional activation matrix that gives the response of that matrix at every spatial
position. That is, the network will learn filters that activate when they see some type of
visual feature such as an edge of some orientation or a blotch of some color.

1. Sunitha K. A, Anitha Saraswathi.P, Aarthi.M, Jayapriya. K, Lingam Sunny, Deaf Mute

Communication Interpreter- A Review, International Journal of Applied Engineering
Research, Volume 11, pp 290-296, 2016.

2. Mathavan Suresh Anand, Nagarajan Mohan Kumar, Angappan Kumaresan, An

Efficient Framework for Indian Sign Language Recognition Using Wavelet Transform
Circuits and Systems, Volume 7, pp 1874- 1883, 2016.
3. Mandeep Kaur Ahuja, Amardeep Singh, Hand Gesture Recognition Using PCA,
International Journal of Computer Science Engineering and Technology (IJCSET),
Volume 5, Issue 7, pp. 267-27, July 2015.

4. Sagar Pomorze, Prof. Abdul Sattar, Hand gesture recognition system for dumb people.

8 | Page
 Sign language Converter

5. International Journal of Science and Research (IJSR)

6. Chandandeep Kaur, Nivit Gill, An Automated System for Indian Sign Language
Recognition, International Journal of Advanced Research in Computer Science and
Software Engineering.

7. Pratibha Pandey, Vinay Jain, Hand Gesture Recognition for Sign Language
Recognition: A Review, International Journal of Science, Engineering and Technology
Research (IJSETR), Volume 4, Issue 3, March 2015.

8. Nakul Nagpal, Dr. Arun Mitra., Dr. Pankaj Agrawal, Design Issue and Proposed
Implementation of Communication Aid for Deaf

9. Dumb People, International Journal on Recent and Innovation Trends in Computing

and Communication, Volume: 3 Issue: 5, pp- 147 149.

10. S. Shirbhate1, Mr. Vedant D. Shinde2, Ms. Sanam A. Metkari3, Ms. Pooja U. Borkar4,
Ms. Mayuri A. Khandge/Sign-Language- Recognition-System.2020 IRJET Vol3
March,2020.

3. OBJECTIVE

 This project aims at identifying alphabets and sentences in sign language from
corresponding gestures and vice-versa.
 Gesture recognition and sign language recognition has been a well research topic for ASL
but has been rarely touched for its Indian counterpart.
 We aim to tackle this problem but instead of using high end technology like gloves or
Kinect for gestures recognition we aim at recognition from image.
 It can be obtained from say webcam; and then use computer vision techniques and ML
techniques for extracting relevant features and subsequent classification.

9 | Page
 Sign language Converter

4. METHEDOLOGY

A. Forms of Input
Our project is intended to get inputs in multiple formats. The inputs can be of forms:
 Text input
 Live speech input

Fig. 1: Front end of the System

B. Speech Recognition
The live speech is received as input from the microphone of our system. This is
done using the Python package PyAudio. PyAudio is a Python package that is used to
record audio on a variety of platforms. The audio thus received is converted into text
using Google Speech Recognizer API. It is an API that helps to convert audio to text
by incorporating neural network models. In the input format of giving the audio file,

10 | P a g e
 Sign language Converter

the received audio is translated into text by using this Google Speech Recognizer. For
lengthier audio files, the audio is divided into smaller chunks on the basis of the
occurrence of silence. The chunks are then passed into the Google Speech Recognizer
to efficiently convert into text.

Fig. 2: Block diagram of Speech to Text Conversion

Fig. 3: Speech Input

11 | P a g e
 Sign language Converter

C. Porter Stemming Algorithm

Porter Stemming algorithm provides a basic approach to conflation that may work
well in practice. Natural Language Processing (NLP) helps the computer to understand
the human natural language. Porter Stemming is one of the Natural Language
Processing techniques. It is the famous stemming algorithm proposed in 1980. Porter
Stemmer algorithm is known for its speed and ease. It is mainly used for data mining
and to retrieve information. It produces better results than any other stemming
algorithms. It has less error rate.

The system removes the morphological and in flexional endings of the English
words. The system uses Porter stemming Algorithm to remove the commonly used
suffixes and prefixes of the words and find the root word or original word. For example,
the Porter stemming algorithm reduces the words “agrees”, “agreeable”, “agreement”
to the root word “agree”. Because of this stemming, we can reduce the time taken for
searching the sign language for the given word.

D. Text to Sign Language

The system iterates through every word in the processed text sentence which is
received from the previous step and searches the corresponding sign language video
sequences in the local system. If the word is found, the system shows the output as a
video sequence using the OpenCV module in Python.

OpenCV is an open-source library mainly used for image processing, video

analysis, and many more functionalities related to computer vision. The System passes
the path of the video sequence to the OpenCV module to play the video. It shows the
video sequence frame by frame.

If the word is not found in the local system, the system will search for the word in
a sign language repository named “Indian Sign Language Portal”. The system looks for
the video link in the Indian Sign Language Portal by webscraping. And plays the
corresponding sign language video sequence.

12 | P a g e
 Sign language Converter

Webscraping is the process of extracting the content from the website.

WebScraping is achieved using the BeautifulSoup module. BeautifulSoup module in
Python helps to get or search or navigate or modify the data from the Html files by
using parsers.

Fig. 4: Block diagram for showing the sign language

When we speak the sentence “Hello, what is your name” as input through the microphone,
the text is processed and converted to “Hello, what your name” to give faster conversion by
removing the filling words. The following output pops up each video in the sequence as –

Fig. 4(a), Fig. 4(b), Fig. 4(c), Fig. 4(d) shows the output sign languages for the given sentence “Hello,
what is your name”.

13 | P a g e
 Sign language Converter

Fig. 4(a): Output sign (Hello)

Fig. 4(b): Output sign (What)

14 | P a g e
 Sign language Converter

Fig. 4(c): Output sign (your)

Fig. 4(d): Output sign (name)

15 | P a g e
 Sign language Converter

5. SYSTEM REQUIREMENTS

Software Requirements

A. Python
Python is a high-level, object-oriented, interactive, and interpreted programming
language. It is easy to write, learn and read. It is a free and open-source language. It has a
large set of libraries. Python is used in almost all the domains in the computer industry.

B. OpenCV
OpenCV (Open-Source Computer Vision Library) is an open-source library present
in Python. It is used has many machine learning algorithms which are used for image
processing, video analysis, and real-time vision applications.

C. PyAudio
PyAudio is a Python module to interface with audio drivers present in the device.
It helps to play and record audio on various platforms.

D. Tkinter
Tkinter is a standard Graphical User Interface (GUI) package for Python. It is
available in most operating systems. It provides an easy and faster way to construct many
widgets with basic functionality. It creates widgets like button, frame, checkbox, text field,
etc.,

E. BeautifulSoup
BeautifulSoup is a Python library that provides an easy way to extract information
from web pages. It isolates each tag, division, and section to make the searching faster and
easier.

16 | P a g e
 Sign language Converter

6. RESULT

Most of the time is taken for playing the video sequence in the Sign Language translator.
We have noted the time taken for the translator to translate from speech to sign language
from the command line. Speech recognizer takes time depending on the length of the
sentence we speak. The text analysis takes a minimal amount of time in converting the text
sentence.
• Time taken to convert the speech containing the word “nice” to sign language
is 10.37 seconds (from the start of audio recording to translation).
• It takes 28.94 seconds to convert the speech “Hello, what is your name?” to
sign language (from the start of audio recording to translation)

7. CONCLUSION AND FUTURE WORKS

Speech to sign language translation is a necessity in the modern era of online

communication for hearing impaired people. It will bridge the communication gap between
normal and hearing-impaired people.

The future work is to develop a chat application incorporated with this sign language
translation system. This can be used in team meeting applications, where a live translator
feature can be added to the application. Also, a sign language to text translating option can
be added to this application.

17 | P a g e
 Sign language Converter

8. REFRENCES

1. Ankita Harkude, Sarika Namade, Shefali Patil, Anita Morey “Audio to Sign Language
Translation for Deaf People” ISSN: 2277-3754, International Journal of Engineering
and Innovative Technology (IJEIT) Volume 9, Issue 10, April 2020.

2. Farahanaaz Shaikh, Shreya Darunde, Nikita Wahie, Swapnil Mali “Sign Language
Translation System for Railway Station Announcements”, Institute of Electrical and
Electronics Engineers (IEEE), IEEE Bombay Section Signature Conference (IBSSC),
2019

3. Vajjarapu Lavanya, Akulapravin, And M.S., Madhan Mohan “Hand Gesture

Recognition and Voice Conversion System Using Sign Language Transcription
System" ISSN:2230-7109 (Online) | ISSN:2230-9543 (Print) IJECT Vol. 5, Oct-Dec
2014.

4. Amit Kumar Shinde and Ramesh Khagalkar “sign language to text and vice versa
recognition using computer vision in Marathi” International journal of computer
Application (0975-8887) National conference on advanced on computing (NCAC
2015).

5. Dr. Sami M. Halawani. Arabic sign language translation system on mobile devices.
2008.

6. V. Lpez-Ludea, C. Gonzlez-Morcillo, J.C. Lpez, E. Ferreiro, J. Ferreiros, and R. San

Segundo. Methodology for developing an advanced communications system for the
deaf in a new domain. Knowledge-Based Systems, 56:240 – 252, 2014.

18 | P a g e