PROJECT

BRANCH : COMPUTER SCIENCE AND ENGINEERING

TITLE :Hybrid Deep Learning Approach to Emotion-Infused Music

Recommendations
SUPERVISOR : Dr. C.Madhusudhan Rao

S.NO NAME ROLLNO

1. GANDE SAIKIRAN 20951A05E6

 Outline

⦿ Introduction
⦿ Problem Definition
⦿ Objective
⦿ Outcome
⦿ Existing Methodology
⦿ Proposed Methodology
⦿ Physical Architecture
⦿ Dataset Details
⦿ Performance Metric
⦿ Algorithm
⦿ Literature Survey
⦿ References

2
 Introduction

o In the digital age, the sheer volume of available music and videos poses a significant
challenge for users seeking personalized content recommendations. As the demand for
tailored experiences grows, there is a pressing need for intelligent systems that can
accurately discern and recommend content based on user emotions and preferences.
o The ability to recognize and respond to human emotions is a critical aspect of creating a
more engaging and personalized music recommendation system. Understanding the
intricate relationship between emotions and music can significantly enhance the overall
user experience, leading to increased user satisfaction and engagement..
o While numerous music recommendation systems exist, many of them fail to fully
capture the emotional nuances that significantly impact user preferences. Existing
systems often rely on simplistic metadata or collaborative filtering techniques, which
overlook the complex emotional dimensions that contribute to an individual's music
choices. 3
 Problem Definition

Traditional music recommendation systems often lack the ability to capture the complex
emotional nuances that significantly influence user preferences. Existing methods based on
either content-based or collaborative filtering techniques fail to effectively incorporate the
emotional aspects of music, limiting the overall personalized and engaging user
experience. Recognizing the limitations of conventional methods, there is a growing
demand for innovative approaches that combine the strengths of deep learning with
emotion recognition to create more effective and accurate music recommendation systems.
Integrating deep learning models with emotion detection techniques can facilitate the
extraction of intricate emotional patterns from music data, enabling the development of a
more comprehensive and personalized recommendation framework.

4
 Objective

Specific objectives include:

o Construct an advanced emotion recognition model leveraging deep learning techniques,
capable of accurately extracting and interpreting intricate emotional patterns from
diverse music data sources, including audio and video.
o Build a hybrid recommendation system architecture that combines the power of deep
learning with emotion recognition to generate highly accurate and personalized music
recommendations.
o Measure the system's accuracy in predicting user emotional preferences and gather user
feedback to evaluate the overall satisfaction and engagement levels with the
recommended music content.
o Performance of the developed hybrid deep learning approach with existing state-of-the-
art techniques, highlighting the advantages and improvements achieved in terms of
accuracy, efficiency, and the ability to capture and respond to user emotional preferences

in the domain of music recommendations. 5

 Expected Outcome
o Improved Recommendation Accuracy:The hybrid deep learning approach is
expected to significantly enhance the accuracy of music audio and video
recommendations by effectively capturing and analyzing the emotional content
embedded within the multimedia data
o Enhanced Personalization and User Engagement:By integrating advanced deep
learning techniques with emotion recognition models, the system can provide highly
personalized music recommendations tailored to the user's emotional state and
preferences
o Efficient Extraction of Emotional Patterns: The hybrid approach is expected to
demonstrate its capability to efficiently extract and interpret intricate emotional
patterns from various data sources, including audio and video content.

o Seamless Integration of Multi-modal Data: The hybrid approach is anticipated to

showcase its ability to seamlessly integrate and analyze multi-modal data sources, such
as audio features, video content, lyrics, and user interaction patterns 6
 Existing Methodology
o Feature Extraction and Analysis: This involves extracting relevant features from music
data, such as acoustic features, lyrics, and metadata. These features are then analyzed to
identify emotional patterns and characteristics embedded within the music.
o Sentiment Analysis and Affective Computing: Utilizing sentiment analysis techniques,
the system evaluates the emotional content of the music and maps it to specific
emotional categories or tags. Affective computing techniques are also employed to
recognize emotional states from various multimedia data sources, contributing to a more
comprehensive understanding of user preferences.
o Machine Learning and Deep Learning Models: Various machine learning and deep
learning models are employed to analyze and predict user preferences based on
historical data and user interactions.

7
 Proposed Methodology

o Multi-modal Data Integration: The system begins by integrating multi-modal data

sources, including audio features, video content. This step involves preprocessing and
feature extraction to create a comprehensive representation of the emotional content
within the music and video data.
o Real-time User Interaction Analysis: The system continuously analyzes user
interactions, feedback, and emotional responses to refine the recommendation process in
real-time. By monitoring user engagement and satisfaction with the recommended
content, the system adapts and adjusts its recommendations to cater to evolving user
preferences and emotional states.
o Hybrid Recommendation System Architecture: The system's architecture combines
the power of deep learning with emotion recognition to generate personalized and
emotionally relevant music video and audio recommendations.

8
Physical Architecture

9
Mathematical Model

10
Conceptual Model

11
Dataset Details

12
 Dataset Attributes

1. `lastfm_url`: The URL of the track on Last.fm.

2. `track`: The name of the track.
3. `artist`: The name of the artist who performed the track.
4. `seeds`: A list of related tracks that could be used as seeds for recommendation
algorithms.
5. `number_of_emotion_tags`: The count of emotion tags associated with the track.
6. `valence_tags`: Tags indicating the emotional valence of the track (e.g., happy, sad,
etc.).
7. `arousal_tags`: Tags indicating the arousal level associated with the track (e.g.,
calm, exciting, etc.).
8. `dominance_tags`: Tags indicating the dominance level associated with the track
(e.g., submissive, powerful, etc.).
9. `mbid`: MusicBrainz ID for the track.
10. `spotify_id`: Spotify ID for the track.
11. `genre`: The genre(s) associated with the track.

13
 Performance Metric

⦿ Mean dwell time to the eye is a metric commonly used in eye-tracking studies to
assess the average duration that an individual's gaze remains fixed on a specific point
of interest, such as an object, image, or text, within a visual stimulus. This metric is
important in understanding visual attention and user engagement with various visual
stimuli, including digital interfaces, advertisements, web pages, and other visual
content.

14
 Algorithm

⦿ MTCNN, which stands for Multi-task Cascaded Convolutional Networks, is a popular deep
learning-based algorithm used for face detection and facial attribute analysis. MTCNN is known
for its ability to perform three tasks simultaneously:

Face Detection: MTCNN is capable of detecting faces within an image, regardless of their
size or orientation. It uses a cascade of three neural networks to progressively narrow down the search
area for faces.

Facial Landmark Detection: Once a face is detected, MTCNN is also capable of identifying
key facial landmarks such as the eyes, nose, and mouth. This helps in tasks such as face alignment and
pose estimation.

Face Alignment: MTCNN can align the detected faces to a standardized position or
orientation, making it easier to perform further analysis or tasks on the face, such as recognition or
expression analysis.

15
 Algorithm

⦿ MTCNN has gained popularity due to its robust performance, even in cases of
occlusion, pose variations, and complex lighting conditions. It's widely used in various
applications such as face recognition, facial attribute analysis, and facial expression
recognition.

⦿ The algorithm typically involves a series of convolutional neural networks (CNNs)

working in a cascaded manner to progressively refine the face detection process. Each
stage of the cascade helps to filter out non-face regions and refine the bounding boxes,
ultimately leading to accurate face detection and landmark localization.

⦿ MTCNN has been implemented in various deep learning frameworks such as

TensorFlow and PyTorch, making it accessible and easy to use for researchers and
developers working on face-related tasks. Its ability to handle real-world challenges in
face detection has made it a key component in many computer vision applications.
16
Code implementation

17
Code implementation

18
Code implementation

19
20
Results

21
Results

22
Results

23
 Literature Survey

No Title Techniques/ Findings

Algorithms Used
l9
1. Han, Y., Yang, Feature Comprehensive review of music emotion
Y., & Vaquero, extraction, recognition
T. (2014) machine
learning
2. Li, M., Wang, Content Survey on music emotion recognition
S., & Chen, analysis, from a content perspective
L. (2018) feature
ediapipe
extraction

3. Yang, Y. H., Multi-level Overview of content-based music

& Chen, Y. H. feature emotion recognition.
(2018) extraction

24
 Literature Survey

4. Laurier, C., Feature Experiment on music mood and arousal

Herrera, P., extraction, using machine learning techniques
& Sordo, M. machine
(2018) learning

5. Kim, S., & Convolutional Music emotion recognition using

Kwon, Y. neural convolutional neural networks
(2018) networks

25
 References

• Han, Y., Yang, Y., & Vaquero, T. (2014). Music Emotion Recognition: A State of the
Art Review. In 2014 IEEE International Conference on Multimedia and Expo
Workshops (ICMEW) (pp. 1-6). IEEE.
• Li, M., Wang, S., & Chen, L. (2018). A survey on music emotion recognition: from a
viewpoint of content analysis. Multimedia Tools and Applications, 77(1), 1335-1359.
• Yang, Y. H., & Chen, Y. H. (2018). An Overview of Content-Based Music Emotion
Recognition. IEEE Transactions on Affective Computing, 9(2), 220-237.
• Kim, Y., Lee, J. H., & Lee, G. G. (2018). Music Emotion Recognition Using Deep
Neural Networks with Cross-Feature Transfer Learning. Applied Sciences, 8(3), 322.
• Laurier, C., Herrera, P., & Sordo, M. (2018). Music Mood and Arousal: An Experiment
with Machine Learning Techniques. In Proceedings of the 19th International Society
for Music Information Retrieval Conference (ISMIR) (pp. 526-533).
• Kim, S., & Kwon, Y. (2018). Music emotion recognition using convolutional neural
networks. Multimedia Tools and Applications, 77(8), 10109-10127.
• Yang, Y. H., & Chen, Y. H. (2018). Music emotion recognition using multi-level
features. IEEE Access, 6, 45413-45424.

26
27