Vivify—Emotion Recognition and Music Recommendation Using Transformers

Music and literature are powerful mediums capable of evoking a wide range of emotions. This research combines these two forms of art to provide an immersive reading experience by exploring the use of music to enhance the emotional impact of a book. The primary aim is to help the reader to develop a deeper emotional connection between the reader and the text. A novel framework, Vivify is proposed that uses RoBERTa for text emotion recognition and LSTM for music emotion recognition. Vivify system analyses the emotional tone of the text and recommends congruent background music using Cosine similarity. Unlike existing emotion-based recommender systems which rely on categorical emotions or language-dependent features, Vivify introduces a continuous valence–arousal alignment between textual and musical modalities, ensuring language-independent and fine-grained emotional congruence. A user satisfaction survey with 63 participants was conducted on three criteria namely, satisfaction, engagement and congruence of the recommended music to evaluate the holistic performance of the proposed system. The proposed Vivify has achieved a Mean Absolute Error (MAE) of 0.1275 for text and 0.3110 for music. The results show that the Vivify system enhances personalization and engagement of the reading activity which can be effectively used in education and entertainment. The proposed system is language-independent and focuses only on background music rather than on lyrics and thereby ensuring the suitability across linguistic contexts. This approach facilitates a personalized and immersive reading experience with reader engagement and emotional connection.