Abstract
Songs are a powerful means of expressing emotions through melody and lyrics. This study focuses on understanding and classifying the emotions present in songs, including positive, negative, and neutral emotions. Using large-scale language models (LLMs) such as BERT, RoBERTa, and DistilBERT, two datasets related to song emotions were analyzed. Despite their effectiveness in capturing emotions in lyrics, these models faced notable challenges. The variability in sentence length in musical data hindered generalization, and the imbalance in the distribution of emotions in the datasets affected the models’ ability to address the issue. To overcome these limitations, the creation of a third dataset specifically designed for song sentiment analysis was proposed. This new dataset addressed sentence length challenges by providing examples of song lyrics of varying lengths, enabling more effective model training. Furthermore, data imbalance was addressed through careful sample selection, representing a wide range of emotions in songs. The third dataset underwent classification using large-scale language models, achieving promising results. The accuracy metric reached an impressive 96.34%, highlighting the effectiveness of this approach in song sentiment analysis. This study underscores the importance of understanding emotions in songs and offers practical solutions to enhance the capabilities of language models in this task.
Introduction
Emotions represent a capacity inherent to the human being, resulting from the interaction of biological, social, cultural, and mainly neural elements. According to neurobiology, the brain initiates reactions based on the interpretation of stimuli, cultural education, and cognitive functions. As individuals mature, their capacity to experience and transmit emotions broadens, extending not only to individuals but also finding expression in poetry, literature, and contemporary music.
A musical piece has the ability to activate emotional areas and evoke feelings through rhythm. However, rhythm alone is not the exclusive path to emotions. The selection of words and phrases by composers also holds substantial importance in expressing emotions, such as sadness, happiness, and other feelings. The classification of emotions can be facilitated by the application of Natural Language Processing (NLP) methods (Poria et al., 2017).
Identifying emotions in sentences is a challenging task due to their subjective nature, often reflecting personal views and feelings. This complexity is compounded by linguistic variations in the use of Spanish in regions such as Mexico, Chile, and Spain. Furthermore, the length and diversity of the text samples being analyzed are critical for accurate categorization. In this project specifically focused on emotions of song lyrics, two different Spanish datasets were analyzed. One dataset includes a mix of Spanish variations, while the other is focused on Mexican Spanish. Through a systematic approach, emotional expressions were thoroughly examined, and a comparison was made between the two datasets. This comparative analysis highlighted how emotions are conveyed in different variations of language. By merging insights from both datasets, an effective framework for classifying emotions was established, considering both linguistic differences and universal emotional aspects.
The rest of this article is structured as follows: In Section 2, the motivation behind this work and why it is relevant are addressed; Section 3 analyzes previous work related to songs; Section 4 focuses on datasets and how the new dataset was built; Section 5 details how Large Language Models were applied; Section 6 details the hyperparameter tuning and finally, in Section 7, the results obtained in this work are presented and compared with previous research in the same field.
Motivation
Spanish is one of the most widely spoken languages in the world, it is officially spoken in around 21 countries, mainly: Spain, Mexico, Guatemala, El Salvador, Honduras, Nicaragua, Costa Rica, Panama, Colombia, Ecuador, Peru, Bolivia, Chile, Argentina, Uruguay, Paraguay, Venezuela, Puerto Rico, Dominican Republic, Cuba and Equatorial Guinea 1
In the world, Latin America represents the greatest wealth in terms of variants of the Spanish language, arising from contact between indigenous populations and existing languages. Derived from the above, when analyzing datasets that include these variants, it is very enriching for NLP. Added to this, and as mentioned, music is an excellent channel for feelings, since in the same song, we can find different sensations in the texts. On the other hand, emotion analysis is a topic that has been on the rise, especially with the increased use of GPT (Radford et al., 2019), especially with the prompt, but there hasn’t really been an emphasis on determining some language differences. Focusing on the Spanish language, on the lexicon between countries that speak this language. However, there are linguistic differences perceptible by hearing and writing, especially between the regions of Spain and Latin America. Different emotions can be extracted, but the most classic are positive, negative and neutral (Xue et al., 2020). The way to approach emotion analysis has several approaches, but during this work, we will focus on a classic approach to compare its performance with the two datasets. One important point to highlight is that the datasets were written in Spanish, they don’t have translations or bad interpretations, and for which it is interesting to test the large language models that were trained with large amounts of data and include the Spanish language and those multilanguage models that were made for this purpose.
Large Language Models
Large Language Models (LLMs) are natural language models grounded in transformational structures, as detailed in Vaswani et al. (2017). These models undergo extensive training with the aim of assimilating contextual cues and discerning syntactic patterns. When discussing language-specific models, BERT (Bidirectional Encoder Representations from Transformers) emerges as a significant contender. Developed by Google, BERT is a pre-trained model, as indicated in Devlin et al. (2018). It has been trained on vast quantities of textual data, focusing on deducing contextual relationships between words within both lengthy passages and individual paragraphs. What sets BERT apart from other language systems is its bidirectional approach, capturing context from both left-to-right and right-to-left directions. This duality enables BERT to cater to diverse languages and regions. An extension of this approach is the Multilingual BERT, which broadens its applicability through multilingual training techniques, as elucidated in Pires et al. (2019).
The integration of multilingual capabilities into these models offers a range of advantages, as outlined in Pires et al. (2019).
Among the most renowned multilingual models, BERT stands out significantly, boasting pre-training across 102 languages (Devlin et al., 2018). The multilingual BERT model adheres to the foundational principles of BERT. In essence, the model proposes two sentences to discern the language’s structural intricacies and then determines whether these sentences are sequentially related. This approach facilitates an internal representation of the English language. Additionally, the DistilBERT model comes into play, presenting a “distilled” version of the multilingual BERT model. This version is capable of distinguishing between English and other languages. With 6 layers, 768 dimensions, and 12 heads, the model comprises a total of 134 million parameters. Notably, DistilBERT exhibits twice the speed of BERT-base, making it a practical option.
One of the early research works on emotion investigation is conducted in He et al. (2008), where a database of 1000 songs in English was employed. A categorical approach was utilized to categorize emotions into happy, sad, angry, and relaxed, along with complementary categories: not happy, not sad, not angry, and not relaxed. Their primary focus was on a classic approach using Support Vector Machine (SVM) for emotion classification.
The authors of Edmonds and Sedoc (2021) explore various approaches using the Edmonds Dance dataset, which is in English. In this study, they deploy BERT to train the system at the song level and achieve noteworthy results for the emotion ”anger,” with an accuracy metric of 0.88.
In the domain of Emotion Classification, considerable efforts have been expended. As highlighted in Srinilta et al. (2017), a polar classification strategy was employed for a collection of Thai songs, focusing solely on the lyrics. The study proposed utilizing a lexicon alongside traditional machine learning methods. Regarding the song’s components (title, verse, chorus, pre-chorus, and bridge), emphasis was placed on solely using the chorus and verses as the corpus. This selection was influenced by the likelihood that these two sections are most indicative of the song’s theme.
The research cited in Kumar and Minz (2013) used an ontology called SentiWordNet, which contains indicators of positive or negative connotations of words. This ontology was used to extract emotional features from song lyrics in order to identify the moods associated with these compositions. The experiments were performed using a set of 185 songs and three different classification algorithms were used: Naive Bayes, K-Nearest Neighbor and Support Vector Machines (Support Vector Machines, SVM). This research approach sought to understand how the SentiWordNet ontology can contribute to the emotional analysis of song lyrics. The results of these experiments shed light on how different classification algorithms approach the task of classifying songs based on their emotional state. Ultimately, these efforts have shown how ontological tools and machine learning algorithms can be combined to gain a deeper understanding of the emotional connotations in artistic expressions like songs. In the cited article, a comprehensive comparison of the performance of different ”word embedding” models, which were previously trained on song lyric analysis and polarity assessment in movie reviews, is made. The results obtained illustrate that the models trained with data from tweets show a better performance in the analysis of song lyrics. On the other hand, models that were based on datasets from Google News and Common Crawl seemed to be better for movie analysis. This is due to the notorious similarity of the vocabulary used in both contexts.
On the other hand, it was discovered that there are combinations of models that are not often used in the field of classification, as mentioned in reference Rhanoui et al. (2019). An example of this is the use of convolutional neural networks (CNN), a model that combines convolutional networks and bi-directional LSTMs. The purpose of this integration is to capture relevant text features at different levels. These experiments, performed on diverse datasets, reveal an impressive 90.66% accuracy rate.
This new approach, which combines features of bidirectional LSTMs and convolutional networks, shows how innovation in model mixing can achieve excellent results in a classification task. The high accuracy achieved in these experiments underscores the importance of exploring unconventional combinations to improve text parsing ability. Finally, this study highlights how creativity in model structure can contribute significantly to optimizing text classification in artistic contexts and beyond.
Datasets
The initial group of information collected, known as
Within this dataset, a total of 91 songs are gathered, all of them written in Spanish without any type of discrimination, covering both the versions from Latin America and the regions of Spain. This diverse range covers a wide range of musical styles, including rhythms such as bachata, pop, and ballad, among others. Each of these songs has been meticulously broken down into short paragraphs through a manual process. This division was made following the emotional flow of the sentences, paying special attention to maintain the coherence of the sentiment expressed in each paragraph. In this way, it is guaranteed that there are no truncated thoughts or sentences that end in empty words. Taken together, this painstaking segmentation work has resulted in a data collection made up of 1,477 individual items. In the song tagging process, an approach uniquely designed by those responsible for this dataset was applied. To carry out the labeling, three fundamental emotions were taken into account: ”S” to denote emotional neutrality, ”P” to indicate emotional positivity, and ”N” to represent emotional negativity.
The inherent distribution of this dataset is reflected in Table 1. It is important to highlight that this set exhibits an imbalance in terms of the number of samples present in each emotional category. This imbalance can be easily discernible when looking at the numbers. Managing this imbalance is crucial, since it can influence the development of future research and applications based on these data. In this sense, adequately addressing this disparity is essential to guarantee unbiased and robust results, and to achieve a more complete representation of emotions in the analyzed songs.
Distribution of the Corpus Textos de Canciones en Español, Set 1 (Garcia-Vazquez et al., 2023).
Distribution of the Corpus Textos de Canciones en Español, Set 1 (Garcia-Vazquez et al., 2023).
.
In Figure 1 it can be seen the distribution of the dataset 1.
Distribution of dataset 1.
The second collection of data is called
This particular set comprises of 200 songs, all of them written in Spanish, with a focus on the characteristic linguistic variants of Mexico. The selection covers musical genres such as banda, pop, regional music, Mexican, rock and cumbia, thus reflecting the rich musical diversity of this region.
A relevant aspect of this set is its structure. Each song has been carefully divided into concise paragraph snippets through a manual process. This segmentation has been carried out in a coherent way with the flow of the sentences, ensuring that there are no incomplete ideas or sentences that end abruptly. In all, this painstaking labor of dividing has resulted in a collection of 4,555 individual fragments.
For the labeling, three main emotions were considered: S, neutral; P, positive; N, negative. The table 2 represents the distribution of the dataset.
Distribution of the Texts of Songs in Mexican Spanish corpus, Dataset 2.
.
In Figure 2 it can be seen the distribution of the dataset 2.
Distribution of dataset 2.
In a previous work (Garcia-Vazquez et al., 2023) presented experiments based on dataset 1 and dataset 2. Table 3 shows the results.
Results of Datasets 1 and 2 for the Accuracy Metric Performed in Previous Experimentation.
.
However, even though competitive results were achieved in previous experiments, superior results were obtained joining these datasets. Superior results were observed in dataset 1 during the study of these datasets, leading to the following hypotheses:
To address the above limitations, the two datasets were merged, to increase the advantages of each one. The data was alternated one with another, to generate an organic dataset and when dividing them, they were not biased towards a single set. Since both datasets were adjusted to have exactly the same columns, labels, and increasing the phrase length by 30%, the fragments belonging to dataset 2 were largely homologous to this new dataset. Since dataset 1 is in CSV (Comma Separated Values) format, a change in the encoding mode was required so that the letters with tildes would appear correctly, respecting the UTF-8 encoding. This new dataset was used to label a third, wich was created using web scraping as shown below.
It is worth noting that since this work is specifically addressed to the implementation of LLMs, no additional preprocessing step is required. The main goal here is to allow the model to effectively infer and understand the context presented in the song snippets.
For classification, we use the same models as in the first experiments to verify and compare the results. The dataset was divided into 80% for training and 20% for validation. The models used for this task were the following:
BERT-base-multilingual-cased RoBERTa-base DistilBERT-base-multilingual -cased
Since 2 different datasets were joined, a maximum length of 300 characters was considered, since that was the maximum length used for dataset 1.
RoBERTa
Since this variant of the model is an adaptation of BERT in which the sentence pretraining step was omitted, training with small batches and variable learning rates was carried out instead. This has proven extremely useful in the task of classifying datasets, since these are also divided into small pieces of information. In terms of hyperparameters, a learning rate that adjusts as epochs progress was established, with the purpose of addressing the challenge of local minima. The batch size was set at 24 items, a choice supported by the literature and the processing capabilities of the equipment used in the tests. This value is optimal, since selecting a smaller size would significantly increase the risk of not reaching adequate generalization. A similar strategy was applied to the validation batch size, keeping the same value, since this configuration is positively adapted to this type of model. Specific details about the hyperparameters fitted for this model, in relation to both datasets, are presented in Table 4.
Modified Hyperparameters Proposed for the RoBERTa LLM.
Modified Hyperparameters Proposed for the RoBERTa LLM.
.
The basic variants of the multilingual BERT model was selected as the second option. This choice considers the capital letters due to the outstanding performance of this model in various applications. Some of the hyperparameters that were previously described in the previous model have been kept, since they have proven their efficiency in its operation. A learning rate initialized at a standard value was adopted, with the particularity of being updated at each training epoch, adapting according to a pattern of updating. For the batch size, it was set at 36 items, and the same batch size will be maintained for the validation phase. This configuration was deliberate, considering the nature of the data and the available computational capacity. Table 5 contains detailed information about the hyperparameter configurations. Performed on this model in relation to both datasets.
Modified Hyperparameters Proposed for the BERT LLM.
Modified Hyperparameters Proposed for the BERT LLM.
.
It is important to note that the choice of the base version of multilingual BERT is based on its historically outstanding performance. The conservation of certain hyperparameters from the previous model builds on previous positive experience, allowing a solid foundation for the current configuration. The adaptive learning rate strategy aims to optimize the weight adjustment process at each stage of training. The decision to maintain a consistent lot size is supported by the consistency and uniformity this provides during the training and testing phase.
To close the selection of models, the basic variant of the multilingual DistilBERT model was selected, with special attention to capitalization. This choice is justified in the search for a more agile and efficient version of the BERT model, enabling more precise predictions with a controlled use of computational resources.
DistilBERT, in its base version, is designed to leverage the strengths of BERT, while offering a lighter and faster alternative, ideal for situations where resources are limited.
Table 6 provides a detailed analysis of the hyperparameters that have been specifically fitted for this model in the context of the two datasets.
Modified Hyperparameters Proposed for the DistilBERT LLM.
Modified Hyperparameters Proposed for the DistilBERT LLM.
.
The choice of the base version of the DistilBERT multilingual model was based on its ability to provide accurate results while reducing computational load. The capitalization of capital letters is a key aspect in this choice, as it contributes to the effectiveness of the model in different language contexts. DistilBERT represents a more agile solution compared to BERT, being an efficient answer to obtain accurate predictions when the available resources are limited.
For each of the LLMs used, experimentation began with the base of each model, followed by a grid search over the hyperparameter set recommended by the authors of BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), and DistilBERT (Sanh et al., 2019). A total of 18 full training runs were conducted for each hyperparameter combination. As an additional metric for selection, we experimented with the best-performing ones already obtained during the study (Garcia-Vazquez et al., 2023).
Results
The results derived from the datasets and their evaluation through the BERT, RoBERTa and DistilBERT models are documented in the corresponding table. In the table 7, the values related to the accuracy and F1 metrics in validation are presented.
BERT, RoBERTa and DistilBERT Results with Text of Spanish Songs for the Accuracy Metric and F1 Metric.
BERT, RoBERTa and DistilBERT Results with Text of Spanish Songs for the Accuracy Metric and F1 Metric.
.
It is important to note that the comparison is not 100% direct, as the state of the art does not use the same datasets. However, as mentioned, the work cited in reference (Garcia-Vazquez et al., 2023) did use the same datasets as us. Therefore, in Table 8, a comparison of the results is shown regarding datasets 1 and 2 in relation to our dataset number 3.
Comparative Table of the Previous Experiments and Results Obtained.
.
This study has focused on the analysis and automation of song text categorization in various variants of the Spanish language. To achieve this goal, models such as BERT Multilingual, RoBERTa, and DistilBERT Multilingual have been employed, sharing notable similarities in their approach and functionality.
The results reflect that the merging of datasets leads to an improvement in overall performance. Notably, the inclusion of the F1 metric, which was not considered in previous experiments. This inclusion has resulted in favorable outcomes in the evaluation of the models.
It is evident that models adapted with specific Spanish language datasets, as in the case of RoBERTa, achieve significant and highly competitive results compared to multilingual models such as BERT and DistilBERT.
The obtained results are not solely a product of dataset consolidation; through this research, we verified that sentence length played a crucial role and helped counteract the individual disadvantages of each dataset, as more features could be extracted from the presented data. Another point was the search for hyperparameters, which, although very similar among themselves, yielded good results and should be considered for future implementations of this type of data.
All of the above contributes to an overall improvement in the performance of the models. However, there is potential to enrich and expand sentences within dataset 1. This additional perspective could add greater information richness to language models, thereby contributing to their capacity for improvement and optimization. In summary, this study has shed light on the categorization of song texts in Spanish through the implementation of various models, opening the door to future research on the improvement and refinement of this technique.
Footnotes
Acknowledgements
The authors wish to thank the support of the Instituto Politécnico Nacional (COFAA, SIP-IPN, Grant SIP 20240610) and the Mexican Government (SECIHTI, SNI).
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.