A novel focal-loss and class-weight-aware convolutional neural network for the classification of in-text citations

Abstract

We argue that citations, as they have different reasons and functions, should not all be treated in the same way. Using the large, annotated dataset of about 10K citation contexts annotated by human experts, extracted from the Association for Computational Linguistics repository, we present a deep learning–based citation context classification architecture. Unlike all existing state-of-the-art feature-based citation classification models, our proposed convolutional neural network (CNN) with fastText-based pre-trained embedding vectors uses only the citation context as its input to outperform them in both binary- (important and non-important) and multi-class (Use, Extends, CompareOrContrast, Motivation, Background, Other) citation classification tasks. Furthermore, we propose using focal-loss and class-weight functions in the CNN model to overcome the inherited class imbalance issues in citation classification datasets. We show that using the focal-loss function with CNN adds a factor of $(1 - p_{t})^{γ}$ to the cross-entropy function. Our model improves on the baseline results by achieving an encouraging 90.6 F1 score with 90.7% accuracy and a 72.3 F1 score with a 72.1% accuracy score, respectively, for binary- and multi-class citation classification tasks.

Keywords

Citation classification class imbalance convolutional neural network deep learning focal loss

1. Introduction

According to Boyack et al. [1] and Bornmann et al. [2], a publication may be cited to extend the ‘state of the art’, employ its algorithms and techniques, provide a background context of a research problem or merely make a comparison. The connection between the cited and the citing document can be regarded as a creative act that justifies the need for that information in the citing [3,4]. The relationship between a cited work and the citing document must be determined through the clues presented by the latter’s author, and, to adequately measure the work’s impact, data on the citation’s purpose, function and motive should be extracted from the citation’s context [5,6]. Citations doubtless provide a quantitative way to measure the quality of a published work, but they cannot reveal the actual relationship between the documents, nor are they all of the same importance [7].

Gross and Gross [8] pioneered the study of citation analysis, and, since then, much research has been conducted in this field. In these studies, the term ‘citation’ is mostly used for ‘the work cited’, but it is also used to refer to ‘the action of citing’. Brittain [9] defines a ‘citation’ as each instance where an author cites work in an article and a ‘reference’ as the work that is being cited. Citations have always been analysed by authors to study the flow of knowledge and measure the impact and influence of scholarly works [10 –12]. Frost [13] explains that what is being referred to can be expected to connect to the citing work in a logical way. For instance, an author may be researching a similar topic to that in the cited work, or the cited work may have provided the background for that research. Alternatively, the author may be acknowledging the pioneering work; that is, the cited work might be the motivation for the research.

In recent years, the study of the citation classification task has shifted from dealing with multiple categories, as initially proposed by Moravcsik and Murugesan [14], to just two categories: important and non-important. A citation that extends or uses the cited work in a meaningful way is defined as important, whereas one that is mentioned in the literature review or that compares the cited work with the citing article’s study is regarded as non-important or incidental.

Over the last few years, many authors have proposed citation classification techniques to segregate important citations from non-important citations. Zhu et al. [15] use a supervised machine learning approach for this task. Their study reveals that among all the classification features, the feature that focuses on the total number of times a paper is cited in the citing paper is the most important feature in recognising important references. According to Zhu et al., the feature selection is the most crucial and significant task in citation classification. The valuable features support to score higher performance results in terms of higher classification accuracy and F1 score.

Valenzuela et al. [16] deploy a range of 12 new features to address a similar classification task. The authors show that only 14.6% of citations appear to be important (influential) and that the features like ‘self-citation’ and ‘total count of citations per section’ are the best performing features to predict influential citation. Furthermore, to identify the most influential features in citation classification, Pride and Knoth [17] combine the 40 features proposed by Zhu et al. [15] with the 12 proposed by Valenzuela et al. [16] to explore that a group of only three features such as abstract similarity, author overlap and total number of direct citations leads to achieve higher accuracy results. The authors report, as did Valenzuela et al. [16] and Zhu et al. [15], that the most significant feature to predict citation’s behaviour as influential citation is the ‘number of times a reference is cited in the citing paper’.

In a more recent study, Hassan et al. [5] address the problem of citation classification by comparing classic machine learning models (support vector machine (SVM) and random forest (RF)) with the long short-term memory (LSTM) model. They employed four state-of-the-art models [16], [18 -20], and a new LSTM-based version to distinguish influential from non-important citations using the 64 features extracted from Valenzuela et al.’s [16] dataset. The model outperformed state-of-the-art techniques by 11.25%; however, since the study used a limited dataset of only 465 citations, the deep learning model could not be applied in the true spirit.

Studies that address the classification of citations into important and non-important usually exploit supervised machine learning techniques. By contrast, Hassan et al. [21] cluster citations in their dataset into these two groups by using a Self-Organising Map (SOM) to find the qualitative representations of the features and for the better data visualisation. Interestingly, the non-important citation class formulates an independent cluster with neighbouring neurons. Their result findings indicate that it is quite convenient to identify the non-important citation class compared with the important class, yet the classification suffers from the issue of class imbalance.

Several studies have recently demonstrated the power of machine learning approaches to extract in-text important citations from scholarly publications, yet they display major drawbacks, including the non-trivial feature engineering task and the absence of any dataset annotated by human experts with which to train their machine learning models.

Given the crucial need for citation classification and the fundamental issue of imbalance between the important and non-important classes, in this study, we propose the following:

First, to compile a comprehensive, labelled dataset of 9518 citation contexts extracted from the Association for Computational Linguistics (ACL) anthology of research articles, by merging the citation scheme presented by Jurgens et al. [22] and Teufel et al. [23], we present a six-class annotation scheme for citation classification: Use, Extend, CompareOrContrast, Motivation, Background, and Other. We merge these six into two major classes of important and non-important citations. Table 1 shows the classification scheme, with examples.

Second, and foremost, we propose a novel focal-loss and class-weight-aware convolutional neural network (CNN) architecture to overcome the inherited issue of class imbalance in citation classification. Finally, we show that using the focal-loss function adds a factor of $(1 - p_{t})^{γ}$ to the general cross-entropy function and that the proposed CNN architecture outperforms the existing state-of-the-art models in distinguishing important from non-important citations.

Table 1.

The citation classifications with examples.

Class label		Example
Binary	Multi
Important	Use	… For our experiments, we used the standard division of the WSJ (Marcus et al. 1993), with sections 2 through 21 for training (approx. 40,000 sentences) and section 23 for testing (2416 sentences 100 words); section 22 was used as development set …
	Extend	… Our work extends directions taken in systems such as Ariane (Vauquois and Boitet, 1985), FoG (Kittredge and Polguere, 1991), JOYCE (Rambow and Korelsky, 1992) and LFS (Iordanskaja et al., 1992) …
Non-important	Motivation	… This approach has now gained wide usage, as exemplified by the work of Collins (1996, 1999), Charniak (1996, 1997), Johnson (1998), Chiang (2000) and many others. The other innovation of DOP was to take (in principle) all corpus fragments, of any size, rather than a small subset …
	Background	… Part of speech taggers typically require input in the format of a single sentence per line (for example Brill’s tagger (Brill, 1992)) and parsers generally aim to produce a tree spanning each sentence …
	CompareOrContrast	… Similar to (Li et al., 2013a), we use a pipeline summarization framework where multiple compression candidates are generated for each pre-selected important sentence, and then an ILP691-based summarization model is used to select the final compressed sentences …
	Other	… Classical planning algorithms are inappropriate because the tutor must be able to change plans based on the 52 student’s responses. For this reason, we have adopted the ideas of the philosopher Michael Bratman (1987, 1990). Bratman uses the term ‘practical reason,’ to describe his analysis since he is concerned with how to reason about practical matters. For human beings, planning is required to accomplish one’s goals …

The rest of this article is organised as follows. Section 2 presents the literature in the domain of citation classification. Section 3 describes our compiled dataset and proposed approach to citation context classification. Section 4 presents the experimental details and the results achieved, with a comparison of baseline and state-of-the-art results. Finally, Section 5 offers concluding remarks.

2. Literature review

An enormous number of studies have been conducted on undertaking citation content analysis, citation classification and definition of the connection between the cited and the citing works [2], [7], [24]. The earliest work on citation content analysis was by Cole [25], who studied the scholarly importance of Robert K. Merton. He examined 123 articles that referred to Merton and found that about half of them mentioned him in a ceremonial manner. While Cole’s work measured a scholar’s scientific standing, Garfield [26] tested the effects of researchers’ critical approach on how they allotted citations in their works. Oppenheim and Renn [27] conducted a citation content analysis to find why certain older documents had a high citation count many years after publication. The results showed that these older works were used as a background reference in about 39% of citing articles and that at least 1 of the 23 older works was cited as other, related work in 20%.

Moravcsik and Murugesan [14] were among the pioneers of citation context analysis. They classified the references of 30 randomly selected articles published between 1968 and 1972 in Physical Review into four pairs of opposite characteristics: conceptual versus operational, evolutionary versus juxtapositional, organic versus perfunctory and confirmative versus negational. Slightly later in 1975, Chubin and Moitra [28] examined 10 full-length articles and 33 letters published between 1968 and 1969 in Physical Review, Nuclear Physics, Physical Review Letters and Physical Letters B. They slightly modified Moravcsik and Murugesan’s [14] taxonomy and proposed two sets of categories, namely affirmative and negational.

Later, Krampen et al. [29] conducted a citation context analysis of 5958 references from 90 randomly selected publications in international scientific psychological journals. The analysis showed that almost 60% directly utilised the theories or concepts (20%), methods (9%) and results (30%) of the cited works. The authors revealed that only 1% of citations are critical in nature, while almost one-third are perfunctory, as an overview of related work (25%) or a mention without additional information (5%). Recently, Teufel et al. [18] undertook citation classification using pattern- and location-based and cue word–based features. Their methodology achieved an accuracy score of 0.77 with a 0.57 macro F-measure using a 10-fold IBK algorithm. Dong and Schäfer [30] identified citations using a limited collection of lexicons and discourse features involving regular expressions applied at the level of the sentence and parts of speech (POS) to find syntactic information. Their model uses Naïve Bayes (NB), a machine learning classifier that has been shown to function well, and it achieved a 0.66 F1 score with their proposed technique.

For the task of citation classification, Abu-Jbara et al. [19] used a range of lexical, structural, and syntactic features. Their data samples employed the SVM classifier with linear kernel and C = 1.0, and their proposed model, the conditional random field (CRF) classifier, to add the adjacent sentences as the citation’s context. Their dataset comprises 14,000 citations and is not publicly available. The model with the SVM classifier achieved a 0.575 F1 score with 0.7 accuracy. Valenzuela et al. [16] undertook this task using a combination of contextual and text-based features with the RF and SVM models. Using a dataset of 450 citations, their supervised approach achieved a 0.57 F1 score with 0.80 precision–recall curve scores.

For this same task of citation text classification, Hassan et al. [5,20] deployed five machine learning models – NB, decision tree, k-nearest neighbour, SVM and RF – grouping the citations into important and non-important classes. They designed a range of textual, contextual and cue word–based features using a dataset of 465 citations. With an RF classifier, their model achieves a 0.91 precision–recall curve with 0.84 receiver operating characteristic (ROC) curve scores. Hassan et al. [31] used the PLOS dataset to track network citation behaviour between countries and universities. Their study analysed the information flows of research articles and citation results, arguing that not all are of equal significance. Further, Small [32] investigated citations’ context by examining the top thousand biomedical articles available at PubMed Central (PMC), and classified citations in terms of their location: whether in the method section or in the non-method section.

To investigate how scholars structure their research contributions through the various forms of citations and how this representation impacts the entire field, Jurgens et al. [22] conducted the largest citation behavioural study to date. They developed a state-of-the-art classifier that employs pattern-based, topic-based and lexical and grammatical features. With a dataset of 1969 citations, their supervised model achieved a 0.53 F1 score. Bakhti et al. [33] designed a semi-automatic citation annotation scheme to classify citation contexts into six types. With a dataset of 2000 citations, they used the SVM model with three-gram features to achieve a 0.837 F1 score and a 0.89 accuracy score.

Similarly, to distinguish between citations based on the polarity of the sentiment expressed in the citation context, Wang et al. [34] explored linguistic patterns. Their approach employed the CRF model with POS and n-gram-based features to achieve a 0.882 F1 score. Recently, Tuarob et al. [35] proposed understanding the evolving relationship between two algorithms by analysing the information available in the citation’s context. They compiled a set of heterogeneous features comprising structural, lexical and morphological, grammatical, sentiment- and venue-based and cue word–based features to segregate citations based on their usage. Their ensemble machine learning method obtained average F1 scores of 0.905 and 0.749, respectively, for the citation classification of binary- and multi-class algorithms. In this work, we downloaded a comprehensive dataset of 9518 citation contexts from the ACL anthology. We devised a deep learning–based CNN citation context classification model that has weight-balance and focal-loss functions to handle the issue of data imbalance. Table 2 presents a detailed overview of the existing studies performed for the citation text classification task. All these studies have used feature-based techniques to classify citations using Binary (B) and Multi (M) classification schemes.

Table 2.

F1 measure and the accuracy results for citation classification studies.

	Model	Classificationscheme	No. ofcitations	Type of feature	Reportedevaluationmeasure
Teufel et al. [18,23]	IBk algorithm, k = 3	M	2829	Cue words and pattern-based	F1: 0.57Acc: 0.77
Dong & Schäfer[30]	Naïve Bayes	M	1768	14 textual, physicaland syntactic	F1: 0.66
Abu-Jbara et al.[19]	SVM: linear kernel, C = 1.0	M	14,000	10 lexical and structural	F1: 0. 575
Valenzuela et al.[16]	Random forest	M	465	12 contextual and textual	F1: 0. 75AUCPR: 0.80
Hassan et al. [20,31]	SVM and random forest	B	465	14 context, cue wordsand textual	AUCPR: 0.84
Small[32]	Logistic regression	B	646,347	TF-IDF-based	Acc: 0.92
Bakhti et al.[33]	Support vector machine	M	2000	2, 3 gram and cue words	F1: 0.837Acc: 0.89
Jurgens et al. [22]	SVM linear kernel	M	1969	30 structural, lexical,morphologicaland grammatical and field-based	F1: 0.53
Wang et al.[34]	CRF with POS and n-gram	M	3500	10 pattern and polarity-based	F1: 0.882
Tuarob et al. [35]	SVM	M	8796	52 context-based and TF-IDFcontent-based	F1: 0.749
Tuarob et al. [35]	X(RF), T(SVM), 0.749	B	8796	52 context-based and TF-IDFcontent-based	F1: 0.905

SVM: support vector machine; CRF: conditional random field; POS: part of speech; M: multi; B: binary; TF-IDF: term frequency-inverse document frequency.

3. Data and methodology

In this section, we describe our dataset and give details of the proposed citation context classification model. Figure 1 shows the high-level architecture of the approach, which takes ACL anthology articles as input data for the task of citation context classification.

We extracted the citation text from the full text of the articles and performed pre-processing to remove any junk characters. Then, we used fastText to generate an embedding vector on a million-word Wikipedia dataset. Finally, we applied the CNN model for the task of citation context classification, involving class-weight and focal-loss function variations to handle the class imbalance problem.

3.1. Dataset

Citations play a fundamental role in acknowledging other scholars’ contributions to a scientific article [36]. Numerous frameworks have been introduced to classify the various functions of citation, spanning multiple classes [23], [37]. Although adequate for expert manual assessment, many citation analysis schemes are too fine-grained, with fields that are too unusual to be reliably established, or subjective, requiring a thorough knowledge of the field or to be automated.

Motivated by the need to analyse large-scale patterns in the scientific research community, we have tackled these issues by combining facets of multiple approaches into a single classification. We collected a comprehensive dataset of citation context from two state-of-the-art studies [22] [23]. We combined both datasets and employed the citation scheme designed by Jurgens et al. [22], comprising 9518 citation contexts extracted from ACL anthology research articles. Since the citation context represents the special purpose that a citation serves for a particular research article, this captures the types of citation function, for instance, to give background knowledge or to provide contrast. Thus our citation scheme combines multiple classifications into just six distinct classes, and the details and examples are as shown in Table 1. The citation annotation mechanism was started by taking 10 sample papers from the ACL. The annotators performed two pre-annotation rounds, to discuss and refine their annotation guidelines. Later, from a dataset of 273 ACL articles, they extracted citation text using ParsCit¹ and tagged it by two natural language processing (NLP) experts. The authors have presented all details regarding the annotation procedure in their published research [22].

Our annotation scheme is similar to that proposed by Abu-Jbara et al. [19] and Teufel et al. [23], which has 6 and 12 classes, respectively. In contrast to Abu-Jbara et al.’s [19] schemes, with two separate classes for contrast and comparison, Teufel et al. [23] have multiple discrete classes for each type of contrast and comparison. In our citation scheme, however, both classes are combined into a single class known as CompareOrContrast: this indicates that the authors have linked their article to the work. In addition, our Motivation scheme covers the citation text that belongs to the motivation behind the research and the future work motivations for the current study. Table 3 shows the distribution of data in each class. In addition, we prepared binary-class data by labelling the Use and Extend classes as important and the remainder as non-important.

Table 3.

Data description of each class.

Class label	No. of citations	% of total
Important
Use	838	8.80
Extend	169	1.78
Non-important
Background	3054	32.09
CompareOrContrast	859	9.03
Motivation	260	2.73
Other	4338	45.58
Total	9518

3.2. Deep learning–based approach

Moreover, for the citation annotation, this section presents the details of our deep learning citation context classification approach. We designed the CNN [38] model using fastText pre-trained embedding vectors. To handle the data imbalance, we proposed using two methods – class-weight and focal-loss functions – to improve the model’s accuracy.

3.2.1. Word embedding by fastText

FastText² is an open-source library designed by Facebook to enable experts to learn text representations and text classifiers for efficient text classification. In NLP, word embedding plays an integral role in learning language semantics. Many research studies focus on using Word2Vec [39] embedding vectors to map similar words to nearby points in vector space settings. While this technique supports learning text representations, it has an intrinsic inability to learn rare word representations due to its complete dependence on a predefined dictionary set. To overcome this issue, we used character-level word embedding, for instance, fastText (Figure 2).

By using character-level information, fastText can attain promising results in word representation and sentence classification, particularly in the case of rare words. Each word is described as n-gram vectors and the word itself. For instance, for the word ‘there’, with n = 2, the resultant word vector is the collection of bigrams <th, he, er, re> <there> (<, > are word boundary indicators to separate words and their n-grams). For example, if the word ‘he’ is in the vocabulary, it is shown as <he>. Hence, the technique retains the sense of the short words that may appear in another word’s n-grams, giving users an innate ability to grasp the meaning of a suffix or prefix.

Moreover, fastText is a conceptual framework in which word embeddings are interpreted as the sum of their character-level embeddings, not just the word itself, thus can be applied to various text problems. In training, this approach is well suited to sharing power through words that are made up of common roots. For example, the two individual words ‘counter’ and ‘revolution’ can build up a useful interpretation of the rare word ‘counterrevolution’, which could otherwise be too unusual to learn from the dictionary. In addition to this effective modelling of rare words, character-level embedding supports learning misspelled or slang words, such as ‘pls’, ‘dogs’, ‘check’ and so on, and thus may help to share links across languages that have a common root. The essence of fastText is that, rather than directly learning a feature vector for a word (as in Word2Vec), it uses character-level representations of each n-gram character, where each word is described as a bag of character n-grams: the overall word embedding is the sum of these character n-grams.

In addition, fastText provides embedding vectors that are pre-trained using the fastText technique over multiple data sources in 157 languages on much larger training datasets. Their use can shorten the training time and support higher quality embedding vectors, improving the classification results. To make use of this, for citation context classification, we employed the million-word vector file³ provided by fastText, trained on the Wikipedia 2017, UMBC webbase corpus and statmt.org news datasets, rather than simple Word2Vec.

3.2.2. Citation context representation by CNN

We designed a CNN [38] to classify citations’ context. The model’s architecture is shown in Figure 3.

Let $S$ be the sentence to be classified, where $x_{i}$ is the word belonging to S of length n. Let k be the k-dimensional embedding vector for S and $x_{i}$ be a word that belongs to k. For a sentence S of length n (occasionally, when necessary, we added padding p)

S = x_{1 + n} = | x_{1} + x_{2} + x_{3} + x_{4} + \dots + x_{n} |

(1)

where | | denotes the concatenation of all words belonging to S.

Generally, $x_{i : i + j}$ indicates the concatenation of words from $i$ to $j$

x_{i : i + j} = | x_{i} + x_{i + 1} + x_{i + 3} + \dots + x_{i + j} |

(2)

A convolution layer includes a filter w, applied to a window of j words to generate a single feature. For instance, feature F is generated by using a window of words $x_{i : i + j - 1}$ . See equation (3)

F = f (w . x_{i : i + j - 1} + b)

(3)

In convolution, each filter F is applied to each word in a sentence, $x_{i : i + j} = x_{1 : j} + x_{2 : j + 1} + \dots + x_{n - j + 1 : n}$ to produce a feature map $c_{i}$ . See equation (4)

c_{i} = c_{1} + c_{2} + c_{3} + \dots + c_{n - j + 1}

(4)

Later, a max-pooling function [40] is applied to extract max features from a feature map c for a particular filter F. See equation (5)

{\hat{c}}_{i} = \max (c_{i})

(5)

The main idea of max-pooling is to find the most prominent and important features with the highest feature map score. This pooling function has the intrinsic ability to deal with sentences of variable length. The above steps describe the procedure by which one feature value is extracted from a filter. Our CNN model uses two one-dimensional (1D) convolution layers with 64 filters of various sizes to extract multiple features. After the first 1D, we applied the relu activation function to add non-linearity and then max-pooling (see equations (6) and (7)). Later, the second 1D convolution layer with 64 filters is followed by a global max-pooling function (see equations (8) and (9))

y_{i}^{(1)} = ReLU (c_{1})

(6)

y_{i}^{(1)} = \max (y_{i}^{(1)})

(7)

y_{i}^{(2)} = ReLU (W^{(2)} y_{i}^{(1)} + b^{(2)})

(8)

y_{i}^{(2)} = max (y_{i}^{(2)})

(9)

Furthermore, the feature sets that belong to the penultimate layers are sent to a fully connected layer to compute the probability score for each class (see equations (10) and (11)), and the sigmoid activation function is applied; in the case of multi-class classification, a k-hot scheme with a sigmoid activation function is applied

y^{(3)} = W^{(2)} y^{(1)} + b^{(2)}

(10)

p_{(i)} = \frac{\exp (y_{k}^{(2)})}{ex p^{(y_{k}^{(2)})} + 1}

(11)

To avoid model overfitting, we applied regularisation by imposing a penalty on the loss function to support a reduction in interdependent learning among the neurons. Since dropout is a known approach to regularisation [41], we imposed it on the penultimate layer to restrict the L2 norm of the weight vector. Dropout eliminates co-adaptation of hidden units by randomly removing (dropping the connections by setting to zero) a proportion p of the hidden units during the forward–backpropagation. Hence, instead of normal weighting, as in equation (12), where z is the output of the penultimate layer $z = {{\hat{c}}_{1}, {\hat{c}}_{2}, \dots, {\hat{c}}_{i}}$ , we applied dropout for the unit y in forward propagation (see equation (13))

y = w . z + b

(12)

y = w . (z * δ) + b

(13)

where * denotes the element-wise multiplication of z with the dropout rate $δ$ , $δ$ ~ Bernoulli (p) and $δ$ = 1 for probability p and in other cases, $δ = 0 .$ During backpropagation, a gradient descent approach is used.

3.2.3. Techniques for addressing the class imbalance in CNN

In this section, we address the class imbalance problem by presenting two classifier-level balancing techniques that make the adjustments in the training algorithm, thus preserving the training datasets. Instead of data-level balancing techniques, such as random under-sampling and random oversampling, we deployed the classifier-level techniques of weight-balancing and focal-loss functions. The following paragraphs give details of the techniques employed.

Weight balancing: Weight balancing is the mechanism used to balance our dataset by adjusting the weight of each class sample in computing the loss function. Typically, each sample and each class is of equal weight; however, in the case of highly relevant classes, we may want certain of them to carry more weight. For instance, in case of the ‘important’ and ‘non-important’ binary classes of citation context, rather than spending time and resources in seeking more samples for the minority class, to which only 10% instances belong in our dataset, we applied weight balancing to both our classes (binary classes) whereby a higher score is awarded to the ‘important’ class of greater significance, so its training examples have a greater impact on the loss function. Hence, the weight-balancing mechanism allows each class to contribute equally to the loss function.

Focal loss: Focal loss [42] is a mechanism that puts more emphasis on hard-classified examples than well-classified samples. In our dataset, we typically had some data samples that were easy to classify. In training, these data samples were segregated with ~99% efficacy, unlike the more challenging and complex data examples, yet yielded poor results. The major issue here is that the easily classified training samples contributed just as much to the total cross-entropy loss value as the more challenging data points that served to improve our overall accuracy, upon correct classification, and thus to which more attention should be paid.

Our multi-class data have seven classes, of which Background and None are dominant and thus became well-classified data examples. To handle this data imbalance, we applied a focal-loss function that down-weights well-classified samples, while training lends more emphasis to the minority-class samples that are harder to classify. Equation (14) presents the general cross-entropy function

Cross Entropy = - \log (p_{t})

(14)

Focal Loss = - (1 - p_{t})^{γ} \log (p_{t})

(15)

where

p_{t} = {\begin{matrix} p & y = 1 \\ 1 - p & otherwise \end{matrix}

Equation (15) shows the focal-loss function that adds a factor of $(1 - p_{t})^{γ}$ to the general cross-entropy function. For γ > 0 decreases the relative loss for majority class or well-classified samples, concentrating more on hard and misclassified samples for loss calculation, in the case of multi-class classification. Therefore, we used the focal-loss function to achieve high accuracy by taking our minority-class samples into account.

4. Experimentations and results

In this section, we describe the details of our experiments and later discuss the classification results achieved for both binary classification and multi-classification. We undertook baseline analysis using SVM and NB classifiers with term frequency-inverse document frequency (TF-IDF) word vectors. Furthermore, we compare existing state-of-the-art citation classification techniques – with each other and with our model.

4.1. Experimental setup

This section explains the experimental configuration used with the CNN model. Along with the hyperparameter settings, we describe the pre-trained embedding vectors adopted for both the binary- and multi-class experiments.

4.1.1. Hyperparameters of CNN

To train our CNN model, after extensive experimentation, we set up the model tuning parameters of training epochs = 60, loss function = binary cross-entropy, learning rate = 0.001, optimiser = Adam and batch size = 200. For the regularisation, we set a dropout rate of 0.3.

We undertook no dataset-specific tuning; rather, we used early-stop mechanisms to avoid model overfitting. For both datasets, we assigned 10% of the datasets as testing data and used the rest as training data. Training took place with the Adam update rule via stochastic gradient descent over shuffled mini-batches of size 200 [43].

4.1.2. Pre-trained embedding vector using fastText

We initialised our embedding vectors using fastText with the publicly available million-word vectors trained on the Wikipedia 2017, UMBC webbase corpus and statmt.org news datasets. The initialisation of word vectors with those acquired from an unsupervised neural language model is a popular way to improve performance in the absence of a large supervised training set [40], [44 -46]. Each citation context was independent of the others and of very limited text size, so using pre-trained vectors that had been trained on a larger dataset helped our CNN model to achieve better efficiency. Words that were not available in the pre-trained word vector were randomly initialised.

4.1.3. Experiment variants

Our dataset suffers from a class imbalance in both its binary and multi-classes. Instead of applying random up-sampling and random down-sampling class imbalance techniques to handle this issue, we used class-weight and focal-loss functions.

The following are details of our two experiments: (a) class-weight function for binary classification: to deal with class imbalance issues; (b) focal-loss function for multi-classification: we used a function that adds a factor of $(1 - p_{t})^{γ}$ to the general cross-entropy function. For γ > 0 decreases the relative loss for the majority class or well-classified samples, concentrating on the hard and misclassified samples for loss calculation.

4.2. Classification results

This section presents the results of the binary- and multi-class experiments and a comparative analysis of existing state-of-the-art citation classification studies and our results.

4.2.1. Experimental results of binary classification task

This section presents the results achieved for binary classification. In this case, we used the complete dataset of 9518 instances, of which 1007 belong to the ‘important’ class and the remaining 8511 to the ‘non-important’ class. To handle the class imbalance problem in our dataset, we applied weight balancing to both binary classes, with the greater weight on the ‘important’ class. Figure 4(a) shows the learning accuracy curve of the CNN model for binary classification. Our model started learning with 53% accuracy and gradually reached 98% training accuracy over 100 epochs. For validation testing, the CNN model achieved around 90% accuracy. Our model achieved a 0.906 F-measure with a 0.906 accuracy score. Figure 4(b) illustrates the precision–recall curve for binary classification, demonstrating that both curves describe a similar steady pattern.

In addition, in Table 4, we have presented classification results for each class using class-weight and without class-weight settings. We have observed a significant increase in the performance in terms of precision, recall and F1 score for important class, using class-weight settings. However, the non-important class does not show any significant increase in results using class weight. Furthermore, to perform a comparison analysis, we also computed baseline results using SVM and NB with TF-IDF feature vectors. Figure 5 shows our achieved baseline and CNN results using a dataset of 9518 citations. In the case of binary classification, our designed deep network with class-weight settings have achieved an accuracy score of 0.90 along with the 0.90 F1 score and outperformed both the baseline results.

Table 4.

Precision, recall and F1 scores for each class using non-class-weight (class imbalance) and class-weight (class balancing) experiments.

	Imbalanced classes			Balanced classes
Class label	Precision	Recall	F1 score	Precision	Recall	F1 score
Non-important	0.93	0.98	0.95	0.93	0.97	0.95
Important	0.68	0.35	0.46	0.75	0.68	0.71

4.2.2. Experimental results of multi-classification task

For the multi-classification, we categorised our dataset of 9518 instances into six classes (Use, Extend, Background, Motivation, CompareOrContrast and Other). As shown in the data distribution in Table 2, the Background and None classes are dominant, and their instances became well-classified examples. To deal with this data imbalance, we applied a focal-loss function (a customised loss function) to downplay these two well-classified samples in training and put more emphasis on the minority-class samples (Use, Extend, Motivation and CompareOrContrast), which are harder to classify. Equation (15) shows the focal-loss function that adds a factor of $(1 - p_{t})^{γ}$ to the general cross-entropy function. Hence, we trained our CNN model for multi-class classification with the value of $γ = 1.0$ .

Figure 6(a) shows the learning and validation accuracy curves for multi-class classification. Our multi-class classification model started learning with 33% accuracy and gradually reached 98% training accuracy over 100 epochs, while, in validation, the multi-class CNN model achieved around 72% accuracy. Table 3 shows the F-measure and accuracy scores of our multi-class model. It achieved a 0.723 F-measure with 0.721 accuracy. Figure 6(b) illustrates the precision–recall curve for multi-class classification, in which both the precision and recall curves show a steady gain in precision and recall.

Table 5 presents a detailed overview of achieved results for all six classes using cross-entropy and focal-loss functions. While using focal loss, we see a significant increase in precision, recall and F1 scores for all classes, especially for non-dominant classes.

Table 5.

Precision, recall and F1 scores for each class using cross-entropy and focal loss as loss functions.

	Evaluation
	Cross-entropy loss function			Focal-loss function
Class label	Precision	Recall	F1 score	Precision	Recall	F1 score
Background	0.7	0.68	0.69	0.74	0.73	0.73
CompareOrContrast	0.49	0.38	0.43	0.52	0.45	0.48
Extends	0.69	0.23	0.35	0.7	0.3	0.5
Uses	0.45	0.3	0.36	0.59	0.34	0.43
Motivation	0.73	0.41	0.52	0.77	0.44	0.56
Other	0.77	0.91	0.83	0.79	0.93	0.86

Moreover, we compared our model with two baseline methods, SVM with TF-IDF features and NB with TF-IDF features. These two models were evaluated using our dataset of 9518 citation contexts. Our model, which takes only the citation text as its input, outperformed both baseline methods, achieving 0.72 accuracy for the multi-classes, respectively (see Figure 7). Although existing citation classification techniques that take as input a list of comprehensive features have shown promising results with their deployed datasets, by using only the citation context, our approach has demonstrated significant accuracy and F-measure scores.

5. Concluding remarks

Citation-based metrics is the indicator used by research administrators for the assessment of research. The most well-known types of metrics, such as h-index, impact factor and source normalised impact per paper (SNIP), are quantitative in nature and take no account of the qualitative value of a citation in a scientific publication since they use absolute citation counts, thus cannot discern its importance. In this study, we have compiled a comprehensive dataset of 9518 citations and classified them based on their textual content. Further, for the citation context classification, we have designed a deep neural network–based technique using CNN. This uses a focal-loss and class-weight mechanism to handle the issue of class imbalance in the citation dataset.

Notably, our approach takes only the citation context as its input to achieve over 90% and 70% accuracy, respectively, for binary- and multi-class citation classification. In the future, we seek to explore the other deep learning models with the latest word embedding techniques to further improve the citation classification results.

The data and code used in this article can be downloaded from https://github.com/slab-itu/JIS_Citation_Context.

Footnotes

Acknowledgements

The authors, therefore, greatly acknowledge the Deanship of Scientific Research’s (DSR) technical and financial support.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

This project was funded by the Deanship of Scientific Research (DSR), King Abdulaziz University, Jeddah, under grant No. RG-14-611-40. The authors, therefore, gratefully acknowledge DSR technical and financial support.

ORCID iD

Saeed-Ul Hassan

Notes

References

Boyack

van Eck

Colavizza

et al. Characterizing in-text citations in scientific articles: a large-scale analysis. J Informetr 2018; 12(1): 59–73.

Bornmann

Wray

Haunschild

. Citation concept analysis (CCA): a new form of citation analysis revealing the usefulness of concepts for other researchers illustrated by exemplary case studies including classic books by Thomas S Kuhn and Karl R Popper. Scientometrics 2020; 122(2): 1051–1074.

Saier

Färber

. UnarXive: a large scholarly data set with publications’ full-text, annotated in-text citations, and links to metadata. Scientometrics 2020; 125: 1–24.

Safder

Hassan

S-U

Visvizi

et al. Deep learning-based extraction of algorithmic metadata in full-text scholarly documents. Inf Process Manag 2020; 57(6): 102269.

Hassan

S-U

Imran

Iqbal

et al. Deep context of citations using machine-learning models in scholarly full-text articles. Scientometrics 2018; 177: 1–18.

Safder

Hassan

S-U

. Bibliometric-enhanced information retrieval: a novel deep feature engineering approach for algorithm searching from full-text publications. Scientometrics 2019; 119: 1–21.

Tahamtan

Bornmann

. What do citation counts measure? An updated review of studies on citations in scientific documents published between 2006 and 2018. Scientometrics 2019; 121(3): 1635–1684.

Gross

. College libraries and chemical education. Science 1927; 66(1713): 385–389.

Brittain

. Information and its users: a review with special reference to the social science. Newyork: John Wiley, 1970.

10.

Cole

. Social stratification in science. Chicago: Univ Chicago Press, 1973.

11.

Borgman

. Bibliometrics and scholarly communication: editor’s introduction. Commun Res 1989; 16(5): 583–599.

12.

Luukkonen

. Is scientists’ publishing behaviour rewardseeking? Scientometrics 1992; 24(2): 297–319.

13.

Frost

. The use of citations in literary research: a preliminary classification of citation functions. Libr Q 1979; 49(4): 399–414.

14.

Moravcsik

Murugesan

. Some results on the function and quality of citations. Soc Stud Sci 1975; 5(1): 86–92.

15.

Zhu

Turney

Lemire

et al. Measuring academic influence: not all citations are equal. J Assoc Inf Sci Technol 2015; 66(2): 408–427.

16.

Valenzuela

Etzioni

. Identifying meaningful citations. In: Proceedings of the workshops at the 29th AAAI conference on artificial intelligence, Austin, TX, 25-30 January 2015, pp. 21-26. Palo Alto, CA: AAAI.

17.

Pride

Knoth

. Incidental or influential?-challenges in automatically detecting citation importance using publication full texts. In: Proceedings of the international conference on theory and practice of digital libraries, Lyon, 25-27 August 2017, pp. 572–578. Cham: Springer.

18.

Teufel

Siddharthan

Tidhar

. Automatic classification of citation function. In: Proceedings of the 2006 conference on empirical methods in natural language processing, Sydney, NWS, Australia, 22-23 July 2006, pp. 103–110. New York: ACM.

19.

Abu-Jbara

Ezra

Radev

. Purpose and polarity of citation: towards nlp-based bibliometrics. In: Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: human language technologies, Atlanta, Georgia, 9-14 June 2013, pp. 596–606. New York: ACM.

20.

Hassan

S-U

Akram

Haddawy

. Identifying important citations using contextual information from full text. In: Proceedings of the 17th ACM/IEEE joint conference on digital libraries, Toronto, ON, Canada, 19-23 June 2017, pp. 41–48. New York: ACM/IEEE.

21.

Hassan

S-U

Iqbal

Imran

et al. Mining the context of citations in scientific publications. In: Proceedings of the international conference on Asian digital libraries, Hamilton, 19-22 November 2018, pp. 316–322. Cham: Springer.

22.

Jurgens

Kumar

Hoover

et al. Measuring the evolution of a scientific field through citation frames. Trans Assoc Comput Linguist 2018; 6: 391–406.

23.

Teufel

Siddharthan

Tidhar

. An annotation scheme for citation function. In: Proceedings of the 7th sigdial workshop on discourse and dialogue, Sydney, NSW, Australia, 15-16 July 2006, pp80–87. New York: ACM.

24.

Taşkın

. A content-based citation analysis study based on text categorization. Scientometrics 2018; 114(1): 335–357.

25.

Cole

. The growth of scientific knowledge: theories of deviance as a case study. In: Coser

(ed.) The idea of social structure: papers in honor of Robert K Merton. New York: Harcourt Brace Jovanovich, 1975, pp. 175–220.

26.

Garfield

. High impact science and the case of Arthur Jensen. Curr Contents 1978; 41: 5–15.

27.

Oppenheim

Renn

. Highly cited old papers and the reasons why they continue to be cited. J Am Soc Inf Sci 1978; 29(5): 225–231.

28.

Chubin

Moitra

. Content analysis of references: adjunct or alternative to citation counting? Soc Stud Sci 1975; 5(4): 423–441.

29.

Krampen

Burkard

Montada

. Wissenschaftsforschung in der psychologie. Gottingen: Hogrefe Verlag für Psychologie, 2002.

30.

Dong

Schäfer

. Ensemble-style self-training on citation classification. In: Proceedings of the 5th international joint conference on natural language processing, Chiang Mai, Thailand, 8-13 November 2011, pp. 623–631. Bangkok, Thailand: Asian Federation of Natural Language Processing

31.

Hassan

S-U

Safder

Akram

et al. A novel machine-learning approach to measuring scientific knowledge flows using citation context analysis. Scientometrics 2018; 116(2): 973–996.

32.

Small

. Characterizing highly cited method and non-method papers using citation contexts: the role of uncertainty. J Informetr 2018; 12(2): 461–480.

33.

Bakhti

Niu

Nyamawe

. Semi-automatic annotation for citation function classification. In: Proceedings of the 2018 international conference on control, artificial intelligence, robotics & optimization (ICCAIRO), Prague, Czech Republic, 19-21 May 2018, pp. 43–47. https://www.computer.org/csdl/proceedings-article/iccairo/2018/08698361/19wAYlwrbyg

34.

Wang

Leng

Ren

et al. Sentiment classification based on linguistic patterns in citation context. Curr Sci 2019; 10: 606–616.

35.

Tuarob

Kang

Wettayakom

et al. Automatic classification of algorithm citation functions in scientific literature. IEEE Trans Knowl Data Eng 2020; 32(10): 1881–1896.

36.

Garfield

Merton

. Citation indexing: its theory and application in science, technology, and humanities, volume. 8. New York: Wiley, 1979.

37.

Pham

Hoffmann

. A new approach for scientific citation classification using cue phrases. In: Proceedings of the Australasian joint conference on artificial intelligence, Perth, WA, Australia, 3-5 December 2003, pp. 759–771. Berlin, Heidelberg: Springer.

38.

Kim

. Convolutional neural networks for sentence classification. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), Doha, Qatar, 25-29 October 2014, pp. 1746-1751. Stroudsburg, PA: ACL.

39.

Mikolov

Sutskever

Chen

et al. Distributed representations of words and phrases and their compositionality. In: Proceedings of the advances in neural information processing systems, Lake Tahoe, CA, 12-17 December 2013, pp. 3111–3119. Red Hook, NY: Curram Associates Inc.

40.

Collobert

Weston

Bottou

et al. Natural language processing (almost) from scratch. J Mach Learn Res 2011; 12: 2493–2537.

41.

Szegedy

Liu

Jia

et al. Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Boston, MA, 7-12 June 2015, pp. 1-9. New York: IEEE.

42.

Lin

T-Y

Goyal

Girshick

et al. Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision, Venice, 22-29 October 2017, pp. 2980–2988. New York: IEEE.

43.

Schmidhuber

. Deep learning in neural networks: an overview. Neural Netw 2015; 61: 85–117.

44.

Socher

Huang

Pennin

et al. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In: Proceedings of the advances in neural information processing systems, Granada, 12-15 December 2011, pp. 801–809. Red Hook, NY: Curran Associates Inc.

45.

Iyyer

Manjunatha

Boyd-Graber

et al. Deep unordered composition rivals syntactic methods for text classification. In: Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (Volume 1: Long Papers), volume. 1, Beijing, China, 26-31 July 2015, pp. 1681–1691. Stroudsburg, PA: ACL.

46.

Paul

Nawaz

Korkontzelos

et al. News search using discourse analytics. In: Proceedings of the 2013 digital heritage international congress (DigitalHeritage), volume. 1, Marseille, 28 October-1 November2013, pp. 597-604. New York: IEEE.