A novel filter feature selection method for text classification: Extensive Feature Selector

Abstract

As the huge dimensionality of textual data restrains the classification accuracy, it is essential to apply feature selection (FS) methods as dimension reduction step in text classification (TC) domain. Most of the FS methods for TC contain several number of probabilities. In this study, we proposed a new FS method named as Extensive Feature Selector (EFS), which benefits from corpus-based and class-based probabilities in its calculations. The performance of EFS is compared with nine well-known FS methods, namely, Chi-Squared (CHI2), Class Discriminating Measure (CDM), Discriminative Power Measure (DPM), Odds Ratio (OR), Distinguishing Feature Selector (DFS), Comprehensively Measure Feature Selection (CMFS), Discriminative Feature Selection (DFSS), Normalised Difference Measure (NDM) and Max–Min Ratio (MMR) using Multinomial Naive Bayes (MNB), Support-Vector Machines (SVMs) and k-Nearest Neighbour (KNN) classifiers on four benchmark data sets. These data sets are Reuters-21578, 20-Newsgroup, Mini 20-Newsgroup and Polarity. The experiments were carried out for six different feature sizes which are 10, 30, 50, 100, 300 and 500. Experimental results show that the performance of EFS method is more successful than the other nine methods in most cases according to micro-F1 and macro-F1 scores.

Keywords

Dimension reduction feature selection text classification

1. Introduction

Owing to the use of Internet and electronic devices, digital text documents increased rapidly. With the increase in these textual data, it is necessary to classify them according to their content in order to benefit from these data. The smallest constituent of text documents is word which plays an important role in classification of text documents [1,2]. Text classification (TC) – especially studied in computer science, information retrieval and information science – has found different applications in various domains such as genre classification [3], spam e-mail filtering [4], short message service (SMS) spam filtering [5], author identification [6], topic detection [7], medical document classification [8 –11] and web page classification [12].

Generally, TC contains several steps which are pre-processing, feature extraction, feature selection (FS), feature weighting and classification. As a pre-processing step, tokenisation, stop-word removal, lower-case conversion and stemming techniques are utilised for improving the performance of the classification [13]. The processed tokens are employed to build a list of the features for the textual data. Bag-of-words (BoWs) and vector space model are utilised in order to realise feature extraction step [14]. Then, vector space is created according to the frequency of these features. In this vector space, each term forms a dimension. Even an ordinary text, data set consists of hundreds of thousands of features. The high-dimension vector space is compelling, and the size of this vector space needs to be reduced. While doing this operation, a FS method is required to add the important features to the vector space. During feature weighting step, term frequency–inverse document frequency (TF–IDF) is one of the most used weighting methods [15].

FS methods are generally classified into three different categories [16]. These are filters, wrappers and embedded methods. While filters do not require any classification algorithm in its flow, wrappers and embedded methods need a classification algorithm during the construction of the final feature subset. Requirement of any classification algorithm may increase running time during the selection of feature subset. Because of these reasons, filter-based FS methods are mostly preferred than wrappers and embedded methods.

In this study, a new filter FS method named as the Extensive Feature Selector (EFS) is proposed for TC. The method has been proposed considering the class-based and corpus-based discrimination of the feature. So, class-based probabilities and corpus-based probabilities are utilised for selecting more distinctive features. According to the experimental results presented in the next parts of the study, the proposed method is more successful than the other state-of-the-art methods in terms of macro-F1 and micro-F1 in most cases.

The rest of this study is organised as follows. In section 2, related works about FS methods are summarised. FS methods that are compared with EFS method are briefly described in section 3. The proposed method is explained in section 4 in detail. In section 5, we described experimental settings consisting of data sets, classifiers, success measures, similarity analysis and accuracy analysis. Finally, a conclusion is given in section 6.

2. Related works

In the literature, there are many studies related to filter-based FS methods, FS schemes and two-stage schemes including the combination of filter-based and wrapper-based techniques in TC domain. Filter FS methods are commonly preferred thanks to low processing time. Document frequency [17], information gain [18], mutual information [19], improved Gini index [20] and deviation from Poisson’s distribution [21] are some examples to filter-based FS methods. Uysal and Gunal [22] proposed a new FS method, namely, Distinguishing Feature Selector (DFS). While DFS assigns high scores to discriminative features, it assigns low scores to irrelevant features according to some pre-defined criteria. Yang et al. [23] presented a novel FS technique, namely, Comprehensively Measure Feature Selection (CMFS). CMFS computes the importance of a feature considering both inter-class and intra-class distributions. Experimental studies showed that CMFS is more successful than six FS techniques on three benchmark data sets using two classification algorithms. Zong et al. [24] proposed a new FS method, namely, discriminative feature selection (DFSS) which takes into account both the correlation between terms and classes and the semantic similarity between terms and documents. Experimental works showed that the performance of DFSS is competitive with other FS techniques. Rehman et al. proposed several FS methods for TC [25 –27]. Rehman et al. [27] proposed a novel FS method, namely, relative discrimination criterion (RDC) which takes into account feature counts. RDC is more successful than four different FS metrics on four public data sets using two classifiers. In a study [25], they proposed a novel FS method, namely, Normalised Difference Measure (NDM) which considers the relative document frequencies. NDM is more successful than traditional FS algorithms on different data sets using some classifiers. In addition to this study, they proposed a different new FS technique [26], namely, Max–Min Ratio (MMR) which selects smaller subsets of more informative features even in data sets including highly skewed categories. Kim and Zzang [28] proposed a novel filter-based FS technique called as Trigonometric Comparison Measure (TCM) taking into account relative document frequencies. The method uses true positive rate and false positive rate to determine informative terms for improving the performance of classification. In experiments, the method was compared with eight successful filter-based FS methods on 10 data sets with two classifiers. Experimental studies showed that TCM is more successful than other methods for TC. Pinheiro et al. [29] proposed a filter-based FS method, namely, ALOFT (At Least One FeaTure) which considers specific characteristics of TC domain. The method guarantees that every document has at least one feature in the training set. Also, ALOFT can find out optimum features in a data-driven way. Experimental studies showed that the performance of the method is more successful than five FS methods on three public data sets with two classifiers. In addition to ALOFT, Pinheiro et al. [30] proposed two filter-based FS methods called as the Maximum f Features per Document (MFD) and the Maximum f Features per Document-Reduced (MFDR) for TC domain. These methods guarantee that each document contributes to the final feature set. Experimental studies showed that the proposed methods give better performance than Variable Ranking and ALOFT algorithms with three FS methods on four benchmark data sets using the Naive Bayes classifier.

In addition to filter-based FS methods, there are several FS schemes in the literature [1,2]. In filter-based FS schemes, features are assigned to a score based on their distinguishing power. These features are then sorted in descending order according to their scores. At the last stage, the best N features are selected. Uysal [1] proposed an improved global feature selection scheme (IGFSS) which modifies the standard FS scheme to obtain more informative feature set representing each class equally. Using a local FS method, a label is assigned to each feature taking into account the distinguishing powers of the features in the classes. While some of the classes are represented in a good way by the selected features, several classes may not be represented well. In order to fix this situation, IGFSS constructs the feature set by selecting equal number of features representing each class. Based on this study, a new scheme using variable number of features representing each class during feature set construction process was proposed by Agnihotri et al. [2]. They stated that using equal number of features creates problems in multi-class unbalanced data sets. If an equal number of features are taken from each class, some important features in classes may not be not selected. To solve this problem, based on the distribution of the features in the classes, a variable number of features were selected from each class. Agnihotri et al. [31] proposed a novel Soft Voting Technique (SVT) in order to overcome some shortcomings of IGFSS. SVT utilised the weighted average of three methods such as Odds Ratio (OR), Correlation Coefficient (CC) and Galavotti–Sebastiani–Simi (GSS) Coefficient. The SVT was compared through four classifiers applied on five benchmark data sets. Experimental studies showed that SVT attains an important improvement on classification performance in comparison to standard techniques.

Furthermore, many hybrid methods by combining filters and wrappers have been proposed [32 –34]. Uysal and Gunal [32] proposed an approach, namely, genetic algorithm oriented latent semantic features (GALSF), which consists of FS methods and feature transformation stages. First, filter-based methods were applied for selecting a subset of features. Second, latent semantic indexing (LSI) empowered by genetic algorithm was employed. Experimental studies showed that GALSF obtains better performance than both LSI and filter-based FS techniques on public data sets for different feature sizes. Ghareb et al. [33] proposed a hybrid FS technique based on genetic algorithm (GA). The technique utilises a hybrid technique which takes advantage of both filter-based methods and enhanced GA to solve high dimension of the feature size and increase classification performance. The experimental results showed that hybrid approaches give better performance than only filter-based methods for dimension reduction stage. Uysal [34] analysed two-stage FS methods which consists of local FS methods, feature transformation and wrapper-based FS methods. In this study, four local FS methods were utilised in conjunction with three feature construction techniques named as maximum globalisation technique (MAX), weighted averaging globalisation technique (AVG) and selecting equal number of features for each class (EQ). Then, principal component analysis (PCA), LSI and GA were used. According to experimental results, AVG and EQ methods generally obtained better performance than MAX method. Also, PCA was more successful than LSI and GA methods in feature transformation stage.

3. FS methods

There are many filter-based FS methods in TC domain. In this study, nine filter methods for FS stage were employed. These methods are CHI2, CDM, DPM, OR, DFS, CMFS, DFSS, NDM and MMR. Tables 1 and 2 express some common notations about FS methods for TC.

Table 1.

Contingency table of term t and class $C_{j}$ .

	Containing term t (t)	Not containing term t $(\bar{t})$
In class $C_{j}$ $(C_{j})$	a	c
Not in class $C_{j}$ $(\bar{C_{j}})$	b	d

Table 2.

Preliminary notations for FS methods.

Notation	Value	Meaning
a	$count (t, C_{j})$	Number of documents containing term t in the class $C_{j}$
b	$count (t, \bar{C_{j}})$	Number of documents containing term t in other classes $(\bar{C_{j}})$
c	$count (\bar{t}, C_{j})$	Number of documents not containing term $t$ in the class $C_{j}$
d	$count (\bar{t}, \bar{C_{j}})$	Number of documents not containing term $t$ in other classes $(\bar{C_{j}})$
e	$frequency (t, C_{j})$	The number of term t in the class $C_{j}$
f	$frequency (t, \bar{C_{j}})$	The number of term t in other classes $(\bar{C_{j}})$
N	(a + b + c + d)	Total number of documents in all classes
M	$count (C_{j})$	Total number of classes
P(t)	(a + b)/N	The probability of term t
$P (\bar{t})$	(c + d)/N	The probability of absence of term t
$P (C_{j})$	(a + c)/N	The probability of class $C_{j}$
$P (\bar{C_{j}})$	(b + d)/N	The probability of absence of class $C_{j}$
$P (t, C_{j})$	a/N	The probability of term t for being in class $C_{j}$
$P (t, \bar{C_{j}})$	b/N	The probability of term t for being in other classes $(\bar{C_{j}})$
$P (\bar{t}, C_{j})$	c/N	The probability of absence of term $t$ for being in class $C_{j}$
$P (\bar{t}, \bar{C_{j}})$	d/N	The probability of absence of term $t$ for being in other classes $(\bar{C_{j}})$
$P (t \| C_{j})$	a/(a + c)	The probability of term t when class $C_{j}$ is present
$P (\bar{t} \| C_{j})$	c/(a + c)	The probability of absence of term $t$ when class $C_{j}$ is present
$P (t \| \bar{C_{j}})$	b/(b + d)	The probability of term t when other classes $(\bar{C_{j}})$ are present
$P (\bar{t} \| \bar{C_{j}})$	d/(b + d)	The probability of absence of term $t$ when other classes $(\bar{C_{j}})$ are present
$P (C_{j} \| t)$	a/(a + b)	The probability of class $C_{j}$ when term t is present
$P (\bar{C_{j}} \| t)$	b/(a + b)	The probability of absence of class $C_{j}$ when term t is present
$P (C_{j} \| \bar{t})$	c/(c + d)	The probability of class $C_{j}$ when term $t$ is absent
$P (\bar{C_{j}} \| \bar{t})$	d/(c + d)	The probability of absence of class $C_{j}$ when term $t$ is absent

3.1. CHI2

As a popular FS method, CHI2 [22] method computes divergence from the distribution expected if one regards occurrence of the feature t is independent of the class $C_{j}$ . The mathematical formula of the CHI2 can be computed as below

CHI 2 (t, C_{j}) = \frac{N \cdot {[P (t, C_{j}) \cdot P (\bar{t}, \bar{C_{j}}) - P (t, \bar{C_{j}}) \cdot P (\bar{t}, C_{j})]}^{2}}{P (t) \cdot P (\bar{t}) \cdot P (C_{j}) \cdot P (\bar{C_{j}})}

(1)

CHI2 score of a feature is computed for each class, and it is necessary to globalise class-based scores in order to obtain a unique score for each feature. There are three methods named as which are sum, weighted sum and maximum to globalise this score. In this study, weighted average is preferred to globalise CHI2 score as follows

CHI 2 (t) = P (C_{j}) \cdot CHI 2 (t, C_{j})

(2)

3.2. CDM

CDM derives from OR proposed by Chen et al. [35]. CDM method is much simpler than Multi-Class Odds Ratio (MC-OR). CDM is better than MC-OR according to performance. The formula of the method is as follows

CDM (t) = \sum_{j = 1}^{M} | \log \frac{P (t | C_{j})}{P (t | \bar{C_{j}})} |

(3)

3.3. DPM

DPM focuses on both positive and negative distinguishing features. Also, the method has low computation time [36]. The aim of DPM method is to select features which reveal larger differences among the classes. DPM selects higher distinctive features for TC. The formula of DPM can be defined as follows

DPM (t) = \sum_{j = 1}^{M} | P (t | C_{j}) - P (t | \bar{C_{j}}) |

(4)

3.4. MC-OR

Since the experiments were performed on multi-class data sets, MC-OR is utilised instead of the original OR [35] in this study. MC-OR method selects not only positive features but also negative features. The formula of the MC-OR method is as follows

MC - OR (t) = \sum_{j = 1}^{M} | \log \frac{P (t | C_{j}) \cdot (1 - P (t | \bar{C_{j}}))}{P (t | \bar{C_{j}}) \cdot (1 - P (t | C_{j}))} |

(5)

3.5. DFS

DFS relies on the four pre-defined criteria. DFS selects informative features, while eliminating uninformative features taking into account certain requirements on feature characteristics. The formula of DFS [22] method is constituted as follows

DFS (t) = \sum_{j = 1}^{M} \frac{P (C_{j} | t)}{P (\bar{t} | C_{j}) + P (t | \bar{C_{j}}) + 1}

(6)

3.6. CMFS

CMFS investigates the importance of a feature both in inter-class and intra-class, comprehensively [23]. The formula of CMFS method is as follows

CMFS (t) = \sum_{j = 1}^{M} P (C_{j}) \cdot P (t | C_{j}) \cdot P (C_{j} | t)

(7)

3.7. DFSS

DFSS [24] selects features according to some criteria. These are selecting features with higher term frequency, higher occurrence rate and ignoring features occurring in all of the documents. DFSS selects features which are capable of discrimination between classes. According to these criteria, DFSS method is demonstrated as follows

DFSS (t, c_{j}) = (\frac{\frac{e}{a}}{\frac{f}{b}}) \cdot P (C_{j} | t) \cdot P (t | C_{j}) \cdot | P (C_{j} | t) - P (C_{j} | \bar{t}) |

(8)

This method was globalised by the maximum globalisation technique

DFSS (t) = \max_{j = 1}^{M} (DFSS (t, c_{j}))

(9)

3.8. NDM

NDM is developed by Rehman et al. [25] considering relative documents frequencies. Balanced Accuracy Measure [14] computes score of feature by taking the difference of its document frequency in the positive class and negative class. NDM is constructed by improving the Balanced Accuracy Measure. The formula of the method is defined as follows

NDM (t) = \sum_{j = 1}^{M} \frac{| P (t | C_{j}) - P (t | \bar{C_{j}}) |}{\min (P (t | C_{j}), P (t | \bar{C_{j}}))}

(10)

3.9. MMR

Balanced Accuracy Measure treats two features having the same difference between document frequency in the positive class and negative class. Also, NDM can assign high score to highly uninformative sparse features in large and highly skewed text data sets. However, MMR is the improved version of Balanced Accuracy Measure and NDM. MMR gives better performance on data sets including highly skewed classes [26]. The method is combination of true positives, false positives and their differences. The method is demonstrated as follows

MMR (t) = \sum_{j = 1}^{M} \frac{| P (t | C_{j}) - P (t | \bar{C_{j}}) |}{\min (P (t | C_{j}), P (t | \bar{C_{j}}))} \cdot \max (P (t | C_{j}), P (t | \bar{C_{j}}))

(11)

4. The proposed method

In this study, a new FS method has been proposed considering the class-based and corpus-based discrimination of the feature. It means that the proposed method computes the significance of a term from both class-based probabilities and corpus-based probabilities.

Considering the class-based discrimination of the feature:

If the term is more probably to be found in all documents within the class, the discrimination of the term for the relevant class increases; therefore, it must be assigned a high score.

If the term is more probably to be not found in some documents within the class, the discrimination of the term for the relevant class decreases; it must be assigned a low score.

If the term is more probably to be in other classes than the relevant class, the discrimination of the term decreases for the relevant class; it must be assigned a low score.

Based on the above requirements, the class-based discrimination score of the term is as follows

EF S_{class - based} = (\frac{P (t | C_{j})}{P (\bar{t} | C_{j}) + P (t | \bar{C_{j}}) + 1})

(12)

So, if the term is occurred in the documents in the relevant class, high score must be assigned. Similarly, if the term is not occurred in some of the documents in the relevant class, low score must be assigned. Also, if the term occurs in other classes, low score must be assigned. To assign a fair score, we added the calculated value to the numerator of the formula while it is added to the denominator of the formula to assign a low score. These three probabilities which are $P (t | C_{j})$ , $P (\bar{t} | C_{j})$ and $P (t | \bar{C_{j}})$ satisfy the above three requirements, respectively. From this formula, if the term exists only in all documents of a class, it will be assigned 1.0 score as the top score. We have added ‘+1’ to the denominator in the formula in order to prevent the denominator become 0 for some cases.

Considering the corpus-based discrimination of the feature:

If the class is more probably to be in case of presence of term, the corpus-based discrimination of the feature increases; therefore, it must be assigned a high score.

If the class is less probably to be in case of presence of term, the corpus-based discrimination of the feature decreases; it must be assigned a low score.

If the class is more probably to be in case of absence of term, the corpus-based discrimination of the feature decreases; so, it must be assigned a low score.

Based on the above requirements, the corpus-based discrimination score of the term is as follows

EF S_{corpus - based} = (\frac{P (C_{j} | t)}{P (\bar{C_{j}} | t) + P (C_{j} | \bar{t}) + 1})

(13)

So, if the probability of relevant class increases in case of presence of term, high score must be assigned. Similarly, if the probability of relevant class is low in case of absence of term, low score must be assigned. Also, the probability of relevant class is high in case of absence of term, low score is assigned. To assign a high score, we added the calculated value to the numerator of the formula while it is added to the denominator of the formula to assign a low score. These three probabilities which are $P (C_{j} | t)$ , $P (\bar{C_{j}} | t)$ and $P (C_{j} | \bar{t})$ satisfy the above three requirements, respectively. From this formula, if all of the occurrences of a term is found in relevant class, then the highest score is assigned to 1.0 score as the top score. We have added ‘+1’ to the denominator in the formula in order to prevent the denominator become 0 for some cases.

The class-based and the corpus-based scores range from 0.0 to 1.0. As we multiplied these two values by each other, the final score to be obtained is in the range 0.0–1.0. If we had added these two scores, the score would not have been in the 0.0–1.0 range and would have affected each other. Considering the class-based and the corpus-based discrimination of the feature, the formula of the proposed method EFS is as follows

EFS (t) = \sum_{j = 1}^{M} (\frac{P (t | C_{j})}{P (\bar{t} | C_{j}) + P (t | \bar{C_{j}}) + 1}) \cdot (\frac{P (C_{j} | t)}{P (\bar{C_{j}} | t) + P (C_{j} | \bar{t}) + 1})

(14)

The formula of the EFS method consists of two parts which are class-based and corpus-based. The value ranges of both parts are between 0 and 1. If the feature exists only in all documents of a single class, the values of both parts of the formula become ‘1.0’. So, the highest score for the feature is 1.0 by multiplying two parts of the formula. However, if the feature occurs in all documents of one class as well as in some classes, both class-based and corpus-based scores decrease. In addition, when class-based value is 0, corpus-based value cannot be different from 0 or vice versa. So, there is no damping of class-based and corpus-based scores each other.

EFS is similar to CMFS and DFS in some ways. The method resembles CMFS by multiplying the numerator of the class-based part and corpus-based part. Furthermore, the method resembles DFS with the denominator of the class-based part and the numerator of the corpus-based part. However, the formula of EFS is constructed after comprehensive studies. Experimental studies also reveal the success of the EFS method.

EFS has been studied extensively. Score assignment was made to make the feature more distinctive, considering both class-based probabilities and corpus-based probabilities. The final score was obtained by multiplying the class-based and corpus-based scores. Unlike the FS methods in the literature, EFS chooses distinctive features by considering many probabilities.

A sample collection is presented in Table 3 to show how EFS works. However, information about feature occurrences and scores assigned by EFS is given in Table 4.

\begin{matrix} EFS (' fish') = (\frac{\frac{0}{1}}{\frac{1}{1} + \frac{2}{5} + 1} * \frac{\frac{0}{2}}{\frac{2}{2} + \frac{1}{4} + 1}) + (\frac{\frac{2}{2}}{\frac{0}{2} + \frac{0}{3} + 1} * \frac{\frac{2}{2}}{\frac{0}{2} + \frac{0}{3} + 1}) \\ + (\frac{\frac{0}{2}}{\frac{2}{2} + \frac{2}{3} + 1} * \frac{\frac{0}{2}}{\frac{2}{2} + \frac{2}{3} + 1}) = 1.000 \end{matrix}

\begin{matrix} EFS (' mouse') = (\frac{\frac{0}{1}}{\frac{1}{1} + \frac{3}{4} + 1} * \frac{\frac{0}{3}}{\frac{3}{3} + \frac{1}{2} + 1}) + (\frac{\frac{2}{2}}{\frac{0}{2} + \frac{1}{3} + 1} * \frac{\frac{2}{3}}{\frac{1}{3} + \frac{0}{2} + 1}) \\ + (\frac{\frac{1}{2}}{\frac{1}{2} + \frac{2}{3} + 1} * \frac{\frac{1}{3}}{\frac{2}{3} + \frac{2}{3} + 1}) = 0.410 \end{matrix}

\begin{matrix} EFS (' dog') = (\frac{\frac{1}{1}}{\frac{0}{1} + \frac{1}{4} + 1} * \frac{\frac{1}{2}}{\frac{1}{2} + \frac{0}{3} + 1}) + (\frac{\frac{1}{2}}{\frac{1}{2} + \frac{1}{3} + 1} * \frac{\frac{1}{2}}{\frac{1}{2} + \frac{1}{3} + 1}) \\ + (\frac{\frac{0}{2}}{\frac{2}{2} + \frac{2}{3} + 1} * \frac{\frac{0}{2}}{\frac{2}{2} + \frac{2}{3} + 1}) = 0.341 \end{matrix}

\begin{matrix} EFS (' cat') = (\frac{\frac{1}{1}}{\frac{0}{1} + \frac{4}{4} + 1} * \frac{\frac{1}{5}}{\frac{4}{5} + \frac{0}{0} + 1}) + (\frac{\frac{2}{2}}{\frac{0}{2} + \frac{3}{3} + 1} * \frac{\frac{2}{5}}{\frac{3}{5} + \frac{0}{0} + 1}) \\ + (\frac{\frac{2}{2}}{\frac{0}{2} + \frac{3}{3} + 1} * \frac{\frac{2}{5}}{\frac{3}{5} + \frac{0}{0} + 1}) = 0.305 \end{matrix}

Table 3.

Sample collection.

Class label	Document name	Content
C1	Doc1	cat dog
C2	Doc2	cat fish dog mouse
C2	Doc3	cat fish mouse
C3	Doc4	cat mouse
C3	Doc5	cat

Table 4.

Information about feature occurrences and assigned scores.

Feature	Document frequency	Class occurrences	Score
fish	2	1	1.000
mouse	3	2	0.410
dog	2	2	0.341
cat	5	3	0.305

In this sample collection, maximum score is assigned to ‘fish’ that occurs in all documents of only one class, namely, C1. Also, the features ‘mouse’ and ‘dog’ occur in two classes. Their scores are not equal because document frequency of the features is not the same. So, the feature ‘mouse’ is more distinctive than the feature ‘dog’ due to its occurrence in related classes. The feature ‘cat’ is the most uninformative feature because it occurs in all of the documents in three classes. As the feature ‘fish’ occurs in all documents of just one class, ‘cat’ occurs in all documents of all classes. While ‘fish’ is assigned to the highest score of 1.0, ‘cat’ is assigned the lowest score of 0.305 in this sample scenario. To sum up, EFS logically orders the features according to their class-based and corpus-based discriminating power as ‘fish’, ‘mouse’, ‘dog’ and ‘cat’. As seen in Table 5, CHI2 method ranked the features similar to EFS. So, EFS assigns fair scores to the features and EFS is a good FS method in terms of score assignment.

Table 5.

Feature scores for each FS method.

Feature	CHI2	CDM	DPM	MC-OR	DFS	CMFS	DFSS	NDM	MMR	EFS
fish	6.94	6.91	1.00	6.91	1.000	0.400	0.00	1000.0	5.00	1.000
mouse	4.82	1.38	0.83	6.50	0.654	0.333	1.33	2.333	2.22	0.410
dog	2.93	1.79	0.92	6.62	0.673	0.200	1.50	3.500	3.25	0.341
cat	0.00	0.00	0.00	200.0	0.500	0.360	0.00	0.000	0.00	0.305

FS: Feature Selection; CHI2: Chi-Squared; CDM: Class Discriminating Measure; DPM: Discriminative Power Measure; MC-OR: Multi-Class Odds Ratio; DFS: Distinguishing Feature Selector; CMFS: Comprehensively Measure Feature Selection; DFSS: Discriminative Feature Selection; NDM: Normalised Difference Measure; MMR: Max–Min Ratio; EFS: Extensive Feature Selector.

In this method, we combined class-based and corpus-based discrimination. Thus, we have obtained more distinctive features. When we compare our method EFS with other methods, it can be stated that it chooses more distinctive features by calculating both corpus-based and class-based probabilities separately. Of course, some probabilities used while constructing the formula of the proposed method EFS are partially mentioned separately in some of the other methods. However, the proposed method EFS chooses features considering all the probabilities specified in its formula. The related results on the sample collection show how EFS method works. The performance of EFS is tested on different benchmark data sets in section 5.

5. Experimental studies

EFS was compared with the above-mentioned FS techniques in terms of feature similarity and accuracy. To perform this aim, four different data sets and two different success measures were utilised. The data sets and success measures used in the experiments are explained in the following subsections. Then, similarity ratios of top 10 selected features were presented before giving accuracy scores of all FS methods. Stop-word removal [13] and stemming [37] were carried out as pre-processing steps. Also, TF–IDF [1] was used during feature weighting step.

5.1. Data sets

It is inevitable to use data sets with different characteristics to demonstrate the performance of an FS method. So, experiments were carried out using four data sets which are single-labelled. The first data set is Reuters-21578 [38] which is one of the most utilised data sets in TC domain. In the experiments, we utilised the top 10 classes. The second data set is 20-Newsgroup [38] data set and 10 classes of this data set were utilised. The third data set is the Mini 20-Newsgroup [38] data set constructed using a subset of 20-Newsgroup. The final used data set is the Polarity data set containing movie reviews [39]. Further information about these four data sets is given in the Tables 6 –9, respectively.

Table 6.

Reuters-21578 data set.

Number	Class label	Number of documents (training)	Number of documents (testing)
1	earn	2877	1087
2	acq	1650	719
3	money-fx	538	179
4	grain	433	149
5	crude	389	189
6	trade	369	117
7	interest	347	131
8	ship	197	89
9	wheat	212	71
10	corn	181	56

Table 7.

20-Newsgroup data set.

Number	Class label	Number of documents (training)	Number of documents (testing)
1	alt.atheism	700	300
2	comp.graphics	700	300
3	comp.os.ms-windows.misc	700	300
4	comp.sys.ibm.pc.hardware	700	300
5	comp.sys.mac.hardware	700	300
6	comp.windows.x	700	300
7	misc.forsale	700	300
8	rec.autos	700	300
9	rec.motorcycles	700	300
10	rec.sport.baseball	700	300

Table 8.

Mini 20-Newsgroup data set.

Number	Class label	Number of documents (training)	Number of documents (testing)
1	alt.atheism	70	30
2	comp.graphics	70	30
3	comp.os.ms-windows.misc	70	30
4	comp.sys.ibm.pc.hardware	70	30
5	comp.sys.mac.hardware	70	30
6	comp.windows.x	70	30
7	misc.forsale	70	30
8	rec.autos	70	30
9	rec.motorcycles	70	30
10	rec.sport.baseball	70	30

Table 9.

Polarity data set.

Number	Class label	Number of documents (training)	Number of documents (testing)
1	positive	700	300
2	negative	700	300

As shown in Tables 6 –9, while Reuters-21578 is an unbalanced data set, 20-Newsgroup, Mini 20-Newsgroup and Polarity are the balanced data sets. In addition, train and test splits are pre-partitioned for Reuters data set, but 20-Newsgroup, Mini 20-Newsgroup and Polarity data sets are manually divided into train (70%) and test (30%) splits in the experiments.

5.2. Classifiers

TC is object to classifying uncategorised documents into pre-defined classes. TC aims constructing learning models from labelled documents and performs classification on unlabeled text documents using these models. In this study, we used three successful classifiers for TC domain [25]. These are Multinomial Naive Bayes, Support-Vector Machines (SVMs) and k-Nearest Neighbour (KNN) classifiers.

Multinomial Naive Bayes (MNB) is known as one of the most successful classifiers in TC domain. MNB is a special version of Naive Bayes classifier. While classical Naive Bayes models a document with the presence and absence of certain features, MNB [25] explicitly models it using feature counts. In TC studies, multinomial and multivariate Bernoulli event models are widely utilised. As MNB considers term frequencies, multivariate Bernoulli event model utilises document frequencies.

SVM is one of the most efficient classifiers in the TC studies. SVM classifier has two versions which are linear and non-linear [22]. We used linear version of SVM in this study. The important subject of SVM classifier is the margin concept. LIBSVM library [25] is utilised for SVM classifier with linear kernel.

KNN classifier is an efficient and simple learning algorithm [40]. It aims to estimate the class of test documents according to category information of the k-closest neighbours of the training documents in TC domain. The distance which is test documents and its neighbours can be computed by different metrics. Cosine similarity is utilised for finding similarity between test collection in the experimental studies. The value of k is determined from various values. KNN classifier is realised by setting k-value as 7 for all data sets.

5.3. Success measures

In this study, we used two success measures which are macro-F1- and micro-F1-measures [22] for the performance evaluation of FS methods.

The F1-measure takes into account both precision and recall. F1-measure is calculated separately for each class in macro-averaging, and then the average for all classes is computed [25]. In this situation, each class has same weight. Macro-F1 can be computed as follows

Macro - F 1 = \frac{\sum_{j = 1}^{M} F_{j}}{M}, F_{j} = \frac{2 \cdot p_{j} \cdot r_{j}}{p_{j} + r_{j}}

(15)

where $p_{j}$ and $r_{j}$ are the precision and the recall values of class j, respectively.

However, F1-measure is computed without class information in micro-averaging. Thus, all classification decisions are taken into account in text corpus. When evaluating unbalanced data sets, large categories could dominate small ones while computing micro-averaging. Micro-F1 can be computed as follows

Micro - F 1 = \frac{2 \cdot p \cdot r}{p + r}

(16)

where p and r correspond to the precision and the recall values for all classes.

Micro-F1-measure may not provide a fair evaluation for all cases because of dominating large classes. So, macro-F1-measure was used besides micro-F1 in the experiments.

5.4. Feature similarity analysis

Each FS method selects a feature subset according to its formula. Each feature is assigned a score according to discriminative power of it. Profile of feature subsets which are selected by FS algorithms can be tested with their classification performances. If informative features are assigned high scores by any FS method, the performance of classification obtained by those features will be higher. Likewise, if uninformative features are assigned low scores by any FS method, the classification accuracy of feature subset not including those features will probably be higher. On the contrary, if uninformative features are assigned high scores by any FS algorithm, the performance of classification obtained by those features will probably degraded. First, top 10 features selected by the proposed method are compared with the other methods in Tables 10 –13 for each data set. In these tables, if a feature was not chosen by any other FS method at the same time, we would show it in bold. So, it can be stated that the features shown in bold are specific to an individual FS method. We can infer from the tables that the proposed method EFS selects common features selected by other nine FS methods used for comparison in general. One should note that these nine methods used for comparison are mostly state-of-the-art FS methods for TC. Therefore, constructing a feature set including so many distinct features which are very different from the other state-of-the-art FS methods may not make sense for a new method.

Table 10.

Top 10 selected features for Reuters-21578.

	1	2	3	4	5	6	7	8	9	10
CHI2	ct	net	shr	qtr	rev	loss	profit	note	dividend	div
CDM	net	ct	prior	agricultur	England	monetari	qtr	jan	grain	wheat
DPM	net	ct	export	bank	share	wheat	tonn	compani	agricultur	market
OR	net	ct	agricultur	prior	wheat	grain	england	corn	qtr	profit
DFS	ct	wheat	net	oil	shr	tonn	corn	barrel	qtr	agricultur
CMFS	ct	net	shr	march	qtr	mln	rev	year	loss	dlr
DFSS	shr	rev	qtr	barrel	ct	opec	bpd	crude	net	oil
NDM	rev	qtr	taupo	payout	ct	zeebrugg	wheat	bpd	net	opec
MMR	div	rev	qtly	qtr	ct	shr	mth	bbl	net	barrel
EFS	ct	wheat	net	shr	oil	barrel	qtr	march	tonn	trade

CHI2: Chi-Squared; CDM: Class Discriminating Measure; DPM: Discriminative Power Measure; OR: Odds Ratio; DFS: Distinguishing Feature Selector; CMFS: Comprehensively Measure Feature Selection; DFSS: Discriminative Feature Selection; NDM: Normalised Difference Measure; MMR: Max–Min Ratio; EFS: Extensive Feature Selector.

Table 11.

Top 10 selected features for 20-Newsgroup.

	1	2	3	4	5	6	7	8	9	10
CHI2	basebal	forsal	atheism	auto	motorcycl	rec	sport	comp	hardwar	misc
CDM	auto	dod	forsal	team	basebal	rec	sport	ride	window	alt
DPM	comp	rec	misc	window	hardwar	sy	refer	ms	mac	alt
OR	cantaloup	cs	id	newsgroup	path	subject	date	messag	cmu	srv
DFS	atheism	basebal	motorcycl	forsal	auto	sport	os	ms	mac	hardwar
CMFS	cantaloup	cs	id	newsgroup	path	subject	date	messag	cmu	srv
DFSS	atheism	motorcycl	basebal	forsal	auto	religion	atheist	bike	sport	dod
NDM	atheism	kmr	manti	basebal	powerbook	harlei	bobb	automot	schneider	forsal
MMR	atheism	basebal	motorcycl	forsal	pitcher	islam	rec	auto	livesei	solntz
EFS	atheism	basebal	motorcycl	forsal	auto	sport	hardwar	os	sy	rec

Table 12.

Top 10 selected features for Mini 20-Newsgroup.

	1	2	3	4	5	6	7	8	9	10
CHI2	atheism	motorcycl	forsal	basebal	auto	sport	hardwar	sy	rec	comp
CDM	window	ibm	mac	rec	sy	hardwar	game	comp	os	misc
DPM	comp	rec	window	misc	hardwar	sy	write	ibm	refer	pc
OR	cmu	cantaloup	cs	id	newsgroup	line	messag	srv	subject	path
DFS	atheism	basebal	motorcycl	forsal	auto	sport	graphic	alt	os	ms
CMFS	cmu	cantaloup	cs	id	newsgroup	messag	srv	subject	path	date
DFSS	motorcycl	auto	forsal	basebal	bike	sport	moral	graphic	atheist	religion
NDM	atheism	bike	forsal	motorcycl	basebal	christian	fido	asd	atheist	religion
MMR	atheism	motorcycl	forsal	basebal	bike	auto	sport	atheist	religion	christian
EFS	atheism	basebal	forsal	motorcycl	auto	sport	graphic	alt	os	ms

Table 13.

Top 10 selected features for Polarity.

	1	2	3	4	5	6	7	8	9	10
CHI2	bad	life	perform	great	bore	wast	world	script	worst	plot
CDM	magnific	outstand	uninvolv	plod	ludicr	rivet	themat	numb	seagal	tribut
DPM	bad	wast	bore	life	great	perform	worst	stupid	world	script
OR	outstand	magnific	uninvolv	ludicr	plod	rivet	themat	numb	seagal	flawless
DFS	wast	bad	stupid	worst	bore	suppos	outstand	ridicul	great	excel
CMFS	film	movi	make	time	charact	plai	scene	good	stori	end
DFSS	bad	wast	worst	stupid	bore	life	great	outstand	perform	world
NDM	magnific	outstand	uninvolv	plod	ludicr	rivet	themat	numb	seagal	tribut
MMR	outstand	wast	stupid	magnific	worst	ludicr	bore	bad	uninvolv	balanc
EFS	film	movi	make	time	charact	plai	scene	good	stori	end

5.5. Accuracy analysis

Different number of features selected by each FS technique was fed into MNB, SVM and KNN classifiers. Feature subsets were constructed using 10, 30, 50, 100, 300 and 500 features. Total number of features is 16867, 50419, 14244 and 21875 for Reuters-21578, 20-Newsgroup, Mini 20-Newsgroup and Polarity, respectively. Experimental results are shown in Tables 14 –25. While the highest score was written in bold for corresponding feature size, we showed the highest score for the whole table both in bold and underlined. As seen in Tables 14 –25, there is no unique method which is superior for all sizes with different data sets. However, EFS is more successful than all other methods in terms of the highest score in the tables.

Table 14.

Success measures for Reuters-21578 data set using MNB classifier.

	Micro-F1						Macro-F1
Feature size	10	30	50	100	300	500	10	30	50	100	300	500
CHI2	0.553	0.830	0.874	0.912	0.922	0.926	0.074	0.422	0.527	0.634	0.681	0.689
CDM	0.612	0.792	0.831	0.897	0.900	0.905	0.219	0.419	0.487	0.651	0.652	0.655
DPM	0.787	0.867	0.887	0.913	0.926	0.927	0.379	0.586	0.637	0.671	0.698	0.697
OR	0.596	0.798	0.876	0.899	0.907	0.907	0.220	0.433	0.597	0.655	0.669	0.660
DFS	0.646	0.830	0.895	0.917	0.927	0.926	0.296	0.621	0.670	0.689	0.703	0.695
CMFS	0.661	0.786	0.816	0.912	0.923	0.923	0.157	0.312	0.378	0.630	0.680	0.685
DFSS	0.596	0.787	0.853	0.904	0.919	0.926	0.127	0.524	0.631	0.685	0.692	0.698
NDM	0.605	0.654	0.675	0.731	0.880	0.881	0.146	0.338	0.385	0.684	0.620	0.613
MMR	0.590	0.796	0.889	0.907	0.919	0.926	0.122	0.524	0.654	0.688	0.692	0.702
EFS	0.668	0.883	0.901	0.919	0.928	0.927	0.319	0.649	0.673	0.693	0.705	0.696

MNB: Multinomial Naive Bayes; CHI2: Chi-Squared; CDM: Class Discriminating Measure; DPM: Discriminative Power Measure; OR: Odds Ratio; DFS: Distinguishing Feature Selector; CMFS: Comprehensively Measure Feature Selection; DFSS: Discriminative Feature Selection; NDM: Normalised Difference Measure; MMR: Max–Min Ratio; EFS: Extensive Feature Selector.

Table 15.

Success measures for Reuters-21578 data set using SVM classifier.

	Micro-F1						Macro-F1
Feature size	10	30	50	100	300	500	10	30	50	100	300	500
CHI2	0.761	0.843	0.880	0.919	0.925	0.925	0.150	0.396	0.484	0.584	0.646	0.645
CDM	0.802	0.834	0.857	0.907	0.911	0.911	0.282	0.337	0.413	0.589	0.624	0.619
DPM	0.820	0.887	0.903	0.915	0.925	0.926	0.341	0.526	0.583	0.618	0.656	0.665
OR	0.790	0.840	0.886	0.911	0.919	0.914	0.250	0.361	0.513	0.601	0.647	0.634
DFS	0.819	0.903	0.914	0.925	0.923	0.922	0.315	0.587	0.614	0.631	0.655	0.648
CMFS	0.767	0.809	0.832	0.908	0.922	0.926	0.189	0.333	0.375	0.570	0.651	0.668
DFSS	0.794	0.871	0.899	0.916	0.919	0.925	0.230	0.493	0.570	0.614	0.629	0.656
NDM	0.793	0.827	0.836	0.856	0.896	0.898	0.242	0.349	0.375	0.440	0.561	0.582
MMR	0.785	0.860	0.900	0.917	0.923	0.922	0.220	0.458	0.573	0.622	0.643	0.649
EFS	0.836	0.899	0.909	0.919	0.922	0.927	0.377	0.578	0.593	0.623	0.653	0.670

SVM: Support-Vector Machine; CHI2: Chi-Squared; CDM: Class Discriminating Measure; DPM: Discriminative Power Measure; OR: Odds Ratio; DFS: Distinguishing Feature Selector; CMFS: Comprehensively Measure Feature Selection; DFSS: Discriminative Feature Selection; NDM: Normalised Difference Measure; MMR: Max–Min Ratio; EFS: Extensive Feature Selector.

Table 16.

Success measures for Reuters-21578 data set using KNN classifier.

	Micro-F1						Macro-F1
Feature size	10	30	50	100	300	500	10	30	50	100	300	500
CHI2	0.758	0.852	0.867	0.887	0.877	0.847	0.157	0.436	0.466	0.523	0.495	0.460
CDM	0.803	0.841	0.867	0.895	0.881	0.870	0.265	0.384	0.467	0.550	0.508	0.487
DPM	0.822	0.883	0.883	0.889	0.873	0.834	0.357	0.535	0.547	0.554	0.504	0.436
OR	0.793	0.847	0.886	0.899	0.887	0.870	0.253	0.404	0.547	0.554	0.534	0.493
DFS	0.827	0.901	0.900	0.901	0.897	0.878	0.322	0.602	0.591	0.574	0.556	0.497
CMFS	0.756	0.807	0.831	0.878	0.868	0.859	0.222	0.331	0.375	0.494	0.498	0.502
DFSS	0.795	0.870	0.901	0.908	0.898	0.884	0.229	0.507	0.590	0.600	0.558	0.505
NDM	0.790	0.824	0.832	0.849	0.882	0.881	0.239	0.341	0.372	0.434	0.515	0.513
MMR	0.785	0.866	0.899	0.902	0.896	0.886	0.219	0.501	0.590	0.595	0.545	0.519
EFS	0.852	0.889	0.905	0.893	0.879	0.840	0.444	0.536	0.603	0.552	0.518	0.455

KNN: k-Nearest Neighbour; CHI2: Chi-Squared; CDM: Class Discriminating Measure; DPM: Discriminative Power Measure; OR: Odds Ratio; DFS: Distinguishing Feature Selector; CMFS: Comprehensively Measure Feature Selection; DFSS: Discriminative Feature Selection; NDM: Normalised Difference Measure; MMR: Max–Min Ratio; EFS: Extensive Feature Selector.

Table 17.

Success measures for 20-Newsgroup data set using MNB classifier.

	Micro-F1						Macro-F1
Feature size	10	30	50	100	300	500	10	30	50	100	300	500
CHI2	0.881	0.974	0.969	0.957	0.950	0.950	0.733	0.948	0.939	0.917	0.905	0.905
CDM	0.715	0.920	0.959	0.943	0.938	0.930	0.466	0.834	0.920	0.891	0.883	0.867
DPM	0.846	0.971	0.970	0.955	0.939	0.931	0.694	0.944	0.941	0.914	0.885	0.870
OR	0.182	0.961	0.961	0.946	0.939	0.931	0.018	0.923	0.925	0.898	0.883	0.871
DFS	0.876	0.974	0.968	0.951	0.955	0.949	0.727	0.948	0.938	0.903	0.913	0.903
CMFS	0.182	0.972	0.975	0.970	0.947	0.942	0.018	0.946	0.951	0.942	0.898	0.890
DFSS	0.655	0.872	0.944	0.967	0.957	0.954	0.422	0.719	0.882	0.937	0.918	0.912
NDM	0.478	0.663	0.808	0.927	0.971	0.959	0.254	0.430	0.616	0.852	0.943	0.920
MMR	0.656	0.925	0.975	0.973	0.964	0.957	0.423	0.844	0.951	0.948	0.930	0.918
EFS	0.809	0.976	0.975	0.971	0.945	0.944	0.606	0.953	0.951	0.943	0.895	0.893

Table 18.

Success measures for 20-Newsgroup data set using SVM classifier.

	Micro-F1						Macro-F1
Feature size	10	30	50	100	300	500	10	30	50	100	300	500
CHI2	0.879	0.986	0.987	0.984	0.979	0.977	0.735	0.972	0.974	0.971	0.963	0.959
CDM	0.820	0.970	0.986	0.983	0.978	0.977	0.623	0.941	0.973	0.969	0.961	0.960
DPM	0.878	0.986	0.985	0.981	0.976	0.974	0.757	0.973	0.971	0.967	0.958	0.955
OR	0.182	0.983	0.986	0.983	0.978	0.975	0.018	0.966	0.973	0.969	0.960	0.956
DFS	0.935	0.985	0.987	0.985	0.980	0.975	0.857	0.971	0.976	0.974	0.965	0.956
CMFS	0.182	0.985	0.986	0.985	0.978	0.978	0.018	0.970	0.971	0.971	0.956	0.957
DFSS	0.746	0.935	0.977	0.981	0.984	0.981	0.530	0.878	0.956	0.967	0.972	0.967
NDM	0.589	0.757	0.873	0.935	0.987	0.988	0.358	0.555	0.739	0.887	0.977	0.979
MMR	0.749	0.934	0.987	0.985	0.986	0.982	0.535	0.873	0.975	0.974	0.975	0.969
EFS	0.876	0.986	0.986	0.986	0.980	0.979	0.741	0.972	0.971	0.972	0.960	0.958

Table 19.

Success measures for 20-Newsgroup data set using KNN classifier.

	Micro-F1						Macro-F1
Feature size	10	30	50	100	300	500	10	30	50	100	300	500
CHI2	0.882	0.865	0.841	0.834	0.771	0.723	0.766	0.769	0.743	0.721	0.630	0.577
CDM	0.830	0.957	0.949	0.905	0.806	0.752	0.658	0.917	0.902	0.826	0.678	0.607
DPM	0.866	0.951	0.853	0.651	0.485	0.448	0.740	0.905	0.745	0.484	0.327	0.295
OR	0.182	0.857	0.845	0.813	0.759	0.715	0.100	0.756	0.737	0.690	0.620	0.564
DFS	0.926	0.953	0.964	0.931	0.829	0.761	0.859	0.928	0.932	0.872	0.711	0.622
CMFS	0.182	0.801	0.730	0.477	0.472	0.460	0.100	0.672	0.580	0.329	0.319	0.309
DFSS	0.747	0.925	0.955	0.959	0.924	0.872	0.529	0.844	0.932	0.924	0.860	0.776
NDM	0.589	0.755	0.884	0.939	0.957	0.939	0.356	0.551	0.759	0.884	0.918	0.886
MMR	0.751	0.939	0.958	0.959	0.945	0.861	0.541	0.884	0.927	0.929	0.897	0.762
EFS	0.931	0.962	0.806	0.693	0.605	0.577	0.863	0.936	0.683	0.531	0.450	0.424

Table 20.

Success measures for Mini 20-Newsgroup data set using MNB classifier.

	Micro-F1						Macro-F1
Feature size	10	30	50	100	300	500	10	30	50	100	300	500
CHI2	0.810	0.964	0.955	0.953	0.951	0.942	0.607	0.931	0.913	0.910	0.907	0.889
CDM	0.814	0.957	0.940	0.936	0.905	0.897	0.661	0.916	0.886	0.878	0.824	0.811
DPM	0.800	0.955	0.955	0.944	0.921	0.895	0.645	0.914	0.913	0.891	0.853	0.809
OR	0.182	0.944	0.957	0.917	0.905	0.889	0.018	0.891	0.917	0.845	0.825	0.798
DFS	0.802	0.964	0.953	0.953	0.946	0.936	0.612	0.931	0.910	0.910	0.896	0.880
CMFS	0.182	0.932	0.967	0.949	0.934	0.919	0.018	0.848	0.937	0.903	0.876	0.850
DFSS	0.745	0.913	0.955	0.946	0.946	0.927	0.512	0.819	0.914	0.896	0.893	0.862
NDM	0.561	0.667	0.687	0.783	0.960	0.947	0.322	0.443	0.470	0.617	0.923	0.898
MMR	0.661	0.967	0.964	0.960	0.946	0.944	0.422	0.937	0.931	0.924	0.896	0.893
EFS	0.802	0.971	0.967	0.947	0.951	0.932	0.612	0.944	0.937	0.900	0.907	0.873

Table 21.

Success measures for Mini 20-Newsgroup data set using SVM classifier.

	Micro-F1						Macro-F1
Feature size	10	30	50	100	300	500	10	30	50	100	300	500
CHI2	0.817	0.973	0.973	0.973	0.976	0.971	0.653	0.946	0.946	0.946	0.953	0.944
CDM	0.853	0.966	0.971	0.971	0.964	0.953	0.724	0.932	0.942	0.943	0.930	0.910
DPM	0.810	0.971	0.971	0.971	0.955	0.949	0.671	0.943	0.942	0.943	0.913	0.903
OR	0.182	0.955	0.971	0.966	0.960	0.955	0.018	0.912	0.943	0.933	0.924	0.913
DFS	0.889	0.973	0.973	0.969	0.971	0.967	0.744	0.946	0.946	0.939	0.943	0.938
CMFS	0.182	0.934	0.973	0.979	0.973	0.951	0.018	0.877	0.947	0.959	0.946	0.907
DFSS	0.783	0.966	0.965	0.966	0.964	0.955	0.588	0.932	0.933	0.932	0.930	0.913
NDM	0.667	0.753	0.765	0.807	0.981	0.980	0.429	0.551	0.576	0.661	0.963	0.960
MMR	0.745	0.978	0.971	0.973	0.973	0.971	0.529	0.956	0.943	0.946	0.946	0.944
EFS	0.889	0.976	0.971	0.976	0.976	0.958	0.744	0.953	0.943	0.953	0.953	0.920

Table 22.

Success measures for Mini 20-Newsgroup data set using KNN classifier.

	Micro-F1						Macro-F1
Feature size	10	30	50	100	300	500	10	30	50	100	300	500
CHI2	0.824	0.905	0.907	0.885	0.687	0.526	0.693	0.825	0.830	0.795	0.542	0.371
CDM	0.823	0.925	0.887	0.747	0.533	0.485	0.675	0.859	0.798	0.602	0.366	0.320
DPM	0.773	0.895	0.684	0.547	0.465	0.395	0.625	0.809	0.517	0.393	0.328	0.275
OR	0.182	0.855	0.881	0.793	0.585	0.409	0.018	0.735	0.783	0.653	0.415	0.254
DFS	0.885	0.947	0.940	0.917	0.768	0.608	0.748	0.897	0.884	0.845	0.647	0.464
CMFS	0.182	0.821	0.768	0.481	0.408	0.385	0.018	0.672	0.633	0.347	0.267	0.233
DFSS	0.778	0.932	0.932	0.932	0.885	0.812	0.582	0.873	0.872	0.871	0.787	0.687
NDM	0.664	0.747	0.729	0.753	0.872	0.872	0.425	0.547	0.531	0.582	0.771	0.765
MMR	0.747	0.957	0.946	0.938	0.848	0.529	0.529	0.916	0.895	0.883	0.747	0.372
EFS	0.887	0.958	0.855	0.462	0.425	0.404	0.751	0.918	0.748	0.433	0.321	0.298

Table 23.

Success measures for Polarity data set using MNB classifier.

	Micro-F1						Macro-F1
Feature size	10	30	50	100	300	500	10	30	50	100	300	500
CHI2	0.852	0.851	0.875	0.895	0.889	0.885	0.741	0.740	0.778	0.810	0.800	0.794
CDM	0.710	0.726	0.749	0.769	0.824	0.828	0.459	0.522	0.568	0.616	0.699	0.706
DPM	0.843	0.856	0.882	0.882	0.887	0.887	0.728	0.748	0.788	0.788	0.797	0.797
OR	0.707	0.726	0.749	0.762	0.795	0.810	0.457	0.522	0.568	0.588	0.656	0.677
DFS	0.774	0.868	0.882	0.893	0.892	0.876	0.615	0.767	0.788	0.806	0.805	0.780
CMFS	0.712	0.817	0.826	0.852	0.861	0.842	0.553	0.690	0.703	0.728	0.735	0.715
DFSS	0.844	0.862	0.883	0.885	0.893	0.887	0.730	0.757	0.790	0.794	0.807	0.797
NDM	0.710	0.725	0.742	0.783	0.814	0.817	0.459	0.514	0.556	0.634	0.685	0.690
MMR	0.708	0.805	0.875	0.887	0.881	0.884	0.456	0.661	0.779	0.797	0.787	0.792
EFS	0.712	0.820	0.852	0.883	0.893	0.889	0.553	0.695	0.692	0.790	0.808	0.800

Table 24.

Success measures for Polarity data set using SVM classifier.

	Micro-F1						Macro-F1
Feature size	10	30	50	100	300	500	10	30	50	100	300	500
CHI2	0.835	0.853	0.863	0.882	0.874	0.882	0.711	0.743	0.759	0.785	0.767	0.788
CDM	0.711	0.729	0.754	0.755	0.799	0.829	0.464	0.528	0.583	0.607	0.665	0.708
DPM	0.837	0.863	0.869	0.881	0.873	0.885	0.715	0.758	0.769	0.787	0.773	0.793
OR	0.707	0.729	0.754	0.740	0.806	0.813	0.459	0.528	0.583	0.587	0.675	0.685
DFS	0.837	0.867	0.869	0.881	0.867	0.856	0.715	0.764	0.768	0.787	0.765	0.748
CMFS	0.704	0.813	0.819	0.849	0.861	0.872	0.521	0.685	0.693	0.725	0.738	0.761
DFSS	0.834	0.859	0.865	0.883	0.876	0.864	0.709	0.753	0.762	0.790	0.780	0.760
NDM	0.711	0.728	0.747	0.765	0.794	0.815	0.464	0.521	0.571	0.612	0.656	0.688
MMR	0.810	0.858	0.874	0.883	0.872	0.850	0.670	0.750	0.777	0.790	0.773	0.738
EFS	0.704	0.815	0.825	0.869	0.873	0.886	0.521	0.688	0.702	0.769	0.775	0.795

Table 25.

Success measures for Polarity data set using KNN classifier.

	Micro-F1						Macro-F1
Feature size	10	30	50	100	300	500	10	30	50	100	300	500
CHI2	0.821	0.801	0.819	0.827	0.768	0.771	0.697	0.668	0.693	0.705	0.602	0.591
CDM	0.710	0.725	0.738	0.755	0.784	0.824	0.459	0.511	0.547	0.583	0.668	0.693
DPM	0.837	0.801	0.821	0.828	0.763	0.709	0.720	0.668	0.697	0.705	0.582	0.441
OR	0.707	0.725	0.738	0.719	0.792	0.759	0.457	0.511	0.547	0.495	0.634	0.565
DFS	0.832	0.834	0.844	0.819	0.788	0.788	0.712	0.715	0.730	0.693	0.636	0.622
CMFS	0.701	0.703	0.737	0.795	0.788	0.721	0.537	0.539	0.579	0.656	0.642	0.586
DFSS	0.834	0.817	0.851	0.824	0.772	0.740	0.715	0.690	0.740	0.700	0.606	0.527
NDM	0.710	0.712	0.723	0.759	0.763	0.781	0.459	0.487	0.518	0.586	0.585	0.625
MMR	0.808	0.835	0.843	0.846	0.787	0.774	0.668	0.715	0.729	0.733	0.644	0.604
EFS	0.701	0.725	0.788	0.853	0.793	0.775	0.537	0.564	0.689	0.742	0.682	0.520

For Reuters-21578 data set, the highest score was obtained with the EFS FS method in terms of micro-F1 and macro-F1 using MNB, SVM and KNN classifiers. However, EFS method has achieved the highest score only with MNB and KNN classifiers for 20-Newsgroup and Mini 20-Newsgroup data sets. Also, EFS method has achieved the highest score only with SVM and KNN classifiers for Polarity data set.

While the range of micro-F1 scores is between 0.553 and 0.928, the range of macro-F1 scores is between 0.122 and 0.705 for Reuters-21578 data set using MNB classifier. For MNB classifier, the highest the micro-F1 score and macro-F1 score which were obtained with EFS method using 300 features on Reuters-21578 data set are 0.928 and 0.705, respectively. It should be noted that EFS method also performed best for most feature sizes such as 30, 50, 100 and 300.

While the range of micro-F1 scores is between 0.761 and 0.927, the range of macro-F1 scores is between 0.150 and 0.670 for Reuters-21578 data set using SVM classifier. For SVM classifier, the highest the micro-F1 score and macro-F1 score which were obtained with EFS method using 500 features are 0.927 and 0.670 for Reuters-21578 data set, respectively. It should be noted that DFS also performed best for most feature sizes such as 30, 50 and 100. The proposed method EFS obtained the best performances in the other feature sizes such as 10 and 500.

While the range of micro-F1 scores is between 0.756 and 0.905, the range of macro-F1 scores is between 0.157 and 0.603 for Reuters-21578 data set using KNN classifier. For KNN classifier, the highest the micro-F1 score and macro-F1 score which were obtained with EFS method using 50 features are 0.905 and 0.603 for Reuters-21578 data set, respectively. It should be noted that DFS, DFSS and MMR also performed best for different feature sizes such as 30, 100, 300 and 500. Also, it is possible to conclude that MNB classifier is more efficient than SVM and KNN classifiers on Reuters-21578 data set in terms of the highest performances.

While the range of micro-F1 scores is between 0.182 and 0.976, the range of macro-F1 scores is between 0.018 and 0.953 for 20-Newsgroup data set using MNB classifier. For MNB classifier, the highest the micro-F1 score and macro-F1 score which were obtained with EFS method using 30 features on 20-Newsgroup data set are 0.976 and 0.953, respectively. It should be noted that EFS also performed best only with two feature dimensions which are 30 and 50.

While the range of micro-F1 scores is between 0.182 and 0.988, the range of macro-F1 scores is between 0.018 and 0.979 for 20-Newsgroup data set using SVM classifier. For SVM classifier, the highest the micro-F1 score and macro-F1 score which were obtained with NDM method using 500 features are 0.988 and 0.979 on 20-Newsgroup data set, respectively. It should be noted that different methods such as CHI2, DPM, DFS, NDM, MMR and EFS also performed best for different feature sizes.

While the range of micro-F1 scores is between 0.182 and 0.962, the range of macro-F1 scores is between 0.100 and 0.936 for 20-Newsgroup data set using SVM classifier. For KNN classifier, the highest the micro-F1 score and macro-F1 score which were obtained with EFS method using 30 features are 0.962 and 0.936 on 20-Newsgroup data set, respectively. It should be noted that different methods such as DFS, DFSS, NDM, MMR and EFS also performed best for different feature sizes. Also, it is possible to conclude that SVM classifier is more efficient than MNB and KNN classifiers on 20-Newsgroup data set in terms of the highest performances.

While the range of micro-F1 scores is between 0.182 and 0.971, the range of macro-F1 scores is between 0.018 and 0.944 for Mini 20-Newsgroup data set using MNB classifier. For MNB classifier, the highest the micro-F1 score and macro-F1 score which were obtained with EFS method using 30 features are 0.971 and 0.944 on Mini 20-Newsgroup data set, respectively. It should be noted that EFS also performed best only for two feature sizes which are 30 and 50.

While the range of micro-F1 scores is between 0.182 and 0.981, the range of macro-F1 score values is between 0.018 and 0.963 for Mini 20-Newsgroup data set using SVM classifier. For SVM classifier, the highest the micro-F1 score and macro-F1 score which were obtained with NDM method using 300 features are 0.981 and 0.963 on Mini 20-Newsgroup data set, respectively. It should be noted that different methods such as CHI2, DFS, CMFS, NDM, MMR and EFS also performed best for different feature sizes.

While the range of micro-F1 scores is between 0.182 and 0.958, the range of macro-F1 score values is between 0.018 and 0.918 for Mini 20-Newsgroup data set using KNN classifier. For KNN classifier, the highest the micro-F1 score and macro-F1 score which were obtained with EFS method using 30 features are 0.958 and 0.918 on Mini 20-Newsgroup data set, respectively. It should be noted that different methods such as DFSS, NDM, MMR and EFS also performed best for different feature sizes. Also, it is possible to conclude that SVM classifier is more efficient than MNB and KNN classifiers on Mini 20-Newsgroup data set in terms of the highest performances.

While the range of micro-F1 scores is between 0.707 and 0.895, the range of macro-F1 scores is between 0.456 and 0.810 for Polarity data set using MNB classifier. For MNB classifier, the highest the micro-F1 score and macro-F1 score which were obtained with CHI2 method using 100 features are 0.895 and 0.810 on Polarity data set, respectively. It should be noted that EFS also performed best only for two feature sizes which are 300 and 500.

While the range of micro-F1 scores is between 0.704 and 0.886, the range of macro-F1 score values is between 0.464 and 0.795 for Polarity data set using SVM classifier. For SVM classifier, the highest the micro-F1 score and macro-F1 score which were obtained with EFS method using 500 features are 0.886 and 0.795 on Polarity data set, respectively. It should be noted that different methods such as DPM, DFS, DFSS, MMR and EFS also performed best for different feature sizes.

While the range of micro-F1 scores is between 0.701 and 0.853, the range of macro-F1 score values is between 0.441 and 0.742 for Polarity data set using KNN classifier. For KNN classifier, the highest the micro-F1 score and macro-F1 score which were obtained with EFS method using 100 features are 0.853 and 0.742 on Polarity data set, respectively. It should be noted that different methods such as CDM, DPM, DFS, DFSS, MMR and EFS also performed best for different feature sizes. Also, it is possible to conclude that MNB classifier is more efficient than SVM and KNN classifiers on Polarity data set in terms of the highest performances.

5.6. Statistical analysis

To show the validity of the proposed EFS method, t-test is utilised. Table 26 demonstrates the result of p-values from one-tailed, paired t-test.

Table 26.

Statistical significance of EFS method for (a) micro-F1 and (b) macro-F1 success measures with different classifiers.

Classifier (a)	EFS andCHI2	EFS andCDM	EFS andDPM	EFS and MC-OR	EFS andDFS	EFS andCMFS	EFS andDFSS	EFS andNDM	EFS andMMR
MNB	0.3337	0.0001	0.3150	0.0553	0.2336	0.0982	0.0495	0.0086	0.1231
SVM	0.4290	0.0001	0.1830	0.0927	0.1118	0.1352	0.0971	0.0060	0.1714
KNN	0.0763	0.1133	0.0275	0.2984	0.0069	0.0407	0.0386	0.3284	0.0476
Classifier (b)	EFS andCHI2	EFS andCDM	EFS andDPM	EFS and MC-OR	EFS andDFS	EFS andCMFS	EFS andDFSS	EFS andNDM	EFS andMMR
MNB	0.1264	0.0005	0.4105	0.0199	0.1151	0.0452	0.0749	0.0118	0.1502
SVM	0.0259	0.0012	0.2482	0.0552	0.1093	0.0883	0.0702	0.0061	0.1313
KNN	0.2540	0.4451	0.0018	0.1818	0.0053	0.0233	0.0794	0.4109	0.0940

EFS: Extensive Feature Selector; CHI2: Chi-Squared; CDM: Class Discriminating Measure; DPM: Discriminative Power Measure; MC-OR: Multi-Class Odds Ratio; DFS: Distinguishing Feature Selector; CMFS: Comprehensively Measure Feature Selection; DFSS: Discriminative Feature Selection; NDM: Normalised Difference Measure; MMR: Max–Min Ratio; MNB: Multinomial Naive Bayes; SVM: Support-Vector Machine; KNN: k-Nearest Neighbour.

The micro-F1 results showed that the performance gains obtained from the EFS method compared with CDM, MC-OR, CMFS, DFSS and NDM methods are statistically significant with rather high confidence with 90% levels for the MNB classifier. In addition, the performance gains obtained from EFS method compared with CDM, MC-OR, DFSS and NDM methods are statistically significant with rather high confidence with 90% levels for the SVM classifier. Also, the performance gains obtained from EFS method compared with CHI2, DPM, DFS, CMFS, DFSS and MMR methods are statistically significant with rather high confidence with 90% levels for the KNN classifier. The macro-F1 results showed that the performance gains obtained from the EFS method compared with CDM, MC-OR, CMFS, DFSS and NDM methods are statistically significant with rather high confidence with 90% levels for the MNB classifier. In addition, the performance gains obtained from EFS method compared with CHI2, CDM, MC-OR, CMFS, DFSS and NDM methods are statistically significant with rather high confidence with 90% levels for the SVM classifier. Also, the performance gains obtained from EFS method compared with DPM, DFS, CMFS, DFSS and MMR methods are statistically significant with rather high confidence with 90% levels for the KNN classifier. As seen in Table 26, almost all p-values except a few cases are at 90% level. These results confirm the success of the EFS method against other nine FS methods used for comparison.

6. Conclusion and future works

In this study, we proposed a new filter-based feature ranking approach called EFS for TC domain. Besides, we presented a comprehensive analysis of nine state-of-the-art FS methods in the literature. The proposed method EFS selects features according to class-based and corpus-based probabilities which contribute to determining distinctiveness of features. According to the experimental results, there is no method achieving the highest performance for all feature dimensions. However, EFS is more successful than other FS methods in terms of the highest scores on various data sets. The performance of EFS was compared with nine well-known filter-based FS techniques using different data sets, classifiers, success measures and statistical analysis. As a result, EFS is more successful than the others for most of the cases on each data set in terms of highest scores. As a future work, EFS may be applied to other domains in text mining problems.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

ORCID iD

Bekir Parlak

References

Uysal

. An improved global feature selection scheme for text classification. Expert Syst Appl 2016; 43: 82–92.

Agnihotri

Verma

Tripathi

, Variable global feature selection scheme for automatic classification of text documents. Expert Syst Appl 2017; 81: 268–281.

Onan

. An ensemble scheme based on language function analysis and feature engineering for text genre classification. J Inform Sci 2018; 44: 28–47.

Bhowmick

Hazarika

. E-Mail spam filtering: a review of techniques and trends. In: Kalam

Das

Sharma

(eds) Advances in electronics, communication and computing. New York: Springer, 2018, pp. 583–590.

Sjarif

NNA

Azmi

NFM

Chuprat

et al. SMS spam message detection using term frequency–inverse document frequency and random forest algorithm. Proced Comput Sci 2019; 161: 509–515.

Gupta

Sahoo

Roul

. Authorship identification using recurrent neural networks. In: Proceedings of the 2019 3rd international conference on information system and data mining, Houston, TX, 6–8 April 2019, pp. 133–137. New York: ACM.

Chang

Hsieh

Chen

et al. A semantic frame-based intelligent agent for topic detection. Soft Comput 2017; 21: 391–401.

Parlak

Uysal

. On classification of abstracts obtained from medical journals. J Inform Sci 2019; 46: 648–663.

Parlak

Uysal

. Classification of medical documents according to diseases. In: 23rd signal processing and communications applications conference (SIU), Malatya, Turkey, 16–19 May 2015, pp. 1635–1638. New York: IEEE.

10.

Parlak

Uysal

. The impact of feature selection on medical document classification. In: 11th Iberian conference on information systems and technologies (CISTI), Las Palmas, 15–18 June 2016, pp. 1–5. New York: IEEE.

11.

Parlak

Uysal

. On feature weighting and selection for medical document classification. In: Rocha

Reis

(eds) Developments and advances in intelligent systems and applications. New York: Springer, 2018, pp. 269–282.

12.

Onan

. Classifier and feature set ensembles for web page classification. J Inform Sci 2016; 42: 150–165.

13.

Uysal

Gunal

. The impact of preprocessing on text classification. Inform Proces Manag 2014; 50: 104–112.

14.

Forman

. An extensive empirical study of feature selection metrics for text classification. J Mach Learn Res 2003; 3: 1289–1305.

15.

Sebastiani

. Machine learning in automated text categorization. ACM Comput Surv 2002; 34: 1–47.

16.

Guyon

Elisseeff

. An introduction to variable and feature selection. J Mach Learn Res 2003; 3: 1157–1182.

17.

Yang

Pedersen

. A comparative study on feature selection in text categorization. In: ICML ’97: proceedings of the fourteenth international conference on machine learning, Nashville, TN, 8–12 July 1997, pp. 412–420. New York: ACM.

18.

Lee

. Information gain and divergence-based feature selection for machine learning-based text categorization. Inform Proces Manag 2006; 42: 155–165.

19.

Liu

Sun

Liu

et al. Feature selection with dynamic mutual information. Pattern Recogn 2009; 42: 1330–1339.

20.

Shang

Huang

Zhu

et al. A novel feature selection algorithm for text categorization. Expert Syst Appl 2007; 33: 1–5.

21.

Ogura

Amano

Kondo

, Feature selection with a measure of deviations from Poisson in text categorization. Expert Syst Appl 2009; 36: 6826–6832.

22.

Uysal

Gunal

. A novel probabilistic feature selection method for text classification. Knowl Based Syst 2012; 36: 226–235.

23.

Yang

Liu

Zhu

et al. A new feature selection based on comprehensive measurement both in inter-category and intra-category for text categorization. Inform Proces Manag 2012; 48: 741–754.

24.

Zong

Chu

L-K

et al. A discriminative and semantic feature selection method for text categorization. Int J Product Econ 2015; 165: 215–222.

25.

Rehman

Javed

Babri

. Feature selection based on a normalized difference measure for text classification. Inform Proces Manag 2017; 53: 473–489.

26.

Rehman

Javed

Babri

et al. Selection of the most relevant terms based on a max-min ratio metric for text classification. Expert Syst Appl 2018; 114: 78–96.

27.

Rehman

Javed

Babri

et al. Relative discrimination criterion – A novel feature ranking method for text data. Expert Syst Appl 2015; 42: 3670–3681.

28.

Kim

Zzang

. Trigonometric comparison measure: a feature selection method for text categorization. Data Knowl Eng 2019; 119: 1–21.

29.

Pinheiro

Cavalcanti

Correa

et al. A global-ranking local feature selection method for text categorization. Expert Syst Appl 2012; 39: 12851–12857.

30.

Pinheiro

Cavalcanti

Ren

. Data-driven global-ranking local feature selection methods for text categorization. Expert Syst Appl 2015; 42: 1941–1949.

31.

Agnihotri

Verma

Tripathi

et al. Soft voting technique to improve the performance of global filter based feature selection in text corpus. Appl Intel 2019; 49: 1597–1619.

32.

Uysal

Gunal

. Text classification using genetic algorithm oriented latent semantic features. Expert Syst Appl 2014; 41: 5938–5947.

33.

Ghareb

Bakar

Hamdan

. Hybrid feature selection based on enhanced genetic algorithm for text categorization. Expert Syst Appl 2016; 49: 31–47.

34.

Uysal

. On two-stage feature selection methods for text classification. IEEE Access 2018; 6: 43233–43251.

35.

Chen

Huang

Tian

et al. Feature selection for text classification with Naïve Bayes. Expert Syst Appl 2009; 36: 5432–5435.

36.

Chen

C-M

Lee

H-M

Chang

Y-J

. Two novel feature selection approaches for web page classification. Expert Syst Appl 2009; 36: 260–272.

37.

Porter

. An algorithm for suffix stripping. Program 1980; 14: 130–137.

38.

Asuncion

Newman

. UCI machine learning repository, 2007, https://archive.ics.uci.edu/ml/index.php

39.

Parlak

Uysal

. The effects of globalisation techniques on feature selection for text classification. J Inform Sci. Epub ahead of print 18 June 2020. DOI: 10.1177/0165551520930897.

40.

Tan

. An effective refinement strategy for KNN text classifier. Expert Syst Appl 2006; 30: 290–298.