Extraction of protein–protein interactions (PPIs) from the literature by deep convolutional neural networks with various feature embeddings

Abstract

The automatic extraction of protein–protein interactions (PPIs) reported in scientific publications are of great significance for biomedical researchers in that they could efficiently grasp the recent research results about biochemical events and molecular processes for conducting their original studies. This article introduces a deep convolutional neural network (DCNN) equipped with various feature embeddings to battle the limitations of the existing machine learning-based PPI extraction methods. The proposed model learns and optimises word embeddings based on the publicly available word vectors and also exploits position embeddings to identify the locations of the target protein names in sentences. Furthermore, it can employ various linguistic feature embeddings to improve the PPI extraction. The intensive experiments using AIMed data set known as the most difficult collection not only show the superiority of the suggested model but also indicate important implications in optimising the network parameters and hyperparameters.

Keywords

Bio-text mining convolutional neural networks information extraction machine learning protein–protein interaction extraction

1. Introduction

Protein–protein interaction (PPI) extraction aims to determine whether pairs of protein names that appeared in natural language sentences interact with each other or not [1], which is of great significance for biomedical researchers in that they could easily grasp and comprehend the recent research achievements for their further studies. For this reason, numerous attempts have been devoted to improve the performance of the PPI extraction using various methods such as rule-based and statistical machine learning methods [2]. Among them, the statistical machine learning methods are recently regarded as the most promising approaches due to their superiority in performance [2 –7].

We can think of the PPI extraction as an application of binary relation extraction [8,9] in that pairwise relationships are identified in natural language sentences by inspecting their contextual linguistic expressions. However, unlike the relation extraction in general domains, the PPI extraction should deal with various biomedical jargons and specialised conventions for expressing interactions of entity pairs. Table 1 shows the intrinsic distinctions between two kinds of sentences selected from two different collections: AIMed for the PPI extraction [1]¹ and SemEval-2010-T8 [10] for relation extraction. As shown in this table, the sentences from the biomedical domains containing PPIs have many domain-specific expressions (‘Phe93’, ‘Escherichia’, ‘coli-expressed’, ‘peptide ligand’, etc.) that function as important context features when we apply machine learning methods for detecting the PPIs. In this situation, traditional machine learning approaches typically represent the features as one-hot vectors (Bag-Of-Words (BOW)) to train statistical models, which might lead to both ‘vocabulary gap’ and ‘data sparseness’ problems [7]. Moreover, all the protein names are blinded in the collections for fair evaluations without external PPI resources as seen in this table. Thus, we cannot use either the surface forms of the target protein names or their detailed types that could play critical roles in identifying in-between relations, which makes the PPI extraction task more challenging. Both the above-mentioned problems and the state-of-the-art performance [7] suggest that there is still much room for improving the effectiveness of PPI extraction further.

Table 1.

Sentences expressing PPIs and semantic relations between common nouns

Collections	Sentences containing pairwise interactions (relations)
PPI extraction [1]	Substitution of Phe93 with alanine resulted in a dramatic decrease in ENTITYONE binding to the Escherichia coli-expressed extracellular domain of the BIOENTITY ( ENTITYTWO or BIOENTITY) and no detectable binding to full-length mutant receptor expressed in COS cells. Remarkably, Phe93 forms extensive contacts with a peptide ligand in the crystal structure of the ENTITYONE bound to a BIOENTITY (BIOENTITY), suggesting that Phe93 is also important for ENTITYTWO binding. This work shows that single and double Ala substitutions of His18 and Phe21 in BIOENTITY reduced up to 77-fold the binding affinity to ENTITYONE ( ENTITYTWO ) and B (BIOENTITY) and to the BIOENTITY. This study demonstrates that ENTITYONE recognises and activates ENTITYTWO , BIOENTITY and the BIOENTITY by distinct mechanisms.
Relation extraction [10]	Because we made observations, we know the fate of the cat in the box . [Content-Container( box , cat )] Garbage is rapidly becoming the topic of discussion in business meetings, political conventions and even at the dinner table. [Message-Topic( discussion , Garbage )]

Collections

Sentences containing pairwise interactions (relations)

PPI extraction [1]

Substitution of Phe93 with alanine resulted in a dramatic decrease in ENTITYONE binding to the Escherichia coli-expressed extracellular domain of the BIOENTITY ( ENTITYTWO or BIOENTITY) and no detectable binding to full-length mutant receptor expressed in COS cells.
Remarkably, Phe93 forms extensive contacts with a peptide ligand in the crystal structure of the ENTITYONE bound to a BIOENTITY (BIOENTITY), suggesting that Phe93 is also important for ENTITYTWO binding.
This work shows that single and double Ala substitutions of His18 and Phe21 in BIOENTITY reduced up to 77-fold the binding affinity to ENTITYONE ( ENTITYTWO ) and B (BIOENTITY) and to the BIOENTITY.
This study demonstrates that ENTITYONE recognises and activates ENTITYTWO , BIOENTITY and the BIOENTITY by distinct mechanisms.

Relation extraction [10]

Because we made observations, we know the fate of the cat in the box . [Content-Container( box , cat )]
Garbage is rapidly becoming the topic of discussion in business meetings, political conventions and even at the dinner table. [Message-Topic( discussion , Garbage )]

PPI: protein–protein interaction.

In order to alleviate the problems above and achieve better results without much efforts on tedious feature engineering, several recent approaches to relation extraction adopt the deep neural networks [11 –13]. However, even though they emphasise the fact that the most beneficial point of using the deep neural networks is that it is unnecessary to overly rely on handcrafted features, they actually utilise them by multiple embedding mechanisms [14] to come up with the state-of-the-art results. Not surprisingly, therefore, it seems quite reasonable to apply the additional features to conventional deep neural networks for further enhancing the performance of relation extraction and the PPI extraction.

This article introduces a deep convolutional neural network (DCNN) equipped with various feature embeddings to overcome the limitations of the existing machine learning-based PPI extraction methods. The proposed model learns and adjusts word embeddings based on the publicly available word vectors and also exploits position embeddings to pinpoint the positions of the target protein names in the sentences. Furthermore, it can employ additional linguistic feature embeddings to improve the performance of PPI extraction. An intensive experiment of this article using AIMed [1] which is known as the most difficult collection shows not only the superiority of the proposed model but also some practical insights when optimising hyperparameters for the model.

2. Related works

We can classify the existing approaches to PPI extraction into three categories [2]: linguistic models [15 –17], rule-based approaches [18,19] and statistical machine learning methods [2 –7]. Recently, with excellent statistical models invented as well as vast amount of data sets constructed and publicly available, supervised machine learning models have become attractive to many researchers for fairly boosting the performance of PPI extraction.

As one of the early attempts to extract interactions and relations between biomedical entities such as enzymes, genes and proteins from scientific journal articles, Humphreys et al. [20] utilised their information extraction system developed to participate in the Defense Advanced Research Projects Agency (DARPA) Message Understanding Conferences (MUCs) for two biomedical text mining applications where information about enzymes, metabolic pathways and protein structures was extracted from biomedical literature. Airola et al. [3] proposed a graph–kernel model by transforming the original dependency tree into directed graphs to alleviate the existing parse tree kernel methods. In particular, they introduced a new evaluation method based on cross-corpus learning using five PPI corpora. Miwa et al. [4] introduced composite kernels employing parse trees and dependency graphs as well as flat lexical features to extract PPIs and showed competitive results compared with other methods. Choi and Myaeng [21] indicated the inefficiency of the conventional complicated kernel models and achieved better performance using only single parse tree kernel by optimising parameters such as tree kernel decay factors. Tikk et al. [5] conducted a series of comprehensive experiments to compare nine different convolution kernel methods with rich linguistic information for PPI extraction. Their results showed that kernels using dependency trees outperform kernels based on full parse trees. More recently, Chang et al. [22] introduced a kernel for interaction patterns in which they capture frequent PPI patterns and integrate them with convolution tree kernels to extract PPIs. In order to overcome the existing limitations of using BOW features (one-hot features) when applying machine learning models such as support vector machines (SVMs), Li et al. [7] introduced a composite kernel model using distributional word representation (word embeddings), lexical features and parse tree features which achieves the state-of-the-art results (69.7 F-score in AIMed).

Zeng et al. [12] exploited a DCNN to extract lexical and sentence-level features and applied it into extracting relations between nominals. Their experimental results using SemEval-2010 showed that the approach significantly outperforms the state-of-the-art methods. Also, Dos Santos et al. [13] introduced classification by ranking convolutional neural network (CR-CNN) that uses a convolutional layer to produce a distributed vector representation of the text and compared it with the class representations in order to produce a score for each class. Their approach achieved slightly higher accuracy on the same data set than Zeng et al. [12] with word and position embeddings. Accordingly, this article is the first attempt to fully exploit the characteristics of the convolutional neural networks with various embeddings for the PPI extraction.

3. DCNNs for PPI extraction

The proposed model shown in Figure 1 is essentially based on a sentence-based convolutional neural network model proposed by Collobert et al. [23], where sentences with variable lengths are converted into k-dimensional global feature vectors by applying convolution approach and multiple feature embeddings. First, each word of the input sentence is transformed into a real-valued vector by looking up the word vector table generated by Collobert et al. [23]. As the purpose of the model is to predict whether there is an interaction between a pair of protein names, it is necessary for the model to know which input words are actually target protein names in a sentence. For doing this, the network adopts position features that are relative distances from the two target proteins [12,13]. Both the word feature vectors and position feature vectors are then concatenated and fed into convolution layer to generate a sentence feature vector of the input sentence.

Meanwhile, the model also extracts entity, lexical, syntactic and semantic features from the input sentence and forms a linguistic feature vector. Finally, a global feature vector is created by merging the two feature vectors above and then fed into a sigmoidal classifier to compute a confidence value between 0 and 1 representing whether two proteins interact or not. The following sections describe the process in more detail.

3.1. Sentence feature embeddings

In order to generate a sentence feature vector, the network first executes the embedding process against each word in a sentence S using a word vector look-up table [23] to make a word embedding matrix E of S with T words such that

E = [w_{s}, w_{1}, w_{2}, \dots, w_{T}, w_{e}]

(1)

where $w_{i} \in R^{d}$ is an embedding vector of a word $w_{i}$ and both $w_{s}$ and $w_{e}$ are zero-initialised padding vectors which denote the beginning and end of a sentence, respectively. Although the external look-up table is applied to initialise the embedding matrix E in the network at the beginning, the values of $w_{s}$ , $w_{e}$ and $w_{i}$ become adapted during the training stage so that a corresponding loss function (section 4.2.) is minimised. Furthermore, to let the network know which words are the target protein names to be examined, the model computes the relative position indices from them for each of the word. For instance, for the sentence in Figure 1 including the padding symbols, Table 2 shows the two sets of the relative position indices.

Table 2.

Relative position indices of the sentence in Figure 1 (‘Isolation of human PROT1 and its binding specificity with PROT2’)

Words	Relative positions from PROT1	Relative positions from PROT2
Padding (s)	−4	−10
Isolation	−3	−9
of	−2	−8
human	−1	−7
PROT1	0	−6
and	1	−5
its	2	−4
binding	3	−3
specificity	4	−2
with	5	−1
PROT2	6	0
Padding (e)	7	1

Each of these relative distance values is embedded as a randomly initialised numerical vector and concatenated with the correspondent word embedding vector $w_{i}$ . As there are two target proteins in a sentence, each word $w_{i}$ has two position embedding vectors $p_{i}^{1} \in R^{m}$ and $p_{i}^{2} \in R^{n}$ , where m and n are the hyperparameters to determine the dimensions of the position vectors which are set to the numbers of the distinct position values extracted from the entire training set. Therefore, a column vector $x_{i} \in R^{d + m + n}$ of the final embedding matrix becomes

x_{i} = {(w_{i}, p_{i}^{1}, p_{i}^{2})}^{T} = {(w_{i 1}, \dots, w_{id}, p_{i 1}^{1}, \dots, p_{im}^{1}, p_{i 1}^{2}, \dots, p_{in}^{2})}^{T}

(2)

where $w_{ij}$ is the jth element of the word embedding vector of $w_{i}$ and $p_{ik}^{1}$ and $p_{ik}^{2}$ are the kth elements of the position embedding vectors $p_{i}^{1}$ and $p_{i}^{2}$ related to the first and second protein names in a sentence, respectively.

In order to generate a fixed length sentence feature vector, the model applies the convolution approach to the embedding matrix $X = [x_{1}, x_{2}, \dots, x_{S}] \in R^{S \times (d + m + n)}$ established as above. Given a column vector $x_{t}$ , a matrix-vector operation is applied to its contextual vectors $(x_{t - w_{d} / 2}, \dots, x_{t}, \dots, x_{t + w_{d} / 2})^{T}$ , whose dimensions are $w_{d} \cdot (d + m + n)$ due to the concatenation of $w_{d}$ successive column vectors centralised in $x_{t}$ such that

c_{t} = g_{1} (W^{1} \cdot {(x_{t - w_{d} / 2}, \dots, x_{t}, \dots, x_{t + w_{d} / 2})}^{T})

(3)

where $w_{d}$ is the window (context) size of the convolution layer, $W^{1} \in R^{(w_{d} \cdot (d + m + n)) \times h_{1}}$ is a weight matrix to be trained and $g_{1}$ is a nonlinear activation based on rectified linear unit (ReLU) [24] function. The dimension of $c_{t}$ is a hyperparameter ( $h_{1}$ ) determining the dimension of the final sentence feature vector. The resulting convolved matrix $C = [c_{1}, c_{2}, \dots, c_{T}]$ is used to extract the h₁-dimensional fixed-sized distributed vector representation $s = (s_{1}, s_{2}, \dots, s_{h_{1}})$ for the sentence by applying max-over-time operation to each column vectors $c_{t}$ as follows

s = (\max (c_{11}, \dots, c_{1 T}), \max (c_{21}, \dots, c_{2 T}), \dots, \max (c_{h_{1} 1}, \dots, c_{h_{1} T}))

(4)

Finally, we obtain a single sentence feature vector $s = (s_{1}, s_{2}, \dots, s_{h_{1}})$ which is independent of the length of the sentence and embraces both the word and target position embeddings along with their contextual implications.

3.2. Linguistic feature embeddings

Several recent studies in relation extraction show that the above sentence feature representations alone could lead to the state-of-the-art accuracy without any complex feature engineering. The more exhaustive analysis of their experimental results, however, indicates that their increased scores are not significantly superior than the previous feature-intensive approaches [11 –13]. They are just satisfied with having achieved the competitive performance without conducting the tedious feature selection and extraction process while there is no reason to settle for the status quo if we could obtain far more superior results even with the labour-intensive process.

This article presumes that the endeavour for the ‘meaningful features excavation’ especially for relation extraction is almost on the verge of completion by numerous research activities so far [25]. In this respect, a variety of the linguistic features which are proven to be useful are exploited to further improve the deep learning-based PPI extraction using the embedding mechanism used in the previous section. Table 3 shows the details of the five types (entity, lexical, dependency, parse and semantic features) of linguistic features used in this model. Each feature type has multiple linguistic features that could provide pivotal clues to identify interactions in sentences. As you can see in Table 3, lexical features include the contextual words near the target entities and their bigrams as well. Dependency features comprise the shortest dependency paths of part-of-speech tags, words and dependency types between the two entities. Also, this article uses the syntactic structures encompassing the two entities at hand without their terminal nodes (word nodes) as parse tree features. Semantic features are generated by converting all the words in the lexical features into their first parent synset identifiers of WordNet [26] using JWI library². Finally, entity features include entity mention strings, entity types and their bigrams. Figure 2 shows an example of the extracted features from a sentence.

Table 3.

Five types of linguistic features extracted for feature embeddings

Types	Feature Name	Details
Lexical features	WORDS-BETWEEN-NONE	Whether there are words in between two entities (0/1)
	WORDS-BETWEEN-ONE	The word if it is the only word in between two entities
	WORDS-BETWEEN-FIRST	The first word if there exist two or more words in between two entities
	WORDS-BETWEEN-LAST	The last word if there exist two or more words in between two entities
	WORDS-BEFORE-FIRST	The word just before the first entity
	WORDS-BEFORE-SECOND	The word just before WORDS-BEFORE-FIRST
	WORDS-AFTER-FIRST	The word just after the second entity
	WORDS-AFTER-SECOND	The word just after WORDS-AFTER-FIRST
	WORDS-BETWEEN-BOWS	Bag-Of-Words (BOWs) of all the words in between the two entities
	WORDS-BETWEEN-BIGRAM	Word Bigram in between the two entities
Dependency features	DEPS-TAG-PATH	POS tag sequence in the shortest dependency path between the two entities
	DEPS-LEMMA-PATH	Word sequence in the shortest dependency path between the two entities
	DEPS-TYPE-PATH	Dependency type sequence in the shortest dependency path between the two entities
Parse tree features	PATH-ENCLOSED-TREE	Syntactic tree between the two entities except terminal nodes
Semantic features	HYPER-BETWEEN-ONE	Parent synset of the word if it is the only word in between two entities
	HYPER-BETWEEN-FIRST	Parent synset of the first word if there exist two or more words in between two entities
	HYPER-BETWEEN-LAST	Parent synset of the last word if there exist two or more words in between two entities
	HYPER-BEFORE-FIRST	Parent synset of the word just before the first entity
	HYPER-BEFORE-SECOND	Parent synset of the word just before WORDS-BEFORE-FIRST
	HYPER-AFTER-FIRST	Parent synset of the word just after the second entity
	HYPER-AFTER-SECOND	Parent synset of the word just after WORDS-AFTER-FIRST
	HYPER-BETWEEN-BOWS	Bag-Of-Synsets (BOSs) of all the words in between the two entities
	HYPER-BETWEEN-BIGRAM	Parent synset bigrams in between the two entities
Entity features (entity)	ENTITY-WORD1	First entity mention string
	ENTITY-WORD2	Second entity mention string
	ENTITY-TYPE1	The type of the first entity
	ENTITY-TYPE2	The type of the second entity
	ENTITY-WORD-PAIR	Mention string bigram of the two entities
	ENTITY-TYPE-PAIR	Mention type bigram of the two entities

Note that each feature item is composed of its name and value separated by two consecutive underbars (‘__’) as in ‘WORDS-BETWEEN-FIRST__treatment’. Given $| F |$ feature items extracted from a training set, we can express a sentence by a $| F | - dimensional$ one-hot vector, $m = (m_{1}, m_{2}, \dots, m_{| F |})$ , where $m_{i}$ is 1 if the sentence has ith feature item and otherwise 0. As mentioned earlier, because one-hot vector mechanism could cause data sparseness problems, the proposed model encodes (embeds) the vectors into h₂-dimensional feature vector such that

f = g_{2} (W^{2} \cdot {(m_{1}, \dots, m_{i}, \dots, m_{| F |})}^{T}) = \frac{1}{1 + e^{- (W^{2} \cdot m^{T})}}

(5)

where $W^{2} \in R^{| F | \times h_{2}}$ is a weight matrix used for encoding the one-hot vector and $g_{2}$ is a sigmoidal activation function which is applied to all the elements of the vector $W^{2} \cdot m^{T}$ . The output vector $f = (f_{1}, f_{2}, \dots, f_{h_{2}})$ becomes the final linguistic feature vector.

In the final step, we produce a global feature vector, $g_{f} = (s, f) \in R^{(h_{1} + h_{2})}$ by concatenating the two separately generated feature embedding vectors $s \in R^{h_{1}}$ and $f \in R^{h_{2}}$ and feeding it into a simple sigmoidal classifier

o = \frac{1}{1 + e^{- (W^{3} \cdot g_{f}^{T})}}

(6)

where $W^{3} \in R^{(h_{1} + h_{2}) \times 1}$ is a weight matrix to produce a scalar output (o) denoting the probability that the two target proteins are interacted in a sentence and a sigmoidal activation function is used for binary classification.

4. Experiments

4.1. Data set and pre-processing

The proposed model is evaluated using AIMed [1] which is known to be the most difficult but widely used data set for PPI extraction evaluation. In Pyysalo et al. [27], the data set has been normalised and anonymised, which means that all the target protein names were blinded so that the reference to external PPI databases is impossible for pair evaluation. Therefore, we cannot exploit any entity feature information (e.g. entity mentions and entity types) which is critical for achieving the state-of-the-art performance especially in relation extraction. The experiments used this transformed version of the data for the fair comparison with other approaches. In order to extract various linguistic features, we used Stanford CoreNLP 3.6 [28]³ for executing the part-of-speech tagging and syntactic analysis. Table 4 shows the statistics of AIMed data set used in the experiments.

Table 4.

Statistics of AIMed data set

	No. of positive instances	No. of negative instances	Total	No. of nested instances
Original data [27]	991	4784	5775	162
Pre-processed data in the experiment	991	4568	5559	162

As seen in Table 4, the pre-processing task discards 216 negative instances. This is because there exist multi-sentence instances in the data set and their two target proteins appear in different sentences. As the model aims to extract PPIs from a single sentence, the experiments ignore all the multi-sentence instances. However, many previous studies fail to explicitly identify and reveal this situation in their reports. Figure 3 shows an example of the multi-sentence instances.

Another reason that we discard these instances is the fact that all of them are negative instances, and it is trivial to judge them as negative using simple heuristics, which might lead to unfair improvement in performance. Furthermore, we found 162 instances that have so-called self-interacted (nested or overlapped) protein pairs in the data set. For this case, we define four special keywords: ‘TWO_HAS_ONE’ (the second protein contains the first), ‘ONE_HAS_TWO’ (the first protein has the second), ‘TWO_OVER_ONE’ (the second protein has partially the first one) and ‘ONE_OVER_TWO’ (the first protein has partially the second one). Also, the corresponding target protein pairs are converted into one of the above four keywords as in ‘Soluble TWO_HAS_ONE and BIOENTITY in toxic epidermal necrolysis: a comparative analysis of serum and blister fluid samples’. In particular, all the self-interacted instances of AIMed belong to the case where the second protein names include the first ones (‘TWO_HAS_ONE’).

4.2. Training and hyperparameters

The DCNN model introduced in this article was implemented on Keras 1.0 [29]⁴ written in Python. It provides highly modular neural networks library and can execute on top of either TensorFlow⁵ or Theano [30]. This article chooses Theano as a backend machine learning library. The network parameters of the proposed DCNN model can be denoted as a quadruple $θ = (X, W^{1}, W^{2}, W^{3})$ . To estimate $θ$ , the model utilises the binary cross-entropy loss with all the N training examples $(x_{n}, y_{n})$ such that

L (θ) = - \frac{1}{N} \sum_{n = 1}^{N} [y_{n} \log {\hat{y}}_{n} + (1 - y_{n}) \log (1 - {\hat{y}}_{n})]

(7)

where ${\hat{y}}_{n} = g (x_{n}; θ) = o (x_{n}; θ)$ with $g (z)$ is the entire network function and $y_{n}$ is the target value (0 or 1). To prevent the overfitting problems, dropout [31] is applied to the word and position embedding layers, convolution layer and feature embedding layer with the ratio of 0.2. The training process uses RMSprop [32] optimisation technique with learning rate of 0.001. In addition, for efficiency in training, the experiments use the early stopping criterion [33] with the loss value by computing (7) (patience = 5) up to maximally 50 epochs.

With no official development data set available, we experimentally inspect the effects of the hyperparameters of the proposed model by conducting 10-fold cross validation: the window size of the convolution layer ( $w_{d}$ ), the output dimension of the convolution layer (h₁) and the output dimension of the feature embedding layer (h₂). The dimension of the word embedding vector (d) is the same as in Collobert et al. [23]. Table 5 presents the hyperparameters with the linguistic features showing the best macro-averaged F-score in the experiments. In this table, the entries in grey cells indicate the default (static) hyperparameters derived from the heuristic process prior to the main experiments. For instance, we set the dropout ratio to 0.2 which shows the best overall performance in the repeated 10-fold cross validations.

Table 5.

Optimal hyperparameters found in the experiments

Hyperparameters	Values
Window size for convolution	$w_{d} = 5$
Word embedding vector dimension	$d = 50$
Output dimension of convolution layer	$h_{1} = 100$
Output dimension of feature embedding layer	$h_{2} = 100$
Learning rate (RMSprop)	$λ = 0.001$
Dropout ratio	$r_{d} = 0.2$
Early stopping patience	$p = 5$
Linguistic features used	Lexical/dependency/parse/semantic

Although we observed that the above hyperparameter setting shows the best performance among others, the intensive analysis of the change in F-scores by various parameter combinations reveals some interesting and distinctive empirical aspects of our proposed model, which will be explained in the following section.

4.3. Performance comparison with other approaches

In order to compare our approach with the others, we select four previous approaches as seen in Table 6, all of which employed SVM with kernel methods to exploit various features and the word embeddings to improve the performance of PPI extraction. Because it was quite obvious that approaches using other machine learning models such as conditional random fields (CRF) and maximum entropy (ME) have shown much poor performance, we excluded them in the performance comparison. As seen in this table, it is quite obvious that the proposed DCNNs overwhelmingly outperform the other previous methods. The results indicate that the extremely beneficial effects of the convolutional approach on the relation extraction [11,12,37] are quite of service to the PPI extraction. Due to the remarkable and unexceptional increase in performance, we carefully checked our evaluation schemes⁶ and conducted the experiments repeatedly to be assured with the results. Note that in 10-fold cross validation, we do not share all the feature embeddings except word embeddings between the nine training folds and one test fold while many deep learning approaches to NLP tend to share the extracted feature embeddings as in Chiu and Nichols [38].

Table 6.

Performance comparison with other systems.

Systems	Methods and features	F1
Miwa et al. [4]	Multiple kernels with multiple parsers (SVM)	60.8
Choi and Myaeng [21]	Single optimised and tuned tree kernels (SVM)	67.0
Sætre et al. [34]	Composite kernels with BOW and dependency path (SVM)	64.2
Bui et al. [35]	Feature-based hybrid approaches (SVM)	61.2
Yang et al. [36]	Multiple kernels with intensive linguistic features (SVM)	64.4
Qian and Zhou [6]	Single tree kernel with multiple parsers (SVM)	58.1
Li et al. [7]	Multiple kernels with word embeddings (SVM)	69.7
Proposed DCNN	DCNN with word/position embeddings	83.6
	+ Lexical feature embeddings	79.4
	+ Dependency path feature embeddings	83.0
	+ Parse tree feature embeddings	81.7
	+ Semantic (WordNet) feature embeddings	85.2

SVM: support vector machine; BOW: Bag-Of-Words; DCNN: deep convolutional neural network.

Another interesting but rather unsatisfactory phenomenon is that appending feature embeddings do not lead to gradual increase in performance, whereas applying all the linguistic feature embeddings produces the best F-score. For example, applying the lexical features to the basic DCNN with the word and position embeddings lowers the score by 4.2 points, which means that the lexical embeddings alone impede the PPI extraction due to the data sparseness problem with lack of the training data. As the previous studies showed, the word/position embeddings seem to play a pivotal role also in the task. It is the same for the cases of using dependency path and parse tree feature embeddings. This could be explained by the fact that all of the feature embeddings are in complementary relations for better performance, while the partial utilisations of them could become harmful due to data sparseness problem mentioned earlier even though exploiting the embedding mechanisms. Nevertheless, further empirical investigations are needed to explain it, and the following section attempts to obtain the bottom of this seemingly unstable result.

4.4. Detailed analysis of performance fluctuation

In this section, we investigate the variation of the macro-averaged F-scores according to the variety of hyperparameters. Figure 4 shows the effects of the hyperparameters that achieved the best performance in Table 6.

As seen in Figure 4, among the hyperparameters, the window size seems to have limited impact on the performance, whereas the dimensions (units) of the convolution layer (h₁) and feature embedding layer (h₂) cause some fluctuations in the F-scores. For instance, while we achieve the best score (85.2) with $(h_{1} = 100, h_{2} = 100)$ , the score sharply decreases to 78.3 when we apply $(h_{1} = 300, h_{2} = 100)$ . In addition, the score increases again when we use $(h_{1} = 400, h_{2} = 100)$ . These fluctuations in performance are worthwhile to thoroughly examine for better understanding the behaviour of the networks. Therefore, in order to investigate these situations in more detail, we obtain the F-scores of all the combinations of the two hyperparameters (the dimensions of the convolution layer and feature embedding layer) as presented in Figure 5.

In Figure 5, the horizontal axis presents all the combinations of $h_{1}$ and $h_{2}$ from 100 to 500 by the hundred. For example, the axis number 1 denotes the combination of $(h_{1} = 100, h_{2} = 100)$ , 2 means $(h_{1} = 100, h_{2} = 200)$ and 25 does $(h_{1} = 500, h_{2} = 500)$ . As seen in this figure, the fluctuation with $w_{d} = 5$ is the most severe and unstable even though it has the highest score showing also the lowest score (68.4) with $(h_{1} = 200, h_{2} = 500)$ . However, we can note that the variation in performance with $w_{d} = 7$ is rather small and thus stable. Furthermore, the overall average of F-scores is the highest when we use $w_{d} = 7$ according to the exponential trend analysis represented by the dotted lines in this figure. From the observation, we could carefully conclude that it would be beneficial to exploit more contexts in convolutional approaches for achieving more stabilised performance with respect to the two hyperparameters.

The same interpretation as in Figure 5 is successfully applied to Figures 6 –8. However, the resulting aspects in Figure 8 probably need somewhat different explanations because we can note more fluctuations and lower average scores in the case of $w_{d} = 7$ . The use of the entire parse trees (only one in a single instance) could cause data sparseness problems and, therefore, influence the overall variation in performance. Therefore, it is valuable to look into the entire performance trajectory in accordance with the hyperparameters in order to catch the behaviour and thus characteristics of the deep neural networks for the PPI extraction.

It took about 4 weeks to execute the entire experiments and empirical analysis using Linux server with two quad-core central processing units (CPUs) and one graphics processing unit (GPU; GTX 980 Ti), which inhibits from conducting more intensive empirical evaluations with much more hyperparameters. Nevertheless, the experimental results provided us with several useful implications with regard to the PPI extraction using DCNN as follows:

We could achieve the remarkably superior performance (85.2 F-score) by means of DCNNs with word, position and all the linguistic feature embeddings introduced in this article.

Also, it seems that only when the entire linguistic feature embeddings are applied to the networks, the best F-scores could be achieved in the evaluations.

It is worthwhile to investigate the overall fluctuation in performance according to the hyperparameters such as the output dimensions of the convolution layer and the feature embedding layer for better understanding the behaviour of the DCNN for the PPI extraction.

The pinpointed optimal hyperparameters of the networks are not always the appropriate answers for achieving the stable performance due to the fact that the training schemes of the deep neural networks are inherently stochastic and, therefore, occasionally lead to many different optimal network parameters.

We noted that the proposed model suffers from the sparseness of additional linguistic features such as dependency, parse tree and semantic features showing rather severe fluctuations as seen in Figures 5, 7 and 8. In other words, except for the lexical features, all the additional linguistic features are not properly embedded in the model due to the lack of the training data (note that we only used the training data for the feature embedding).

Accordingly, the experimental results showed that it is necessary to come up with more concrete and detailed evaluation policies to fairly evaluate the deep neural network models for the PPI extraction.

5. Conclusion and future works

This article introduces a DCNN equipped with various feature embeddings to improve the existing machine learning-based PPI extraction methods. The proposed model learns and optimises word embeddings based on the publicly available word vectors and also exploits position embeddings to identify target protein names in sentences. Furthermore, it can employ additional linguistic feature embeddings to further improve the performance of PPI extraction. The intensive experiments using AIMed [1] which is known as the most tricky collection showed not only the remarkable superiority of the proposed model but also some practical and useful insights when training and optimising hyperparameters for the model.

Future works should involve the complete multi-corpora evaluation for PPI extraction as in Qian and Zhou [6], Li et al. [7], Pyysalo et al. [27] and other previous studies. In addition, it is necessary to conduct more concentrative evaluations by applying various options, hyperparameters and even different topologies of the deep neural networks such as recurrent neural networks (RNNs). Finally, we plan to build and apply the linguistic feature embedding vectors using large-scaled external language resources to improve the performance of the PPI extraction.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

This work was supported by Kyonggi University Research Grant 2014 (South Korea).

Notes

References

Bunescu

Kate

et al . Comparative experiments on learning information extractors for proteins and their interactions. Artif Intell Med 2005; 33: 139–155.

Zhou

Extracting interactions between proteins from the literature. J Biomed Inform 2008; 41: 393–407.

Airola

Pyysalo

Björne

et al . All-paths graph kernel for protein-protein interaction extraction with evaluation of cross-corpus learning. BMC Bioinformatics 2008; 9: 1–12.

Miwa

Sætre

Miyao

et al . Protein–protein interaction extraction by leveraging multiple kernels and parsers. Int J Med Inform 2009; 78: e39–e46.

Tikk

Thomas

Palaga

et al . A comprehensive benchmark of kernel methods to extract protein–protein interactions from literature. PLoS Comput Biol 2010; 6: e1000837.

Qian

Zhou

Tree kernel-based protein–protein interaction extraction from biomedical literature. J Biomed Inform 2012; 45: 535–543.

Guo

Jiang

et al . An approach to improve kernel-based protein–protein interaction extraction by learning from large-scale network data. Methods 2015; 83: 44–50.

Zhang

Zhou

Exploring syntactic structured features over parse trees for relation extraction using kernel methods. Inform Process Manag 2008; 44: 687–701.

Zhou

Zhong

Biomedical relation extraction: from binary to complex. Comput Math Method M 2014; 2014: 298473.

10.

Hendrickx

Kim

Kozareva

et al . SemEval-2010 Task 8: multi-way classification of semantic relations between pairs of nominals. In: Proceedings of the 5th international workshop on semantic evaluation, Stroudsburg, PA, pp. 33–38, Association for Computational Linguistics, http://www.aclweb.org/anthology/W09-2415

11.

Liu

Sun

Chao

et al . Convolution neural network for relation extraction. In: Motoda

Cao

et al . (eds) Advanced data mining and applications. Berlin, Heidelberg: Springer, 2013, pp. 231–242.

12.

Zeng

Liu

Lai

et al . Relation classification via convolutional deep neural network. In: Proceedings of COLING, pp. 2335–2344, http://www.aclweb.org/anthology/C14-1220

13.

Dos Santos

Xiang

Zhou

Classifying relations by ranking with convolutional neural networks (arXiv preprint), arXiv:1504.06580 (2015, accessed 17 May 2015).

14.

Plank

Moschitti

. Embedding semantic similarity in tree kernels for domain adaptation of relation extraction. In: Proceedings of the 51st annual meeting of the association for computational linguistics (ACL 2013), Sofia, 4–9 August 2013, volume 1: long papers, pp. 1498–1507. The Association for Computer Linguistics.

15.

Sekimizu

Park

Tsujii

Identifying the interaction between genes and gene products based on frequently seen verbs in Medline abstracts. Genome Inform Ser Workshop Genome Inform 1998; 9: 62–71.

16.

Temkin

Gilder

MR.

Extraction of protein interaction information from unstructured text using a context-free grammar. Bioinformatics 2003; 19: 2046–2053.

17.

Papanikolaou

Pavlopoulos

Theodosiou

et al . Protein–protein interaction predictions using text mining methods. Methods 2015; 74: 47–53.

18.

Ono

Hishigaki

Tanigami

et al . Automated extraction of information on protein-protein interactions from the biological literature. Bioinformatics 2001; 17: 155–161.

19.

Blaschke

Hirschman

Valencia

Information extraction in molecular biology. Brief Bioinform 2002; 3: 154–165.

20.

Humphreys

Demetriou

Gaizauskas

Bioinformatics applications of information extraction from scientific journal articles. J Inf Sci 2000; 26: 75–85.

21.

Choi

S-P

Myaeng

S-H

. Simplicity is better: revisiting single kernel PPI extraction. In: Proceedings of the 23rd international conference on computational linguistics, Stroudsburg, PA, pp. 206–214, Association for Computational Linguistics, http://www.aclweb.org/anthology/C10-1024

22.

Chang

Y-C

Chang

N-W

et al . An interaction pattern kernel approach for Protein-Protein Interaction extraction from biomedical literature. In: Cheng

S-M

Day

M-Y

(eds) Technologies and applications of artificial intelligence. Cham: Springer International Publishing, 2014, pp. 36–46.

23.

Collobert

Weston

Bottou

et al . Natural language processing (almost) from scratch. J Mach Learn Res 2011; 12: 2493–2537.

24.

Vinod Nair Hinton

. Rectified linear units improve restricted Boltzmann machines. In: Proceedings of ICML, 2010, pp. 807–814, https://www.cs.toronto.edu/~hinton/absps/reluICML.pdf

25.

Choi

S-P

Lee

Jung

et al . An intensive case study on kernel-based relation extraction. Multimed Tools Appl 2014; 71: 741–767.

26.

Miller

GA.

WordNet: a lexical database for English. Commun ACM 1995; 38: 39–41.

27.

Pyysalo

Airola

Heimonen

et al . Comparative analysis of five protein-protein interaction corpora. BMC Bioinformatics 2008; 9: S6.

28.

Manning

Surdeanu

Bauer

et al . The Stanford CoreNLP natural language processing toolkit. In: Association for computational linguistics (ACL) system demonstrations, pp. 55–60, http://nlp.stanford.edu/pubs/StanfordCoreNlp2014.pdf

29.

Chollet

Keras. GitHub, 2015, https://github.com/fchollet/keras

30.

Theano Development Team. Theano: a Python framework for fast computation of mathematical expressions (arXiv e-prints), 2016, https://arxiv.org/abs/1605.02688

31.

Srivastava

Hinton

Krizhevsky

et al . Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 2014; 15: 1929–1958.

32.

Tieleman

Hinton

. Rmsprop: divide the gradient by a running average of its recent magnitude. In: COURSERA: Neural Networks for Machine Learning, 2012.

33.

Prechelt

Early Stopping – but when? In: Orr

Müller

K-R

(eds) Neural networks: tricks of the trade. Berlin, Heidelberg: Springer, 1996, pp. 55–69.

34.

Saetre

Yoshida

Miwa

et al . Extracting protein interactions from text with the unified AkaneRE event extraction system. IEEE/ACM Trans Comput Biol Bioinform 2010; 7: 442–453.

35.

Bui

Q-C

Katrenko

Sloot

PMA

. A hybrid approach to extract protein-protein interactions. Bioinformatics 2011; 27: 259–265.

36.

Yang

Tang

Zhang

et al . Multiple kernel learning in protein–protein interaction extraction from biomedical literature. Artif Intell Med 2011; 51: 163–173.

37.

Nguyen

Grishman

. Relation extraction: perspective from convolutional neural networks. In: Proceedings of NAACL workshop on vector space modeling for NLP, http://www.cs.nyu.edu/~thien/pubs/vector15.pdf

38.

Chiu

JPC

Nichols

. Named entity recognition with bidirectional LSTM-CNNs, https://arxiv.org/abs/1511.08308