Abstract
In this paper, we propose a method exploiting syntactic structure, semantic relations and word embeddings for recognizing textual entailment. The sentence pairs are analyzed using their syntactic structure and categorization of sentences in active voice, sentences in passive voice and sentences holding copular relations. The main syntactic relations such as subject, verb and object are extracted and lemmatized using a lemmatization algorithm based on parts-of-speech. The subject-to-subject, verb-to-verb and object-to-object similarity is identified using enhanced Wordnet semantic relations. Further similarity is analyzed using modifier relation, number relation, nominal modifier relation, compound relation, conjunction relation and negative relation. The experimental evaluation of the method on Stanford Natural Language Inference dataset shows that the accuracy of the method is 1.4% more when compared to the state-of-the-art zero shot domain adaptation methods.
Introduction
The advancement in information technology has led to the proliferation of a vast amount of unstructured data on the Internet. These data contain potential information that has to be processed by employing automatic methods. Natural Language Processing (NLP) facilitates computers to analyze, understand and derive meaning from human languages in an efficient way. The ambiguity that exists in natural languages and the complexity of semantic information in simple sentences make the processing of natural language difficult.
Recognizing Textual Entailment (RTE) is a challenging problem in NLP. RTE is a classification task in which the sentence pairs are classified into three categories - entailment, contradiction, and neutral. If the meaning of one text can be inferred from the other then the two sentences are in entailment relation. The entailing text is called Text (T) and the entailed text is called Hypothesis (H) [1]. If T contradicts H, then the sentences are in contradiction relation. If the two sentences are unrelated, then the relation between them is neutral. The RTE challenge [2] provides a platform for evaluating textual entailment tasks. In this paper, we propose a statistical method for the classification of sentence pairs into entailment, contradiction and neutral by utilizing the syntactic structure, semantics that contained in the sentence pairs and word embeddings. In addition to the extraction of main syntactic relations such as subject, verb and object, we have also considered modifier, nominal modifier, number, compound, conjunction and negative relation for the correct classification of sentence pairs. The contributions of this paper are summarized as follows:
A method is proposed for classifying sentence pairs into entailment, contradiction and neutral categories making use of syntactic structure, semantic relations, and word embeddings for sentence classification and dependency relations with an accuracy of 69.1% which is 1.4% more than the state-of-the-art schemes.
The remainder of this paper is organized as follows: Section 2 describes the related works in Natural Language Inference. Section 3 introduces the proposed method. Section 4 deals with the experiments done. Section 5 details on results and discussions. The conclusions and future works are discussed in Section 6.
Related work
Statistical methods for recognizing textual entailment (RTE) are widely used in literature. Shih et al. [3] proposed a method for contradiction detection in texts by measuring the availability of Mismatch Conjunction Phrases (MCP). Using MCP’s for detecting contradiction solved the problem of shortage of background knowledge. Zanzotto et al. [4] proposed a method for discovering textual entailment relations in sentence pairs using textual entailment patterns. The method relies on linguistic knowledge and on probability distribution. Another method for textual entailment focusing on contradiction detection is proposed by Magnini and Cabrio [5]. In this method, monothematic pairs are created by decomposing Text - Hypothesis pair. Then entailment judgement is assigned on these monothematic pairs. A method for the detection of textual entailment using textual relatedness measure is proposed by Wang and Zhang [6]. This helps to recognize the neutral cases better which inturn boost the performance of RTE task. In 2018, Masatoshi Tsuchiya [7] proposed a method based on statistical hypothesis test which analyze the performance impact caused by hidden bias of training data. The method is applied on Stanford Natural Language Inference (SNLI) and Sentences Involving Compositional Knowledge (SICK) dataset and obtained an accuracy of 63.3% and 56.7% respectively.
In 2008, Islam and Inkpen [8] proposed Semantic Text Similarity (STS), a method for measuring the semantic similarity of texts using a corpus based measure of semantic word similarity and a normalized and modified version of the Longest Common Subsequence (LCS) string matching algorithm. The proposed method determines the similarity of two texts from semantic and syntactic information, in terms of common-word order that they contain. In 2014, Ferreira et al. [9] proposed a sentence similarity measure based on three layer representation. The method considers lexical, syntactic and semantic information contained in the sentences. In 2020, Yang et al. [10] proposed a method to find the similarity between short texts. This method combines semantic and syntactic information by using knowledge and corpora to express the meaning of the term to solve polysemy, and using a constituency parse tree to capture the syntactic structure of short texts. The proposed method is evaluated on 24 widely used datasets on Semantic Textual Similarity (STS) tasks which contain a wide range of domains. Experimental results demonstrate that the proposed method outperforms baseline methods.
In 2019, Reshmi and Shreelekshmi [11] proposed a contextual method based on semantic similarity using Wordnet [12]. The method finds semantic similarities between the sentence pairs. The content words from both the sentences are united and converted into semantic vectors corresponding to each sentence using Word similarity measures defined in the Wordnet. Based on the cosine similarity score, a threshold is applied and the sentence pairs are classified into entailment and non entailment categories. The method is tested on RTE2 RTE5 and SNLI developmental datasets. The overall accuracy obtained on the RTE2 dataset is 71.54% and on the SNLI dataset, the accuracy is 56.002%. The proposed method on RTE5 got an accuracy of 54.44% whereas the model Deep Explorations of Inter- Sentence interactions for Textual Entailment (DEISTE) [13] on RTE5 (pre-trained on Scitail [14]) obtained 60.2% and DEISTE on RTE5 (pre-trained on SNLI) yield an accuracy of 46%.
Recently many deep learning methods are introduced for the inference of Natural Language. In 2015, Bowmann et al. [15] proposed SNLI Dataset, a large annotated corpus for recognizing textual entailment. Such a large corpus being available, many deep learning techniques are proposed for textual entailment and have come up with good results. In 2017, Chen et al. [16] suggested an enhanced Long Short Term Memory (LSTM) for natural language inference. The method gained an accuracy of 88.6% on the SNLI dataset, which outperforms previously used methods. In 2019, a multi-task learning method for understanding the relationship between sentences is proposed by Choi and Lee [17]. The proposed model does not require any external tools and task-specific parameters, unlike feature-based models. The model reads the plain text and generates a single universal representation to predict multiple tasks. The proposed model outperforms other encoding-based models by jointly encoding sentence pairs. The model gains an accuracy of 87.7% on the SNLI dataset. In 2020, a Supervised Attention-based Natural Language Inference (SA-NLI) which unifies the supervised training of attention modules and the training of the NLI model is proposed by Li et al. [18]. The proposed method solves the explanatory problem of the attention mechanism. The proposed method is tested on SNLI and Scitail datasets which gained 88.4% and 79.5% accuracies respectively. In 2020, Du et al. [19] proposed a method to conduct natural language inference with Enhanced Relation-Head-Dependent triplets (RHD triplets), which are constructed via enhancing each word in the RHD triplet with its associated local context. The approach is tested on SNLI developmental and test sets and gained 92.5% and 87.5% accuracies respectively.
Deep learning methods which use transfer learning is also gaining much attention in the field of NLI nowadays. In 2019, Liu et al. proposed a Multi Task Deep Neural Network (MT-DNN) [20] which incorporates a pre-trained language model BERT (Bidirectional Encoder Representations from Transformer) [21]. Domain adaptation performed on SNLI and Scitail developmental datasets gained an accuracy of 92.2% and 96.3% respectively which shows the model trained using MT-DNN outperforms BERT model. The test result on the same dataset is 91.6% and 95.0% respectively. In 2020, a semantics aware BERT (SemBERT) model [22] which is capable of including contextual semantics over a BERT model is proposed by Zhang et al. The model attained an accuracy of 92.2% and 91.9% on SNLI developmental and test set. In 2020, a Conditionally Adaptive Multi Task Learning (CA-MTL) network for improving transfer learning in NLP using fewer parameters and less data is proposed by Pilault et al [23]. The domain adaptation performed on SNLI and Scitail developmental set achieved an accuracy of 91.5% and 95.6% respectively and that of test set yield 92.1% and 96.8%.
In 2021, Cui et al. [24] proposed a method for natural language inference by projecting the superficial words. The semantics of the words that do not affect the inference is called superficial words. These words cause the overfitting of the model. A parameter-free model is set up for projecting the superficial words. This model also measures the generality in the single domain and zero-shot target domain. In zero-shot domain adaptation, the model is trained in a source domain and tested in a different domain. This approach outperforms the competitors over the source and zero-shot domains. The zero-shot domain adaptation for the SNLI dataset shows an accuracy of 67.7% when trained in Multi-genre NLI Corpus (MultiNLI) [25]. In 2022, Park et al. [26] proposed an adaptive convolution to provide flexibility to the existing convolution neural networks. This approach is tested on various natural language understanding tasks. The method is trained and tested on SNLI and MultiNLI datasets and achieved an accuracy of 87.9% and 74.7% respectively. The transfer learning performed on the model trained using SNLI and tested on SICK-E gained an accuracy of 46.7% [27].
Proposed method
The proposed method exploits lexical, syntactic, and semantic information contained in the pair of sentences. The method includes pre-processing, lemmatization, all-word matching, syntactic matching, and finding the semantic relationships of the words based on Wordnet. This method also makes use of pre-trained word embeddings called GloVe [28] for converting the words into word vectors. The word-to-word similarity is determined with the help of the external knowledge resource, Wordnet. The Wordnet semantic relations synonyms, antonyms, holonyms, hypernyms, hyponyms, meronyms and entailment are used for finding whether two words are similar or not.
The steps involved in classifying the sentence pairs into entailment, contradiction and neutral is given in Algorithm 1. The main steps in the proposed method includes preprocessing, enhanced lemmatization, entailment detection and contradiction detection. The proposed method for the entailment and contradiction detection is given in Algorithm 2 and Algorithm 6 respectively.
Preprocessing
In the preprocessing module, the punctuations and stop-words are removed. The sentences with undefined labels in the dataset are also eliminated. The numbers represented in figures are converted to words so as to help in the process of token matching. The sentences with spelling mistakes are eliminated from further processing. After parsing, the sentences that results in incorrect parsing output and incorrect Parts-Of-Speech tags are also eliminated.
Enhanced lemmatization
The proposed method uses the Lemmatization module present in the Natural Language Tool Kit (NLTK). NLTK fails to provide the dictionary form of certain words. For example, it returns lemmatized form of noun plural, ‘men’ as ‘men’. Similarly, the auxiliary verbs ‘could’, ‘would’, ‘should’ and ‘might’ are lemmatized to the same words. The verb ‘bore’ and ‘borne’ are not lemmatized to ‘bear’. The proposed method includes a lemmatization algorithm for finding the root forms of the verb which are not correctly identified by the NLTK.
Sentence classification using Allword matching
After preprocessing, tokenization of both the sentences are carried out. The tokens (words) are Lemmatized using the Lemmatization algorithm. After eliminating the stop-words in the sentence pairs, the tokens in the Hypothesis are matched with that of the Text. For finding the semantic relations between the words, Wordnet semantic relations are used. In Wordnet, the synsets (synonym sets) of different words are related through any one of relations - synonyms, antonyms, holonyms, hypernyms, hyponyms, meronyms and entailment. If all the tokens are matched either directly or through any of the Wordnet relations, then those sentence pairs are classified as entailment. If the tokens are related through antonym relation then those sentence pairs are classified as contradiction.
Sentence classification using syntactic and semantic relations
The sentence classification using syntactic and semantic relations is discussed in the following sections.
Dependency Parsing
Lexicalized dependency parser in NLTK [29] is employed to parse the natural language sentences. The lexicalized probabilistic parser in NLTK is a factored model for generating natural language tree structures making use of syntactic (Probabilistic Context Free Grammars (PCFG)) structures and semantic (lexical dependency) structures. The scoring of PCFG structures and lexical dependency structres are combined using an A* algorithm. A lexicalized dependency tree is a combination of phrase structure tree and dependency tree. The lexicalized dependency parse tree for an input sentence is shown in Fig. 1.
Lexicalized dependency parse tree. The dependency parser output can be either dependency parse tree as shown in Fig. 1 or dependency triplets which is used in syntactic and semantic analysis of sentences.
Analysis of syntactic structures
The main dependency relationships such as subject, verb and object are extracted from dependency triplet form of the parser output and checked for matching. The other relationships such as modifier, nominal modifier, number relation, compound, conjunction, negation etc are also extracted for the proper inferencing process. While extracting the nominal modifier relationship, there may be subject nominal modifier, verb nominal modifier and object nominal modifier. All these relations are extracted and considered for the inferencing process.
Nominal modifier relation
If the the Hypothesis sentence contains object and the Text sentence doesn’t hold an object, the nominal modifier of the Text is extracted. Matching process is carried out between the object of hypothesis and nominal modifier of Text sentence. Consider the sentence pairs. ‘Two people are waiting for a train at a station.’ and ‘They expect a train to arrive eventually.’ In this case, the hypothesis contains ‘train’ as object whereas the Text sentence doesn’t hold an object. Instead it contains ‘train’ as the nominal modifier of the verb. This is a case of entailment. Further, if the sentences involve the words such as ‘group’, ‘pair’ and ‘bunch’, the nominal modifiers of the subject are considered to correctly extract the subject.
Conjunction relation
For extracting the subject and object relation, we have also considered the conjunction relations. Consider the words that are related through conjunction dependency relation. ‘a man and a woman.’ The parser outputs ‘man’ as the subject. If the other sentence contains ‘couple’ as the subject, matching of subject results in no match. Hence conjunction relation is considered to extract the correct subject ‘man and woman’ so as to match with couple.
Compound relation
If the subject or object is part of a compound word, then to extract the correct subject and object, subject compound and object compound relations are considered. The words that are related through compound relation are extracted and added to the subject and object list accordingly for carrying out correct classification.
Number relation
If the subject and object contains numbers the number relation is extracted and considered for classification.
Preposition relation
For the sentence given below to obtain the correct verb form we have converted the preposition ‘in’ to the verb ‘wearing’ based on the context. ‘Men in blue shirt.’ becomes ‘Men wearing blue shirt.’
Categories of sentences
The sentences in the dataset are classified into three categories - sentences in active voice, sentences in passive voice and sentences in which the verbs represent copular relation with the subject and the object. These sentences are treated separately to extract the syntactic relations. For sentences in active voice, the subject, verb and object are extracted and matched against corresponding relations of the other sentence. The subject, verb and the object of the sentences in passive voice are also extracted after parsing. For example, consider the passive sentence ‘Ravan was killed by Ram.’ In this case, ‘Ravan’ is the passive subject and ‘Ram’ is the object. After the extraction of the syntactic relations, the syntactic matching is carried out to find the relation of the sentence pairs. Consider the sentence, ‘A person is on a beach.’ Here the subject and the predicate are related through the copular verb ‘is’. These kind of sentences are to be considered separately for the extraction of subject and object.
Sentences with multiple subjects, verbs and objects
Some sentences contain more than one subject, verb and object. In such cases, each of these relations extracted and added to the subject, verb and object list of both sentences. Matching is carried out by considering each member of the list with that of other sentence list.
Analysis of Semantic Relations
After extracting syntactic relations from the sentence pairs, the semantic relations between subject and subject, verb and verb and object and object are analyzed. The semantic relations are identified using Wordnet and Enhanced Wordnet semantic relations. If none of the semantic relation is an antonym relation, then the sentence pair is classified as entailment. If any one of these relation is an antonym relation then the sentence pair is classified as contradiction.
Similarity using word embeddings
The subject, verb and object are extracted from each sentence. Similarity between subjects, similarity between objects and similarity between verbs are separately analyzed based on Wordnet semantic relation. If no similarity is identified subject, object and verbs of the sentence pairs are represented as word vectors using pre-trained word embeddings, GloVe. Then cosine similarity between the word vectors are calculated using Equation 1.
where v1, v2 are word vectors. After the similarity computation, a threshold is set for the similarity score. If any one of these similarity values is below the threshold, then the sentence pairs are classified as contradiction.
Modifier relation based similarity
For correct classification of sentence pairs, modifier relation is also used. Consider the sentence pair which is a contradiction. ‘A man in a white t-shirt takes a picture in the middle of the street with two public buses in the background.’ and ‘A man is wearing a black shirt’ Since all the colors are semantically related, the Wordnet relation fails to identify the contradiction relation. We make use of modifier relation to identify color for correct classification.
Enhanced Negative Detection
The sentences that contains words such as ‘not’, ‘don’t’ and ‘no’ are identified as negative relations. These relations are extracted and the existence of negative relations are checked for subject negation and verb negation and object negation. If subjects of both the sentences are same and one of the sentence contain negative word prefixing subject, then it is subject negation. Similar is the case with verb negation and object negation. However by identifying a negative relation alone, the sentences can’t be classified as contradiction. For example, consider the following sentence pairs. ‘A man sits on the edge of a cliff.’ and ‘A man is not standing. Even though the sentence contains a negation verb, the relation between them is entailment. Since the Hypothesis contains negation ‘not’ and the verbs in the sentence pairs are antonyms, the sentences are classified as entailment. Consider the sentences, ’A man with a white towel wrapped around the lower part of his face and neck.’ and ‘He doesn’t want hair to land on his neck.’ Even though the hypothesis is identified as negative, the relation between them is neutral. The subject, object and verb negation fails in this case.
Enhanced Wordnet relations
Consider the sentence pair. ‘Two girls play with origami’ and ‘The girls hold paper’ In the above sentence pair, origami and paper are related. But this relationship don’t exist in Wordnet. Hence we add a dictionary to extend this relationship. Consider the contradiction case of the following sentences. ‘A lady reading a story book.’ ‘A lady reads a science textbook.’ To correctly classify these sentences an antonym dictionary of science book and story book is added in the proposed method.
The steps involved in detecting the entailment and contradiction cases are described in further sections.
The steps involved in entailment detection is given in Algorithm 2. The sentence pairs are tokenized and performed token matching using the algorithm given in Algorithm 6. Usually the hypothesis contains less number of tokens. The tokens in the hypothesis sentence are matched with that of the Text sentence. If all the tokens are matched directly or through any of the Enhanced WordNet semantic relations then the sentence pairs are classified as entailment. The syntactic relations of the sentence pairs are extracted and token matching is performed on the relations such as subject, verb and object. If the suject-to-subject, object-to-object and verb-to-verb tokens matched, then the sentence pairs are classified as entailment. If the hypothesis contains object and the text sentence lacks it, then the nominal modifier of verb relation of text sentence is extracted and token match is performed with object of the hypothesis. If a match is found, then sentences are classified as entailment.
Contradiction detection
The steps involved in contradiction detection is given in Algorithm 6. The sentence pairs are tokenized and the resulting tokens are matched using Algorithm 4. If any one of the tokens are matched through antonym relation then the sentence pairs are classified as contradiction. The syntactic relations of the sentence pairs are extracted and token matching is performed on the relations such as subject, verb and object. If the subject-to-subject or object-to-object or verb-to-verb tokens are matched through antonym relation, then the sentence pairs are classified as contradiction. If the above relations are not opposite in meaning, then similarity using word embeddings is found as described in section 3.4.c.i. Detection of contradiction is also done by extracting modifier relations which is described in section 3.4.c.ii and using an enhanced negative detection algorithm which is described in section 3.4.c.iii.
The syntactic structure and the semantic analysis of the proposed method for a sample sentence pair is shown in Fig. 2. The Text sentence and the Hypothesis sentence are parsed to generate the syntactic structure. Consider the Text sentence: ‘A man and woman are shopping at a large store.’ The initial symbol S is rewritten to produce a new state (NP, VP). The NP is rewritten inturn, giving the new state of (DT, NP). The parse continues finding A as DT and produce new state (NN, CONJ, NN) which inturn finds man as NN, and as CONJ and woman as NN. Similarly, VP is rewritten giving the new state (VP, PP) and VP inturn produces the state (AUX, VBG). Hence are is identified as AUX and shopping as VBG. The PP finds at as PP, a as DT, large as JJ and store as NN. The lexicon and parts of speech tags are generated. The same process is repeated for Hypothesis sentence to generate the lexicon. The subject, verb, object and other relations such as modifier, nominal modifier are extracted and lemmatized. We have considered the conjunction relation of Text sentence for extracting the subject as ‘man and woman’ instead of ‘man’ so as to match with the subject, ‘couple’ of Hypothesis. Hence the sentences are classified as entailment.
Syntactic and semantic analysis.
Dataset
The experiment is conducted on SNLI [15] dataset. The dataset contains 570k human written English sentences that are semantically similar or contradictory or neutral. Based on the similarity, the sentences are classified into three labels - Entailment, Contradiction and Neutral. The developmental set contains 10,000 pairs of sentences. The sentences include sentences in active voice, sentences in passive voice, negative sentences, sentences with multiple subjects, verbs and objects. Some sentences have no objects. We have analyzed the whole sentences in the SNLI developmental set to eliminate the sentences with spelling mistakes, incorrect parser output, and incorrect pos tags.
Experiments
The experiments are carried out using Python 3.6 version on Ubuntu 18.04 platform. The performance is evaluated in terms of accuracy, Precision, Recall and F1 score. The accuracy is measured using equation 2.
The F1 score is evaluated using the equation 3.
The Experiment I is carried out by considering the entailment conditions first. That is the sentence pairs are first checked for entailment condition. If the conditions are not satisfied, it is checked for contradiction conditions. If the checking fails, then those sentences are treated as neutral.
The results of the Experiment I are shown in Table 1. The entailment accuracy is high as 99.78% and neutral accuracy is very low as 27.52%. To improve overall accuracy from 64.41% another Experiment II is conducted.
Results obtained on experiment I for SNLI developmental set
Results obtained on experiment I for SNLI developmental set
The results of the Experiment I are recorded. In Experiment II, the same set of sentences are checked for entailment conditions and results are noted. The same set of sentences are again checked for contradiction conditions. If this condition is not satisfied then the sentence pairs are classified as neutral. Those sentence pairs which results in entailment class in Experiment I are labelled as entailment. The sentence pairs which are classified as contradiction in Experiment II are labelled as contradiction.
The correct classification of neutral class is very less in the above case. For neutral accuracy improvement, the sentences which are classified as neutral in Experiment I or Experiment II are classified as neutral. Thus the neutral accuracy is improved to 27.52% and the overall accuracy is improved to 69.1% using sentence classification algorithm given in Algorithm 1.
The accuracy obtained for each class is shown in Table 2.
Results obtained on Experiment II for SNLI developmental set
Results obtained on Experiment II for SNLI developmental set
The comparison of the proposed method with state-of-the-art zero shot domain methods is given in Table 3. The proposed method achieves an accuracy of 69.1% which is better than all other zero shot domain adaptation methods.
Performance comparison of the proposed method with other methods
Performance comparison of the proposed method with other methods
The method e-SNLI shows the result of 46.7% when trained on SNLI and tested on SICK-E dataset. DIESTE shows the result when trained on SNLI and tested on RTE5. The TE Label Prediction model achieved the result when trained and tested on SNLI. In addition to conducting the above Experiments for the SNLI developmental set, we have conducted Experiment II on SNLI test set. The results obtained on the test set is given in Table 4.
Results obtained on SNLI test set
The performance comparison with other deep learning methods is given in Table 5.
Performance comparison of the proposed method with other deep learning methods
The proposed method uses syntactic structure and word level semantic information for textual entailment classification problem. Apart from the main syntactic relations such as subject, verb and object, other syntactic relations such as modifier, number, conjunction, nominal modifier etc are also considered for the semantic analysis of the sentence pairs. The enhanced negative detection approach used in the proposed method increases the accuracy of contradiction. The results of the experiments conducted on SNLI dataset shows that the performance of the method in terms of accuracy is better than the deep learning methods which employs zero shot domain adaptation. The proposed method doesn’t consider phrases and idioms in English language. Also the proposed method doesn’t take care of the different tense forms of the verbs in the sentence. Incorporating the deep learning techniques into the semantic analysis for better zero shot domain performance is future direction of this work.