Abstract
Passage retrieval is an important stage of question answering systems. Closed domain passage retrieval, e.g. biomedical passage retrieval presents additional challenges such as specialized terminology, more complex and elaborated queries, scarcity in the amount of available data, among others. However, closed domains also offer some advantages such as the availability of specialized structured information sources, e.g. ontologies and thesauri, that could be used to improve retrieval performance. This paper presents a novel approach for biomedical passage retrieval which is able to combine different information sources using a similarity matrix fusion strategy based on convolutional neural network architecture. The method was evaluated over the standard BioASQ dataset, a dataset specialized on biomedical question answering. The results show that the method is an effective strategy for biomedical passage retrieval able to outperform other state-of-the-art methods in this domain.
Introduction
Biomedical question answering (QA) systems are an important asset to support clinical decision processes and personal health information needs [12]. The manual effort to find useful information in large biomedical scientific repositories is a challenging task that can be alleviated with accurate automatic QA systems.
Passage retrieval methods analyze a narrow set of documents with the aim of identifying snippets that help to answer a specific question. Increased performance in passage retrieval task tends to produce a significant gain on the overall QA task [25].
Passage retrieval over biomedical domains has received less attention comparably with open domains. This fact reflects on the few QA datasets available for the biomedical domain, as Wasim et al. [27] showed. The biggest collection of question-answer passages for the biomedical domain is the dataset released by BioASQ Question Answering Challenge [25] with 2,747 questions-answer pairs. On the other hand, open domain QA has larger resources with more training data, such as SQuAD dataset with more than 100,000 questions [18], or WikiQA with 3,047 questions [29]. More recently, Google released a new open domain dataset with 307,373 questions. Biomedical passage retrieval remains a real challenge where the presence of technical terminology, compound concepts, complex entities, and elaborated queries make the task harder. In this domain, the specialized information sources have not been fully exploited and most of the approaches take advantage exclusively of textual data as a unique source of information.
In this paper, we present a novel method for biomedical passage retrieval. The model has the ability to combine different information sources. Specifically, the model fuses different term similarity measures which capture different aspects of the question-passage relationship. The similarities are optimally combined by a convolutional neural network. In the reported experiments three different similarity measures were used: biomedical word2vec embedding cosine similarity, term co-occurrence, biomedical concept co-occurrence. The main assumption is that the three similarities are complementary to each other and offer different views of the semantic relations between question and candidate answers. To validate the model performance and compare it against state-of-the-art models, we carried out a systematic evaluation with the BioASQ biomedical question answering challenge dataset [25]. The results show that the proposed model is an effective strategy for passage retrieval in the biomedical domain with results that equal and, in most cases, exceed state-of-the-art methods’ performance.
The paper is organized as follows: Section 2 discusses the background and the related work; Section 3 shows the details of the proposed method; Section 4 presents a systematic evaluation of the method; finally, Section 5 exposes some conclusions and discusses our future work.
Background and related work
Biomedical passage retrieval
Biomedical queries are usually more specific than open domain search queries. This makes difficult the use of general-purpose retrieval models [25]. In the case of biomedical passage retrieval, most methods tend to only use textual information, a representative one is proposed by Brokos et al. [2]. They used an attention-based convolutional neural network to model question-answer pairs (ABCNN) [30]. This approach obtained the best results in the BioASQ 2018 challenge [5]. Other approaches use convolutional neural networks (CNN) and long-short-term-memory networks (LSTM) to represent the question and passage information to obtain a similarity score to rank the candidate answers [5].
Structured information sources
A typical biomedical query could be e.g. “How could we infer functional associations from gene fusion events?” This query involves specific domain knowledge related to genetics. Terminology databases and ontologies such as the Unified Medical Language System (UMLS) [8] encode, in a structured way, a good portion of this domain knowledge. Although specific domain knowledge resources are relevant in the sense that they can be used to alleviate problems as polysemy or synonymy disambiguation, they are not used frequently in passage retrieval tasks [11]. Despite this, there are some passage retrieval methods for open domain that exploit structured representations using different approaches: categorical grammars [6, 31], synchronous context-free grammars [28], reinforcement learning [32], dependency-based trees for compositional semantics [20, 24], and tree transducers [13].
In the open domain field, there are some works that try to incorporate structured based information. Das et al. [3] have demonstrated that a mixed data representation for question answering is better than using either a structured source or text source alone. In this approach the authors produce a joint representation with knowledge base facts and textual information (Universal Schema). A memory network [23] makes use of the produced join representation as an attention mechanism which is combined with the proposed query to select a related entity that answers the query. HAWK is another method developed by Usbeck et al. [26] where textual information is combined with linked data using an 8-step pipeline, which comprises: POS-tagging, NER, dependency parsing and linguistic pruning heuristics among others in order to discard no connected resulting graphs.
Similarity measures and passage retrieval
Almost all passage retrieval methods calculate some sort of similarity between the query and the passage. Some similarities are based on term-term similarities and others involve more semantic information. Semantic similarity measures are mainly based on large corpora where important relational patterns are extracted. Some of the approaches, as for example probabilistic hyperspace analog to language (HAL) [1], propose a semantic window of length K which is moved across the corpus of text.Terms contained in the window co-occur with a strength inversely proportional to term by term distance. They reported that when window size increases (K greater than 5), there was a diminishing on performance in information retrieval task.
Other approaches take into consideration the semantic and ontological relationships that exist between words. Thus, based on this knowledge, semantic similarity can be calculated following the minimal path between two nodes [22]. Ramage et al. have proposed a random walk algorithm [19] that compares the random walk graph generated between two terms to measure the semantic relatedness. They used WordNet and corpus statistics. These approaches are efficient when the coverage of the ontology is wide; in the biomedical domain, it is hard to have a 100% coverage.
Apart from ontological text representations, recently, authors have been working with word embeddings. These models represent each word as an n-dimensional vector, with the property that semantically related vectors are close to each other. Cosine similarity is one of the similarity measures that can be applied when text is represented as vectors. Other measures include Euclidean distance, soft-cosine similarity, and so on. Based on that, it can be said that the similarity measure election will guarantee the success of the model.
In Mikolov’s model [14], the semantic relation strength between a pair of terms is given by the occurrence in context windows. This parameter election will punish distant terms that can give important information, e.g., the following snippet of a biomedical article has two highly related entities “
The consequence will be a low spatial correlation in the semantic vector space. However, in some domains (such as biomedical), it is important to capture also more ’topical’ relationships [9].
In this work, we propose a passage retrieval method that takes advantage of different resources to build similarity measures. The obtained representation fits a deep learning model to extract similarity patterns in order to improve the performance on the passage retrieval task. The proposed approach combines three different similarity representations: 1) word2vec embedding cosine similarity, 2) term co-occurrence and 3) concepts co-occurrence. These similarities, extracted from large corpora, contribute with local and topical relatedness. The way to exploit these similarity patterns is based on a convolutional neural network.
Method
Overall architecture
The overall architecture is depicted in Figure 1. Figure swim lines indicate different stages which are explained in the following sections.
Passage retrieval overall architecture.
The first part of the process is to calculate the co-occurrence between pairs of terms and pairs of biomedical concepts. In this stage, we take a random sample of 30.000 biomedical documents from PubMed Baseline Repository (MBR) document set [17]. The objective is to build the vocabulary and to calculate the co-occurrences for both terms and concepts. For the later, we need to identify the biomedical concepts. For this task, we have used the terminology data source UMLS Meta-thesaurus1 which contains information about over 1 million biomedical concepts and 5 million concept names. As the process to match every term to a concept is computationally expensive, we take advantage of the QuickUMLS tool provided by Soldani et al. that has a good performance identifying concepts in large texts [21].
Experimentally, we have determined that the coverage of UMLS is not 100%. To overcome this limitation, a second check is performed with the Scispacy tool [16]. This Spacy model provides biomedical named entity recognition which increases the biomedical concept identification coverage. Once the vocabularies of terms and concepts were built, we filter out frequent terms and concepts which provide less information. Also, very rare terms and concepts are not taken into account. Figure 2 shows the count frequency of term and concepts.
Term and concept count frequency.
Now we have to indicate if a word appears in a given document and if keep that in a binary vector. The resulting matrix will have a dimension NtimesM, where N is the number of documents and M is the vocabulary size, with value 1 when the vocabulary word appears in the given n-th document.
With the document-word appearance matrix X calculated, we have to calculate the word by word normalized co-occurrence matrix to achieve that we apply the Equation 1.
The produced information in this step is: Term vocabulary Term co-ocurrence matrix Concept vocabulary Concept co-occurrence matrix
The process was also applied to sentence level co-occurrence, but instead of calculating the co-occurrence in documents, we split them into sentences and continued with the same process. Empirically, we have stated that document level similarity matrix achieves higher scores. Henceforth, in this paper we will understand co-occurrence similarity as document level similarity. Once co-occurrence matrices are calculated for terms and concepts, it is time to represent the model input data in the similarity matrices that the CNN model expects (co-occurrence term similarity; co-occurrence concept similarity and cosine similarity).
In order to transform each question and answer term pair (q, a), we have to retrieve the correspondent co-occurrence for the related pair. This process is also done for concepts if the word is identified as such. When a word is not in the vocabulary (term or concept) we fill the related cell with 0, this allows us to align the similarity matrices representation in all the three tensor dimensions.
Cosine similarity
Cosine similarity is another question and passage data representation. Each pair (q, a), is defined as a weighted cosine similarity score between question and passage pair words, as described below.
Here we can observe that the three similarity measures used in the proposed approach capture different aspects of semantic relatedness. When only using one similarity measure, the method may fail to capture all the important aspects of the semantic relatedness. This can be seen in the following example:
Q: Abnormality in which vertebral region is important in Bertolotti’s syndrome?
A: Patients with Bertolotti’s syndrome have characteristic lumbosacral anomalies and often have severe sciatica.
The three similarity matrices visualisation is represented with the following heat maps, see Figure 3.
Example 1. Similarity matrices.
It can be observed that the cosine similarity matrix does not have a high value for “Bertolotti’s” term. It is because there is no vector representation for the term, but the co-occurrence matrices for term or concept have the highest values in the related cell values. In the same way, the “Bertolotti” concept is highly correlated with “syndrome” and “lumbosacral” in the concept co-occurrence matrix which are important concepts to answer the question. In the case of the term co-occurrence matrix, the similarity is less precise but gives a high score for the related term “sciatica”. As the similarity matrices show, they are complementary to each other and they produce important patterns to rank a set of candidate answers.
Another example of how similarity measures contribute to an improved representation is presented in the following example:
Q: Are defects in recombination repair involved in carcinogenesis?
A: Inherited mutations in genes involved in HR are associated with gene rearrangement and may be a prerequisite for tumor development in some cancer-prone hereditary diseases like Bloom, Werner, and Rothmund-Thomson syndromes.
We can see similarity matrices as heat maps in Figure 4. In this case, the cosine matrix has a high similarity score between “recombination” and “rearrangement”, while co-occurrence representation score is low. All three matrices have a high score for “carcinogenesis” in the question and “tumor”, “development” and “cancer” in the answer.
Example 2. Similarity matrices.
The objective with the incorporation of additional and complementary information is to feed the neural model with meaningful features that allows the model to identify when a question and answer pair are highly correlated. During the training phase, the CNN model has to determine those similarities patterns that we hypothetical highlight.
Convolutional Neural Nets (CNN) were originally developed for image processing, where the important information may appear on arbitrary regions of the image, represented frequently as a 3 channel RGB matrix. The same assumption can be applied to our similarity matrices.
Once the (q, a) pairs are represented as the three similarity matrices, we feed them to the CNN model presented in Figure 5. The CNN layer will identify word-similarity patterns in each of the three channels. The patterns are captured for the 64 filters to be then sub-sampled by a pooling layer. The pooling layer for all the filters is merged with a fully connected layer. Finally, an output sigmoid unit produces a similarity score based on the evidence coded by the neural networks units activation values.
Multiple channel convolutional neural network.
Once the training phase has been completed we obtain a similarity discrimination model that is capable of measuring the semantic correlation between question and answer pairs and produce a final score.
The next step is to use the model to rank candidate answers (a1, a2, . . . , a k ) against a given query q. The candidate answers are retrieved based on the highest scores.
Experimental evaluation
The experimentation was carried out over the BioASQ 6 challenge dataset. We evaluate different method combinations in order to measure how important is each of the similarity measures for the passage retrieval task. Finally, we will combine all three similarity matrices to validate the complementary information hypothesis.
Data set
The training was done with the question and answer pairs from 2016, 2017 and 2018 BioASQ Task B training datasets. The obtained dataset was very unbalanced, only 18% of the total number of pairs are labeled as a relevant answer. To balance the dataset, the sample extraction in the training phase was done with the same number of positives and negative samples, this strategy is also applied in the validation phase.
Experimentation models
In order to compare the discriminative power of the proposed model using the related three similarity feature matrices, we introduce the following model configurations: 1) using just the term co-occurrence matrix as input to the CNN (term); 2) using just the concept co-occurrence matrix as input to the CNN (concept); 3) combine term and concept co-occurrence (term + concept); 4) using the cosine similarity matrix (w2v), 5) combine cosine similarity with term co-occurrence (w2v + term); 6) combine cosine similarity with concept co-occurrence (w2v + concept); 7) combining all three similarity measures (w2v + term + concept). Besides these methods, we will compare the latest configuration (w2v + term + concept) against the proposed baseline models: a self-trained finetuning BERT model (BERT) and the winner model from last year BioASQ challenge (aueb-nlp-5). To give a broad definition of BERT model we are going to detail the process followed to finetune BioBert.
Bert finetunned model baseline
Language pre-trained models have proven to be useful for universal textual representations. One of the last pretrained models is BERT (Bidirectional Encoder Representations from Transformers) which has achieved an important result for different NLP tasks. Recently a pretrained BERT model over biomedical and open domain data was released by Lee et al. [7].
In order to validate state-of-the-art methods, we have finetuned BioBert to achieve the passage retrieval task. We have followed the approach for sentence pair classification task. The data used to finetune the model was the same to train the proposed model.
Results and discussion
The results for different model configurations are reported in Table 1.
Snippet retrieval results combining similarity matrices
Snippet retrieval results combining similarity matrices
Results show that the most informative individual similarity measure is the cosine similarity (w2v) with the proposed POS-tagging salience weighting. Term and concept co-occurrences have very close scores in all the batches when used separately. The combination (term + concept) improves significantly the scores as expected. Combining (w2v + term) and (w2v + concept) is quite similar, the scores are close, but when all three similarity measures are jointly used there are important improvements in the MAP metric across all batches.
We present the following question (Q) and answer candidate (A) example extracted from experimental data-set to show the model contribution.
Q: Does echinacea increase anaphylaxis risk?
A: Risk of anaphylaxis in complementary and alternative medicine.
The produced similarity matrices are depicted as heat maps in order to visualize the similarity strength between terms and concepts, see Figure 6. In this example, the concept similarity matrix offers higher values for co-occurrence similarity between echinacea and anaphylaxis allergic reaction. Verifying in the medical literature, there are documented adverse reactions associated with echinacea which support our observations.
Similarity matrices example where concept co-occurrence have a better performance over the others
We have conducted our experimentation with the test batches released for BioASQ 6b. In order to compare our results with state-of-the-art methods, we have included last year winner team (Athens University and Google [2]) results. Since snippet retrieval highly depends on document retrieval, and with the objective to make a fair comparison of our proposed method, we asked the winner team to share with us the documents obtained in document retrieval step. They shared the submitted files and, therefore, a snippet retrieval isolate comparison was possible to carry out.
The scores presented in Table 2 for aueb-nlp-5 [2] were extracted from the BioASQ results leader board table. This is the system that reached the highest scores. In the same way, we reported the scores that our Bert fine-tuned model and our fusion model with three similarity measures obtained when using the same set of documents from aueb-nlp-5.
Snippet retrieval results using the documents provided by AUEB [2]
Snippet retrieval results using the documents provided by AUEB [2]
The proposed model scores are consistent in the five batches and the difference against the best model from last year (aueb-nlp-5) is 3.5 percent points on average, across all the batches. We can also see that the BERT based model is competitive, although their scores are below those from the other two models.
The next comparison was carried out against the 15 best models from the 2018 BioASQ challenge. In order to visualize the scores in a more friendly way, we have consolidated the results from the leader board table in a box plot, see Figure 7. There is one box plot for each batch, and the X-axis corresponds to the reported metrics in BioASQ 6 (mean precision, F-score, recall, MAP, GMAP). The blue point is the score obtained with our model using the documents supplied by [2].
15 best systems results for task 6b, blue points correspond to the proposed model
In batch1, batch3 and batch4 we reached the best results as ilustrated in the boxplot. In batch2, the only measure where the model is not the best is recall. Still, they are in the highest quartile. In the last batch, the scores for all the teams are lower than in previous batches. The result of our model is the highest in MAP and competitive according to the other metrics.
In this paper, we have presented a novel approach for biomedical passage retrieval. The proposed method is based on different similarity measures that offer complementary information in order to semantically match question and answer passages.
The proposed similarity measures come from concepts and term co-occurrence, in addition to a word-embedding cosine similarity. Concepts are extracted using the UMLS terminology data source and the Scispacy model. The multi-similarity representation is exploited by a convolutional neural network which extracts similarity patterns and produces a semantic relatedness score, which is further used to rank the answer passages. We have tested different combinations of similarity measures, and the most accurate was the one in which we used all three similarity measures, which validate the hypothesis that the similarities are complementary to each other.
The proposed model was tested within BioASQ 6b dataset and the scores obtained were compared against the best models reported for the 2018 challenge. The obtained results showed that the proposed model outperformed all the methods used in BioASQ challenge with a substantial difference. Motivated by the obtained results, future work will be focused on extending the similarity representation and exploiting it with more sophisticated neural models that better use the multiple information.
Footnotes
Acknowledgment
COLCIENCIAS, REF. Agreement #727, 2016 provided financial as well as logistical and planning support. Mindlab research group (Universidad Nacional de Colombia sede Bogotá) with the cooperation of INAOE (Instituto Nacional de Astrofísica, óptica y Electrónica) and Universitat Politécnica de Valéncia wich also provided technical support for this work. The work of Paolo Rosso was carried out in the framework of the research project PROMETEO/2019/121.
UMLS Meta-thesaurus http:umlsks.nlm.nih.gov
