Multilingual Question Answering Over Linked Data Building on a Model of the Lexicon-Ontology Interface

Abstract

Multilingual question answering (QA) over linked data can make structured knowledge accessible across languages. However, most state-of-the-art QA systems are limited to English and rely on machine learning methods that lack controllability and extensibility, making their governance and incremental improvement challenging, along with the effort needed to collect and provide training data. To address this, we follow a model-based approach to QA that builds on a declarative model of the lexicon–ontology interface, OntoLex-Lemon, which enables the specification of the meaning of lexical entries with respect to the vocabulary of a particular dataset. From such a lexicon, in our approach, a QA grammar is automatically generated that can be used to parse questions into SPARQL queries. We demonstrate the extensibility of the approach to different languages by adapting it to German, Italian, and Spanish. We evaluate the approach with respect to the QALD benchmarks and show that our approach outperforms the state-of-the-art on all these datasets in an incremental evaluation mode in which additional lexical entries for test data are added. For example, on QALD-9, our approach obtains $F_{1}$ scores of $0.85$ (English), $0.82$ (German), $0.65$ (Italian), and $0.83$ (Spanish) in an incremental evaluation mode in which lexical entries covering test data questions are added. So far there is no system described in the literature that works for at least four languages while reaching state-of-the-art performance on all of them. Finally, we demonstrate the low efforts necessary to port the system to a new dataset and vocabulary.

Keywords

multilingual question answering over linked data model-based QA approach lemon lexica grammar generation parsing

1. Introduction

As the amount of available multilingual linked data keeps growing, intuitive and effective paradigms for accessing and querying this body of knowledge become increasingly important. Previous research has stressed the need to transform linked data into a “global database” in which information is accessible across languages, thus creating a level-playing field in which each language community has equal opportunities to contribute to and benefit from the growing linked data ecosystem (Gracia et al., 2012).

There are approximately $7, 000$ languages spoken worldwide and a large amount of multilingual linked data available. Yet, English predominates in research communities focused on question-answering systems over linked data (QALD) (Shekarpour et al., 2016; Unger et al., 2014). Previous research (Perevalov et al., 2018, 2022d) has examined the availability of multilingual QA systems developed in the last 10 years. Furthermore, the applicability of the machine translation approach to monolingual QA systems (Perevalov et al., 2022a) was investigated. The findings include that there are a few multilingual QA systems over linked data, but most of them can not be used at the moment due to the unavailability of either the online demo, API, or source code. While it is possible to adopt a QA system to any language using machine translation (MT), in most cases, it yields comparable performance (F $_{1}$ -measure) only when the source language is translated into English. However, it requires the availability of a machine translation tool for all languages to English, which is not available for many of the world’s $7, 000$ languages.

State-of-art approaches to QALD are machine learning-based and can be broadly grouped into three categories (Chakraborty et al., 2019): (i) classification-based approaches, where neural models are used to predict one of a fixed set of classes given a question; (ii) ranking-based approaches, where neural networks are used to compare different candidate logical forms with the question to select the best-ranked one; and (iii) translation-based approaches, where the network learns to translate natural language questions (NLQs) into their corresponding logical forms. Many neural network-based approaches have proven effective for the QALD task, ranging from simple neural embedding-based models (Bordes et al., 2014) and attention-based recurrent models (Cheng et al., 2019) to memory-augmented neural controller architectures (Liang et al., 2017; Neelakantan et al., 2017; Yang et al., 2017).

Recently, transfer learning-based QALD (Lukovnikov et al., 2019) have demonstrated that significant improvements can be achieved by fine-tuning a pre-trained language model trained on a large corpus. Yin et al. (2021) used three types of neural machine translation models, including RNN-based, CNN-based, and transformer-based models for translating natural language to SPARQL. Research by Wang et al. (2022) has demonstrated that GNNs naturally fit the graph-structured knowledge and show promising results on the QALD task. The QA-GNN model (Yasunaga et al., 2021), which jointly reasons over knowledge from pretrained language models and knowledge graphs using graph neural networks (GNNs), has been shown to yield strong question-answering (QA) performance compared to models that rely solely on knowledge graphs (KGs) or language models (LMs). While machine learning-based systems have seen increasing performance in recent years, they suffer from a number of limitations.

The first limitation is the lack of principled extensibility and controllability. If the QA system is to be extended to cover a particular type of question that has not been supported so far, current systems would require to add a number of additional training questions and to retrain the system with an uncertain outcome as there is no guarantee that the additional training questions are sufficient for the approach to learn the new question type. A more principled approach here would support adding a new syntactic construction to the system that will parse or generate all questions involving this new syntactic construction rather than only single instances of such questions. The same holds for an extension of a system with a new lexical element.

Adding new training questions involving the lexical element does not guarantee that the system is able to induce the semantics of that lexical element as a basis to parse all the questions involving this element. In this sense, current systems can not be extended to cover new types of questions in a principled way.

The second limitation is related to the difficulty of increasing the capability of these systems for a new language. The capability of the systems that follow an inductive paradigm can be increased for a new language either by adding training data for that language (which is costly to obtain) or by training systems in a cross-lingual fashion, for example, transferring a system relying on cross-lingual embeddings (Jebbara & Cimiano, 2019) or from one language to another via training data projection (Klinger & Cimiano, 2015) using machine translation systems. However, it has been demonstrated that such systems require at least some domain-specific data, in this case, in-language data, to perform well. For example, Klinger and Cimiano (2014) propose an approach based on projecting training data from one language to another based on machine translation techniques (Klinger & Cimiano, 2015). They show that, given that the translation can introduce errors, the performance of the system on the target language depends on the quality of the translation system and the induced word alignment.

Another approach to multilingual question answering is based on the cross-lingual language model (Conneau et al., 2020; Conneau & Lample, 2019) that partially overcomes the lack of a large amount of training data. The approach creates machine-learning models for languages by transferring knowledge from one natural language to another. For example, XLM-Roberta (Conneau et al., 2020) uses self-supervised training techniques to achieve state-of-the-art performance in cross-lingual understanding, in which a model is trained in one language and then used with other languages without additional training data. The approach is deployed on comprehension datasets (i.e., SQuAD benchmark,¹) consisting of questions posed by crowd workers on a collection of Wikipedia articles. To the best of our knowledge, the approach has not been applied to linked data (Berners-Lee, 2006; Bizer et al., 2023), so it remains unclear how far such an approach addresses the challenges of question answering over a knowledge graph (Shekarpour et al., 2016; Unger et al., 2014) and what its performance would be.

We explore a different paradigm to question answering that relies on a model-based approach (Benz et al., 2020). The approach relies on a model of the lexicon-ontology interface (Cimiano et al., 2016) in which the meaning of lexical entries is specified with respect to a given vocabulary or ontology. We build on the OntoLex-Lemon model² (Cimiano et al., 2016) in particular. Given a lexicon in a particular language and for a particular vocabulary/ontology, our approach automatically generates a grammar that allows parsing questions in the specific language into SPARQL queries over the corresponding vocabulary/ontology. This approach is model-based or declarative in the sense that it relies on a declarative specification of the meaning of lexical entries. The approach can be extended in a principled and efficient way as follows:

–
Adding new lexical items: The approach to extending the QA system is principled and consistent in the sense that adding a lexical item to the lexicon guarantees that all questions involving valid constructions in which the lexical item can be used will be parsed correctly.
–
Adding a new syntactic construction: when making available a new syntactic construction to the grammar generation component, then this syntactic construction will be available for the grammars generated for different datasets, so the syntactic construction needs to be added once for a language and it will be available for all datasets.
–
Porting the system to a new language: porting a system to a new language requires creating a lexicon and developing sentence templates for that particular language. This is an effort done once for all the datasets that the QA system will work on.

The way to extend the system is thus principled in the sense that porting the system to a new dataset merely requires providing a new lexicon in the languages the system is expected to support. This can be done by engineers who do not need to be aware of how the grammar is generated but only need to understand how to add appropriate lexical entries. The extension with new syntactic constructions and languages is done once by a language expert in the QA system and will be available for all instances of the QA system that would only need to re-run the grammar generation to incorporate the newly added syntactic constructions.

While we have presented preliminary versions of this work (Benz et al., 2020) before, we have so far not provided evidence of the robustness, controllability, scalability, adaptability, and portability of the system across languages in a systematic evaluation. As one important ingredient towards making information accessible across languages, we investigate how our model-based approach can be extended in terms of coverage in a principled way.

In the current paper, we describe our new approach in full detail and describe its architecture. We further demonstrated that the approach is fit for purpose and outperforms the existing state-of-the-art machine learning-based and rule-based approaches in an incremental evaluation for English on five editions of the QALD dataset, that is, QALD-9, QALD-7, QALD-6, QALD-5, and QALD-3. We provide a proof for the portability of our system to further languages by describing how we adapted it to German, Spanish, and Italian and also show that it provides state-of-the-art results on all series of QALD including the newest version QALD-9 for these three languages. We also demonstrate that our approach outperforms pre-trained state-of-the-art Large Language Models (LLMs), particularly ChatGPT (Faria et al., 2023; Ouyang et al., 2022), in an incremental evaluation, both in zero-shot and few-shot settings.
2. General Approach

In this section, we describe our approach to multilingual grammar generation on the basis of a lexicon represented using the OntoLex-Lemon model. For each syntactic frame supported by our system, we describe generic templates (i.e., grammar rule templates [GRTs]) for the grammar rules generated and provide examples of how these templates are instantiated for specific lexical entries. Our examples use DBpedia (Lehmann et al., 2015) as a reference dataset. However, our approach is dataset-independent and can be adapted to any particular dataset and vocabulary by providing a corresponding lexicon.

2.1. OntoLex-Lemon

In this section, we explain the OntoLex-Lemon model (Cimiano et al., 2016), which is an updated version of the Lemon model (McCrae et al., 2011), and which is the core representation used by our model-based approach to grammar generation. The OntoLex-Lemon model is based around the core module, as depicted in Figure 1. The primary element of this is the lexical entry, which represents a single word and thus collects together all morphological variants of that word, which correspond to forms in the model, and all possible concepts in the given vocabulary/ontology it can refer to. The meaning of a word is thus given by reference to an ontological concept; this connection is modelled via a lexical sense that represents the mapping from a word to an ontological concept.

Figure 1.

The core OntoLex-Lemon model (Cimiano et al., 2016).

Words have a clear syntactic behavior which to a great extent is determined by their syntactic category (verb, noun, adjective, etc.). The way that sentences are composed in a language crucially depends on the syntactic behaviour of the words that make up the sentence. The syntax and semantics module of the OntoLex-Lemon model describes the syntactic behaviour of words in terms of an inventory of syntactic (subcategorization) frames such as transitive verb-frame, intransitive verb-frame requiring a prepositional object, etc. In addition, it describes how the syntactic structure relates to a semantic structure described in terms of vocabulary elements of the given vocabulary/ontology. There are other existing sources that comprise of a repository of frames. A relevant example is the Framester hub (Gangemi et al., 2016) which acts as a hub between FrameNet (Baker et al., 1998), WordNet (Fellbaum, 1998), VerbNet (Kipper et al., 2008), BabelNet,³ DBpedia, Yago, DOLCE-Zero, as well as other resources, while encoding frame-related and lexical knowledge in OWL to support expressive querying and reasoning. In contrast to such resources, the frames we consider in this article are mainly syntactically defined, getting semantics from a relation to a specific vocabulary or ontology.

The core of the lemon model is insufficient for grammar generation to parse natural language sentences into SPARQL queries. This is because it lacks syntactic information and arguments, such as the specific types of arguments needed for certain lexical entries (e.g., direct and indirect objects), their grammatical functions, or the type of a frame (e.g., intransitive, transitive). Lexinfo (Cimiano et al., 2011) is an extension of the lemon model, featuring over 600 distinct linguistic categories. This declarative model allows us to associate linguistic information with elements in an ontology with respect to the syntactic frame (as detailed in Section 2.2), which is crucial for constructing grammatically correct sentences for that element. For example, Figure 2 shows a syntactic frame of the so-called InTransitivePPFrame, which consists of an intransitive verb and a prepositional adjunct.

Figure 2.

The syntax/semantics mapping in LexInfo (Cimiano et al., 2011).

Consider the intransitive verb with prepositional adjunct ‘settle in’ construction: ‘subject settled in object’ and the meaning of which can be captured by the DBpedia property dbp:settled, where the subject is an instance of the domain of the property and the object is an instance of the range of the property. In the general case, however, it will not always be the case that the syntactic subject maps to the domain and the syntactic object to the range of a property. Consider the transitive verb ‘(to) write’ with a construction: ‘subject wrote object.’ In this case, ‘wrote’ corresponds to the property dbp:author, the subject corresponds to the range, and the object corresponds to the domain of the property. The mapping between syntactic and semantic arguments needs thus to be specified explicitly for each syntactic construction. This can be done via the syntax and semantics module of the OntoLex-Lemon model. This is schematically depicted in Figure 2.

2.2. Overall Architecture

Figure 3 shows a high-level architecture of our model-based approach. The core component of the approach is the grammar generator, which takes a Lemon lexicon and a set of sentence templates (detailed in Section 3.1) as input, accesses an RDF dataset and automatically creates a lexicalized grammar. The grammar is lexicalized in the sense that every grammar rule has a lexical anchor as in Lexicalized Tree Adjoining Grammars (LTAG) (Joshi & Schabes, 1997; Schabes & Joshi, 1988). As in LTAG, our rules correspond to full projections of lexical anchors in that all (dependent) arguments are local to each rule.

Figure 3.

High-level architecture of the model-based approach.

A necessary prerequisite for our grammar generation approach is the availability of a Lemon lexicon that describes by which lexical entries the elements (classes, properties) of a given dataset can be verbalized in a particular language. In particular, a lexicon is needed for each language to be supported by the QA system. The grammar generation for a lexical entry with a specific syntactic frame is controlled by GRTs (i.e., generic templates) that describe how specific lexicalized grammar rules can be generated for a given lexical entry. We support the following syntactic frames⁴: –

NounPPFrame: corresponding to a relational noun that requires a prepositional object such as ‘mayor’ (of), ‘capital’ (of), etc.

–

TransitiveFrame: correspoding to transitive verbs such as (to) ‘direct’, (to) ‘marry’, etc.

–

InTransitivePPFrame: corresponding to intransitive verbs subcategorizing a prepositional phrase such as ‘flow’ (through),‘born’ (on), etc.

–

AdjectivePredicateFrame: covering intersective adjectives such as ‘Spanish’, ‘Afghan’, etc. This frame is used for both attributive and predicative use of the adjective.

–

AdjectiveSuperlativeFrame: covering gradable adjectives such as ‘high‘, ‘highest‘, etc.

In the following sections, we first describe the lexical entries (as detailed in Section 2.3) for each of these syntactic frames. Afterwards, we will explain the sentence templates (as detailed in Section 2.4), as they serve as a prerequisite for obtaining the GRTs. Finally, we describe in detail the GRTs (see Section 2.5) and give an instantiated example for a specific lexical entry.

2.3. Creating Lexical Entries

To create a QA system for a given dataset with its associated vocabulary and a specific language, the language expert of that language needs to create lexical entries. In this section, we provide examples of lexical entries for English as language and DBpedia as vocabulary/ontology.

2.3.1. Nouns

Consider the Lemon lexical entry shown in Figure 4 for the relational noun ‘mayor’ (of). The entry states that the canonical written form of the entry is ‘mayor’. It specifies that the syntactic behaviour of the entry corresponds to the one of a NounPPFrame in LexInfo. This means that the lexical entry can be used in a copulative construction ‘X is the mayor of Y’ with two arguments, where copulativeArg corresponds to the copula subject X and the prepositionalAdjunct corresponds to the prepositional object Y. The semantics of the relational noun is captured with respect to the property dbo:mayor,⁵ where the subject of the property is realized by the prepositionalAdjunct and the object of the property is realized by the copulativeArg. The entry also captures that the domain (i.e., the class specified via the property rdfs:domain) of the property is dbo:City in the context of the lexical entry. Correspondingly, the range (i.e., the class that is specified via the property rdfs:range) of the property is dbo:Person.

Figure 4.

The lexical entry for the relational noun ‘mayor’ (of).

2.3.2. Verbs

Consider the Lemon lexical entry shown in Figure 5 for the transitive verb (to) ‘direct.’ The lexical entry states that the canonical form has the written representation ‘direct’. The simple past form is ‘directed’. The semantics of the verb (to) ‘direct’ is expressed by the property dbo:director. The entry specifies that the subject of the property is realized by the direct object of the verb ‘direct’ while the object of the property is realized by the syntactic subject of the verb ‘direct’. The entry also captures that the domain of the property is dbo:Film and the range of the property is dbo:Person.

Figure 5.

The lexical entry for the transitive verb (to) ‘direct’.

An example of an intransitive verb with a prepositional adjunct, ‘flow’ (through), is shown in Figure 6. According to the Lemon entry, the verb has a subject and a prepositional adjunct. The semantics of ‘X flows through Y’ is captured by the property dbo:city, where the subject of the property is realized by the syntactic subject of the ‘flow’ (through) verb, and the object of the property is realized by the prepositional adjunct of the verbal construction. The entry states that the domain of the property is dbo:River and the range of the property is dbo:City.

Figure 6.

The lexical entry for the intransitive verb ‘flow’ (through).

2.3.3. Adjectives

Adjectives have different behaviour in terms of grammar rules generated compared to the entries discussed before. Consider the example entry for the adjective ‘Spanish’ shown in Figure 7. The entry states that the adjective can be used in an attributive frame (i.e., ‘Spanish movie’) as well as in a predicative frame (i.e., ‘the movie is Spanish’). The entry states that the semantics of the adjective ‘Spanish’ can be expressed through a restriction on property dbo:country for the value res:Spain.

Figure 7.

The lexical entry for the adjective ‘Spanish’.

Gradable adjectives are represented using the class AdjectiveSuperlativeFrame in LexInfo and are modelled along the proposal of McCrae et al. (2014) using the lemonOILS vocabulary⁶ (the Lemon Ontology for the Interpretation of Lexical Semantics). The entry ‘high’, shown in Figure 8, is expressed as an instance of the class oils:CovariantScalar, indicating that the adjective is covariant with its bound property dbo:elevation for the degree oils:strong, that is, the ‘higher’ something is, the larger the value of the property dbo:elevation.

Figure 8.

The lexical entry for the gradable adjective ‘high’.

2.4. Creation of Sentence Templates

The sentence templates are manually created by language experts, as indicated in the high-level diagram as the second step. A sentence template defines the structure of a sentence for a specific language, including the order and type of its components, such as the subject, various types of verbs (i.e., main verbs, auxiliary verbs, imperative verbs, etc.), prepositions, objects, and modifiers.

We create sentence templates for each syntactic frame (detailed in Section 3.1.2), not for individual lexical entries. For example, for relative nouns (e.g., NounPPFrame) for English, the language expert is instructed to create sentence templates given sample of a relative noun (e.g., ‘mayor of’) and a dictionary with syntactic categories (i.e., InterrogativeDeterminer $\to$ {‘which’}, InterrogativePronounForPerson $\to$ {‘who’}, InterrogativePronounForThing $\to$ {‘what’}, AuxVerb(be:present:singular) $\to$ {‘is’},…etc.). The dictionary contains a list of words for each syntactic category. For example, in the dictionary, InterrogativeDeterminer is a syntactic category that refers to the word ‘which’. The detailed method of creating sentence templates is discussed in Section 3.1.2. Below are examples of sentence templates for relative nouns created by the language expert for English.

Here, the sentence template InterrogativePronounForPerson AuxVerb(be:present:singular) Determiner(the) RelativeNounSingular Preposition PrepositionalAdjunct? presents the structure for the question ‘Who is the mayor of Paris?’. These five sentence templates describe the structure for questions containing a relative noun. Therefore, these templates (i.e., ST $_{q 1}$ , ST $_{q2}$ , ST $_{q3}$ , ST $_{q4}$ , and ST $_{q2}$ ) are grouped under ST $_{subject}$ , as they all aim to represent questions seeking information about the subject of the sentence.

2.5. Generating Grammars

In our previous work, we described how grammar generation (Benz et al., 2020) works for four syntactic frames according to LexInfo (Cimiano et al., 2011). In our current work, we comprehensively describe the grammar generation process that builds on and substitutes the previous work and describe the methodology of porting it to other languages and datasets.

This section explores the transformation process of sentence templates (as discussed in the previous section) into GRTs. In this article, we use ST as the abbreviation for the sentence template, GRT as the abbreviation for the grammar rule template, and GR for the grammar rule.

When a sentence template (ST) is transformed into a grammar rule template (GRT), three conceptually different types of replacements take place, each serving a distinct purpose. For example, the sentence template ST $_{q1}$ : InterrogativePronounForPerson AuxVerb(be:present:singular) Determiner(the) RelativeNounSingular Preposition PrepositionalAdjunct? presents the structure for the question $q 1$ : ‘Who is the mayor of Paris?’. The template is converted to a GRT by following three steps:

Transformations related to the lexicon: During the process of transforming an ST into a GRT, the parts obtained from the lexical entry are replaced to generalize the resulting template and abstract away from the concrete lexical entry. For example, in the sentence template ST $_{q1}$ , the syntactic categories RelativeNounSingular and Preposition refer to ‘mayor’ and ‘of’, respectively. By wrapping these categories in square brackets (i.e., [NounSingular] and [Preposition]), these symbols can easily be distinguished from other symbols later.

Transformations related to the knowledge base: The sentence template outlines the structure of a sentence. Some syntactic categories in the sentence template represent the syntactic subject or object. In our approach, these are obtained from RDF data (i.e., subject or object of the property). Therefore, when transforming an ST into a GRT, those parts are replaced by non-terminal symbols enclosed in $<>$ along with a mapping function (i.e., SyntacticFunction). The SyntacticFunction is mapped to either the subject or object of the property. For example, in the sentence template ST $_{q1}$ , the syntactic category PrepositionalAdjunct refers to ‘Paris’, which is an entity that can be retrieved from RDF data. Therefore, when transforming the ST into a GRT, PrepositionalAdjunct is replaced by non-terminal (i.e., <NP $_{map(SyntacticFunction), Property}$ >). When instantiating the GRT with a lexical entry (detailed in Section 2.5.1), SyntacticFunction is mapped to either Domain or Range.

Transformations related to language-specific parts: The language-specific components, such as determiners, pronouns, auxiliary verbs, conjunctions, and negation, neither fall under pre-terminals retrieved from the lexicon nor non-terminals retrieved from RDF data. For example, the syntactic category InterrogativePronounForPerson refers to ‘who’, which is a language-specific component independent of the lexicon and RDF data. The purpose of this step is to transform the pre-GRT into an actual GRT that incorporates lexical elements for language-specific components.

The step-by-step process of transformation from a sentence template (ST) via pre-GRTs to a GRT is shown below:

In the following section, we describe the grammar at two levels of abstraction: (i) we describe generic templates (i.e., GRTs) that are independent of a particular lexical entry, and (ii) we provide example instantiations (i.e., grammar rules) of the GRTs for illustration. We also provide the SPARQL queries to retrieve the value of non-terminals of the grammar rules. The SPARQL queries of the individual grammar rules are provided in the Appendix.

2.5.1. NounPPFrame

From the sentence templates of the relative nouns (i.e., syntactic behavior called NounPPFrame in LexInfo), our approach obtains ten GRTs presented into five groups (such as GRT $_{subject}$ , GRT $_{object}$ , GRT $_{amount}$ , GRT $_{ask}$ , GRT $_{nounPhrase}$ ). For example, the GRT $_{subject}$ presents the GRTs obtained by transforming sentence templates of group ST $_{subject}$ (as detailed in Section 2.4). The ST $_{subject}$ presents the structure for questions seeking information about the subject position.

As discussed in the previous section, symbols enclosed in <> represent non-terminals. The non-terminal <NP $_{map(SyntacticFunction), Property}$ > can be substituted by each label of a URI that appears as the subject (resp. object) of the property Property depending on whether the given syntactic function (Copulative Subject, Prepositional Adjunct, etc.) maps to the domain resp. range of the property. Hereby, map(SyntacticFunction) specifies the semantic argument of the property (domain or range) that the given syntactic function maps to.

Symbols enclosed in [] represent proto-terminals that are to be replaced with content specific to the lexical entry, for example, [NounSingular] and [NounPlural] refer to the inflectional forms of the lexical entry. [Preposition] refers to the preposition of the lexical entry.

Here, the first group (i.e., GRT $_{subject}$ ) consists of five grammar rule templates, the second group (i.e., GRT $_{object}$ ) consists of two GRTs, the third group (i.e., GRT $_{amount}$ ) consists of one grammar rule template, the fourth group (i.e., GRT $_{ask}$ ) consists of two GRTs, and finally, GRT $_{nounPhrase}$ consists of one grammar rule template.

The <NP $_{map(SyntacticFunction), Property}$ > is instantiated either to <NP $_{Domain, Property}$ > or <NP $_{Range, Property}$ >, depending on whether map(SyntacticFunction) is Domain or Range. Similarly, the <NP $_{Class, < map(SyntacticFunction),Property >}$ > is instantiated either to <NP $_{Class, Domain,Property >}$ > or <NP $_{Class, Range,Property >}$ >. Therefore, the GRTs with the non-terminal symbols <NP $_{map(SyntacticFunction), Property}$ > on the right are pre-terminal rules and are generated by retrieving the corresponding labels of URIs using the following SPARQL queries:

Here, eval(query) returns a list of the bindings of the query separated by the ’|’ symbol that separates grammar rules for the same pre-terminal symbol on the left.

Consider a particular lexical entry shown in Figure 4 for the relational noun ‘mayor’ (of). [NounSingular] refers to the singular form of the noun (i.e., ‘mayor’) and [NounPlural] refers to the plural form of the noun (i.e.,‘mayors’). The instantiation of the above GRTs for the lexical entry ‘mayor’ (of) is shown below:

The values of the non-terminals <NP $_{Domain, dbo:mayor}$ > and <NP $_{Range, dbo:mayor}$ > are retrieved with the following automatically generated SPARQL queries:

After executing the SPARQL query, we obtain the following results:

The first, second, third, and fourth groups (GRT $_{subject}$ , GRT $_{object}$ , GRT $_{amount}$ , and GRT $_{ask}$ ) contain grammar rule templates of type SENTENCE; that is, they generate a full question. The grammar rules generated by instantiating GRTs of the first group (i.e., GRT $_{subject}$ ) using the lexical entry ‘mayor’ (of) can parse a wide range of questions. For example, (1) ‘Who is the mayor of Paris?’, (2) ‘Who was the mayor of Paris?’, (3) ‘Which person is the mayor of Paris?’, (4) ‘Which person was the mayor of Paris?’, (5) ‘Give me the mayor of Paris.’, (6) ‘Give me all mayors of Paris.’, (7) ‘List all mayors of Paris.’.

The grammar rules generated from the second group (i.e., GRT $_{object}$ ) can parse the questions such as (1) ‘Anne Hidalgo is the mayor of which city?’, (2) ‘Anne Hidalgo was the mayor of which city?’, etc.

The grammar rules generated from the third group (i.e., GRT $_{amount}$ ) can parse the questions such as (1) ‘How many mayors does Paris have?’, etc. The grammar rule from the fourth template (i.e., GRT $_{ask}$ ) can parse boolean questions (i.e., yes–no questions), for example, ‘Is Anne Hidalgo the mayor of Paris?’. The grammar rules from the fifth template (i.e., Grammar Rules from GRT-5) are of type NounPhrase, for example, ‘the mayor of Paris’.

We have presented the GRTs for the relative noun of Agent⁷ type, such as ‘mayor’ (of), ‘president’ (of), etc. However, the GRTs and grammar rules for the relative noun of non-Agent⁸ type, such as ‘capital’ (of), ‘ingredient’ (of), are not detailed here.

2.5.2. TransitiveFrame

From the sentence templates of a transitive verb (i.e., the syntactic behaviour called TransitiveFrame in LexInfo), our approach obtains $15$ GRTs organized into five groups (such as GRT $_{subject}$ , GRT $_{object}$ , GRT $_{amount}$ , GRT $_{ask}$ , GRT $_{nounPhrase}$ ).

The non-terminal <NP $_{map(SyntacticFunction), Property}$ > can be substituted by all labels of URIs that appear as the subject (resp. object) of the property Property depending on whether the given syntactic function (Subject, Object, etc.) maps to the domain resp. range of the property. The lexicalized grammar rules templates are shown below.

Consider a particular lexical entry shown in Figure 5 for the transitive verb (to) ‘direct’. [VerbPresent] and [VerbPast] refer to the inflectional forms of the lexical entry. [VerbPresent] refers to the present form of the verb, for example, ‘direct’, and [VerbPast] refers to the plural form of the verb, for example, ‘directed’.

The grammar rules generated by instantiating grammar rule templates of the first group (i.e., GRT $_{subject}$ ) using the lexical entry (to) ‘direct’ can parse a wide range of questions. For example, (1) ‘Who directs Wait for Your Laugh?’, (2) ‘Who directed Wait for Your Laugh?’, (3) ‘Which person directs Wait for Your Laugh?’, (4) ‘Which person directed Wait for Your Laugh?’, etc. The grammar rules generated from GRT $_{object}$ can parse questions such as (1) ‘What is directed by Mario Monicelli?’, (2) ‘What was directed by Mario Monicelli?’, (3) ‘What were directed by Mario Monicelli?’, (4) ‘Which film is directed by Mario Monicelli?’, (5) ‘Which films are directed by Mario Monicelli?’, (6) ‘Which film was directed by Mario Monicelli?’, (7) ‘Which films were directed by Mario Monicelli?’, (8) ‘Give me all films directed by Mario Monicelli?’, etc.

The grammar rules generated from GRT $_{amount}$ can parse questions such as (1) ‘How many films are directed by Mario Monicelli?’, (2) ‘How often did Mario Monicelli direct?’, etc. The grammar rule from GRT $_{ask}$ can parse boolean questions, for example, 1) ‘Did Mario Monicelli direct Wait for Your Laugh?’, 2) ‘Does Mario Monicelli direct Wait for Your Laugh?’, etc. The grammar rule from GRT $_{nounPhrase}$ can parse the noun phrases such as: (1) ‘film directed by Mario Monicelli’, (2) ‘films directed by Mario Monicelli’, etc.

2.5.3. IntransitivePPFrame

From the sentence templates of an intransitive verb (i.e., the syntactic behaviour called IntransitivePPFrame in LexInfo), our approach obtains $15$ grammar rule templates organized into four groups (such as GRT $_{subject}$ , GRT $_{object}$ , GRT $_{amount}$ , and GRT $_{ask}$ ).

The non-terminal <NP $_{map(SyntacticFunction), Property}$ > can be substituted by all labels of URIs that appear as the subject (resp. object) of the property Property depending on whether the given syntactic function (Subject, PrepositionalAdjunct, etc.) maps to the domain resp. range of the property.

Consider a particular lexical entry shown in Figure 6 for the intransitive verb ‘flow through’ with a prepositional adjunct. [VerbPresent] refers to the present form of the verb (i.e., ‘flow’ or ‘flows’). The GRTs are instantiated as follows for the lexical entry ‘(to) flow through)’.

The grammar rules generated by instantiating grammar rule templates of the first group (i.e., GRT $_{subject}$ ) using the lexical entry ‘flow through’ can parse a wide range of questions. For example, ( 1) ‘What flows through Chennai?’, (2) ‘Which river flows through Chennai?’, (3) ‘Which rivers flow through Chennai? ’, etc. The grammar rules generated from GRT $_{amount}$ allow us to parse the questions such as 1) ‘How many rivers flow through Chennai?’, etc. The grammar rules generated from GRT $_{object}$ can parse the questions such as (1) ‘What does Adyar River flow through?’, (2) ‘Which cities does Adyar River flow through?’, (3) ‘Which city does Adyar River flow through?’, (4) ‘Through which cities does Adyar River river flow?’, etc. The grammar rules generated from GRT $_{ask}$ can parse the boolean questions such as (1) ‘Does Adyar River flow through Chennai?’, (2) ‘Did Adyar River flow through Chennai?’, etc.

The GRTs for temporal expressions involving intransitive verbs, such as ‘dissolve’ (at), or when the subject is of Agent type, as in ‘born’ (in), are not presented here.

2.5.4. AdjectivePredicateFrame

After transforming the sentence templates of intersective adjectives (i.e., the syntactic behaviour referred to as AdjectivePredicateFrame in LexInfo), our approach yields six GRTs organized into two groups (namely, GRT $_{subject}$ and GRT $_{nounPhrase}$ ).

The values of the non-terminals <Noun $_{Class, RestrictionClass(Property, Object)}$ > are automatically generated via the SPARQL query given below.

Consider a particular lexical entry shown in Figure 7 for the adjective ‘Spanish’. [AdjectiveProper] refers to the proper form of the adjective (i.e., ‘Spanish’). The nouns that adjectives modify in the rules come from the classes in the ontology that have instances that are related to res:Spain via the property dbo:country.

The SPARQL query used to retrieve <Noun $_{Class, RestrictionClass(dbo:country, res:Spain)}$ > is given below.

Once the query is executed, a rule like this is generated:

The grammar rules generated by instantiating grammar rule templates of the first group (i.e., GRT $_{subject}$ ) using the lexical entry ‘Spanish’ allow us to parse questions such as (1) ‘Which is a Spanish Athlete?’, (2) ‘Which are Spanish Athletes?’, (3) ‘Give me all Spanish Athletes.’, (4) ‘List all Spanish Athletes.’, etc. The grammar rules generated from GRT $_{nounPhrase}$ can parse questions such as ‘a Spanish Athlete’,‘Spanish Athletes’, etc.

2.5.5. AdjectiveSuperlativeFrame

From the sentence templates of gradable adjectives (i.e., the syntactic behaviour called AdjectiveSuperlativeFrame in LexInfo), our approach obtains six GRTs organized into three groups (such as GRT $_{baseForm}$ , GRT $_{comparative}$ , and GRT $_{superlative}$ ).

Consider the lexical entry shown in Figure 8 for the gradable adjective ‘high’. [AdjectivePositive] refers to the positive form of the adjective, for example, ‘high’. [AdjectiveSuperlative] refers to the superlative form of the adjective, for example, ‘highest’. The lexicalized grammar rules are shown below.

The values of the non-terminals <NP $_{Domain, dbo:elevation}$ > are retrieved with the following automatically generated SPARQL query:

The grammar rules generated by instantiating grammar rule templates of the first group (i.e., GRT $_{baseForm}$ ) using the lexical entry ‘high’ allow us to parse questions such as ‘How high is Zugspitze?’. The grammar rules generated from GRT $_{comparative}$ can parse questions such as ‘Which mountain is higher than the Zugspitze?’, ‘Which mountains are higher than the Zugspitze?’, etc. The grammar rule generated from GRT $_{superlative}$ can parse questions such as ‘What is the highest mountain?’.

2.6. Parsing

In this section, we provide a complete example of parsing a given question (i.e., ‘Who is the mayor of Russia?’) into a SPARQL query using the grammar discussed in the previous section. After that, we explain how the approach parses a complex question (i.e., ‘Who is the mayor of the capital of Russia?’).

For parsing, the lexicalized grammar rules (as detailed in Section 2.5.1) are automatically converted into regular grammar⁹ (Simov et al., 2002; Simov & Osenova, 2004) by replacing the non-terminals with regular expressions (i.e., $([A - Z a - z])$ ). For example, the grammar rule ‘Who is the mayor of <NP $_{Domain, dbo:mayor}$ >?’ is transformed into ‘Who is the mayor of $([A - Z a - z])$ ?’. The regular grammar is shown below:

There are several advantages to using a regular grammar. First, regular grammar is more straightforward compared to state-of-the-art grammar formalisms, such as context-free grammar (Jurafsky & Martin, 2025) or tree-adjoining grammar (Joshi & Schabes, 1997; Lichte & Kallmeyer, 2017). This simplicity can be advantageous in scenarios where a less complex grammar is sufficient for tasks such as question answering over linked data, as is the case in our approach. Second, regular languages can be parsed using finite automata, which are generally more computationally efficient than the more complex mechanisms employed in top-down parsing (Jurafsky & Martin, 2025) for context-free grammar or the substitution and adjunction operations (Vijay-Shanker, 1992) of tree-adjoining grammar.

2.7. Simple Question

The parser takes a sentence in natural language and produces an analysis of the sentence in the form of one or more SPARQL queries. In our example, the input and output of the parser are shown below:

The step-by-step method is as follows:

Find the grammar rule: Given a question in natural language, the parser iterates through all the grammar rules and retrieves the grammar rule when a match is found.

If a match is found, it returns the grammar rule that includes two SPARQL queries (as detailed in Section 2.5.1). Once a match is found, proceed to steps 2 and 3.

For example, the grammar rule matching the question ‘Who is the mayor of Moscow?’ is shown below.

As explained in Section 2.5.1, the values of non-terminal in a grammar rule is extracted using SPARQL query. In regular grammar, non-terminals are replaced by regular expressions, so the values of the regex $([A - Z a - z] *)$ is extracted using the SPARQL query, resulting in a set of strings. The SPARQL query (i.e., SPARQL $_{rule}$ ) of the grammar rule contains a variable (i.e., $A r g$ ) that needs to be replaced with a resource.

If there is no grammar rule that matches the given question, it cannot be parsed.

Extract the entity of the question: After the question ‘Who is the mayor of Moscow?’ is matched with the grammar rule ‘Who is the mayor of $([A - Z a - z] *)$ ?’, the entity of the question is extracted; in this case, ‘Moscow’.

Match the entity with terminals: To match the entity ‘Moscow’ (of the question) with the terminals (of the grammar rule), the parser uses the Jaccard similarity¹⁰ measure with a threshold set to $0.5$ . The matching is done as follows:

Each terminal of step 1a (e.g., ‘Provisional Government of Belgium’) is converted to a set of tokens.

Afterwards, the Jaccard similarity is computed between each terminal and the entity of the question, in this case, ‘Moscow’. The similarity scores are ranked in descending order.

If the similarity score is greater than the threshold (i.e., $0.5$ ), then the top match is selected and the corresponding variable (i.e., $A r g$ ) of the SPARQL query (i.e., SPARQL $_{rule}$ ) is replaced. Otherwise, the question is not parsed. In this case, the score is greater than the threshold and so the parse result is as follows:

2.8. Complex Questions

The grammar rules discussed in the Section 2.5 allow parsing questions that can be represented by a SPARQL query that involves a triple pattern with the predicate rdf:type, a triple pattern with the predicate rdfs:label and one more triple pattern. The approach can thus not parse complex questions involving more triples such as the following one:

In order to allow our QA system to parse such questions correctly, we implemented a limited form of composition that allows certain rules to be nested within others. Specifically, our approach supports the nesting of noun phrases (e.g., ‘the capital of Russia’ or ‘Russian capital’) into the rules of other frames.

The parser takes a sentence ‘Who is the mayor of the capital of Russia?’ and produces an analysis of the sentence in the form of one or more SPARQL queries. In our approach, the input and output of the parser is shown below:

The step-by-step method is as follows:

The question ’Who is the mayor of the capital of Russia?’ matches with the following grammar rule:

The SPARQL query of the grammar rule contains a variable (i.e., $A r g$ ) that needs to be replaced with a resource.

Extract the entity of the question: After the question ‘Who is the mayor of the capital of Russia?’ is matched with the grammar rule ‘Who is the mayor of $([A - Z a - z] *)$ ?’, the entity of the question is extracted; in this case, ‘the capital of Russia’.

Parse simple question: In this case, the parser treats ‘the capital of Russia?’ as a noun phrase and parses it similarly to the simple question discussed in the previous section. The grammar rule matched is shown below:

The parse result of ‘the capital of Russia?’ is

The answer retrieved for this noun phrase is res:Moscow. Therefore, the semantics of the expression ‘the capital of Russia’ can alternatively be expressed by res:Moscow.

Replace the variable: The variable (i.e., $A r g$ ) of SPARQL query (step 1) is replaced by res:Moscow. The parse result of the complex question ‘Who is the mayor of the capital of Russia?’ is as follows:

This simple approach to dealing with complex questions implements a limited form of compositionality that treats the phrase ‘the capital of Russia’ as (non-compositionally) referring to the URI res:Moscow. While this way of parsing certain types of complex questions is effective as shown by our experimental results, it is limited to two-hop queries involving at most a linear chain of two properties in addition to rdf:type (or rdfs:label) statement.

3. Multilingual Grammar Generation

In this section, we detail our model-based approach that automatically generates lexicalized grammar rules from a Lemon lexicon. First, we explain the method for English. After that, we describe the methodology we followed to adapt the system to multiple languages—in our case, German, Spanish, and Italian. We elaborate on the process of implementing corresponding GRTs (as discussed in the previous section) for each of these languages. These GRTs are independent of a particular dataset. Subsequently, the grammar rules are generated through instantiations of the templates using lexical entries. Further, we explain the process of creating lexical entries for these four languages using DBpedia as proof of concept.

3.1. English Grammar Generation

As described in the high-level architecture (as detailed in Section 2.5), the first step is to create lexical entries manually.

3.1.1. Creating Lexical Entries

To create a lexical entry of a relative noun, such as the one shown in Figure 4, the English language expert needs to provide the following information: (i) the singular and plural form of the reference, that is, ‘mayor’ and ‘mayors’ for dbo:mayor; and (ii) the preposition, that is, ‘of’.

3.1.2. Creating Sentence Templates

The sentence templates are manually created by language experts, as indicated in the high-level diagram as the second step. In Section 2.4, we define sentence templates with examples. In this section, we detail the steps necessary to create sentence templates.

We create sentence templates for each syntactic frame (detailed in Section 2.2). For example, for a relative noun (i.e., the syntactic frame called NounPPFrame as detailed in Section 2.5.1), we provide an example of a relative noun of the Agent type, such as ‘mayor of’. An Agent noun refers to a person or entity that performs or is associated with a particular action. The language expert created sentence templates for this sample relative noun. After that, we provided another sample of a relative noun of the non-Agent type, such as ‘capital of’. A non-Agent noun refers to something that is not related to or does not possess the characteristics of an agent. The language expert then created sentence templates for this type. No further samples of relative nouns were provided to the language expert. Note that creating sentence templates is different from creating lexical entries, as sentence templates are created for a syntactic frame only and are independent of lexical entries.

The following provides a detailed method for creating sentence templates for the Agent type. The language expert is provided with the following information:

a sample of relative noun of Agent type $w$ (e.g., $w$ =‘mayor of’)

five example questions $q_{s u b j e c t}$ , $q_{o b j e c t}$ , $q_{s u b j e c t - a m o u n t}$ , $q_{a s k}$ , $q_{n o u n P h r a s e}$ involving that relative noun. The examples are as follows:

The example $q_{s u b j e c t}$ illustrates a question containing a relative noun that seeks information about the subject of a sentence, while the example $q_{o b j e c t}$ provides a question seeking information about the object of a sentence. The question $q_{a m o u n t}$ is quantitative or numerical, and the response would require providing a specific numerical value. The question $q_{a s k}$ is a boolean question that typically elicits a binary response. The reason for presenting various types of questions is to address different variations of sentence templates containing relative nouns.

a set $C$ of syntactic categories (e.g., $C =$ {InterrogativeDeterminer, InterrogativePronounForPerson, InterrogativePronounForThing, RelativeNoun(singular), RelativeNoun(plural), Preposition, PrepositionalAdjunct, AuxVerb(be:present:singular),…, etc.})

with a dictionary $D_{C a t e g o r y}$ , that contains a list of words for each syntactic category in $C$ (e.g., $D_{C a t e g o r y}$ = {InterrogativeDeterminer $\to$ {‘what’}, InterrogativePronounForPerson $\to$ {‘who’}, AuxVerb(be:present:singular) $\to$ {‘is’},…, etc. }).

Note that there are two kinds of syntactic categories: (i) categories that take no parameters, such as InterrogativeDeterminer, PrepositionalAdjunct, etc., and (ii) categories that take one or more parameters, such as AuxVerb(be:present:singular), etc. Therefore, the language expert receives guidelines on which syntactic categories need to be parameterized. For example, the syntactic category AuxVerb needs to be parameterized with be:present:singular. Here, be indicates an auxiliary verb, present indicates the tense (in this case, the present tense), and singular specifies the grammatical number.

For the given example $q_{s u b j e c t}$ , the language expert created the following sentence template:

The language expert also created four variations for sentence templates for the given example, such as:

Similarly, the language expert created sentence templates for the examples of $q_{o b j e c t}$ , $q_{a m o u n t}$ , $q_{a s k}$ , $q_{n o u n P h r a s e}$ . These sentence templates are transformed into GRTs (as discussed in Section 2.5) and the GRTs are subsequently instantiated with lexical entries, resulting in lexicalized grammar rules (GR).

A total of $24$ sentence templates were manually created for the Agent type and $20$ sentence templates for the non-Agent type of relative noun, providing variations in sentence construction for the NounPPFrame.

Similarly, for the TransitiveFrame (as discussed in Section 2.5.2), two samples of monotransitive verbs are provided, where the verb takes only one direct object: (i) a sample (i.e., ‘to marry’) when both the subject and object in the sentence are of the Agent type (e.g., people, organizations, etc.), and (ii) a sample (i.e., ‘to produce’) where the subject is of the Agent type and the object is a thing (e.g., Skype, hovercrafts, etc.). Five examples (i.e., $q_{a c t i v e}$ , $q_{p a s s i v e}$ , $q_{a m o u n t}$ , $q_{a s k}$ , and $q_{n o u n P h r a s e}$ .) are provided for each of these samples. Here, $q_{a c t i v e}$ represents an example that contains the active form of a verb (e.g., ‘Who developed Skype?’).

For the InTransitivePPFrame (as discussed in Section 2.5.3), five samples of intransitive verbs are provided: (i) a sample (i.e., ‘flow through’) where the subject and the object in the prepositional adjunct in the sentence are things, (ii) a sample (i.e., ‘born in’) where the subject is of the Agent type and the object is a thing, (iii) a sample (i.e., ‘born in’) where the subject is of the Agent type and the object is a place, (iv) a sample (i.e., ‘founded in’) where the subject is of the Agent type and the object is a date, and (v) a sample (i.e., ‘start in’) where the subject and the object contain a place.

For the AdjectiveAttributiveFrame and AdjectiveSuperlativeFrame (as discussed in Section 2.5.4 and Section 2.5.5), two samples are provided: (i) a sample of an intersective adjective (i.e., ‘Spanish’), and (ii) a sample of a gradable adjective (i.e., ‘high’). Note that, in order to adapt to another language, we only need to write sentence templates for that language using the same syntactic categories, but the process of grammar generation remains the same.

As shown in Table 1, a total of $44$ GRTs for NounPPFrame, $101$ templates for TransitiveFrame, and $91$ templates for IntransitivePPFrame were created, resulting in a total of $249$ GRTs for the English language. After instantiating these GRTs with $599$ lexical entries of various syntactic frames, a total of $15, 047$ grammar rules were automatically generated. The manually created lexical entries, sentence templates, GRTs and grammar rules are available here¹¹ for all four languages: English, German, Italian, and Spanish. The English grammar is created from English DBpedia.¹²

Table 1.
An Overview of the Number of Manually Created Lexical Entries, Grammar Rule Templates, and Grammar Rules for Different Frame Types of LexInfo for the English Language.

# Lexical # Grammar Rule # Grammar

Frame Entry Templates (GRT) Rule (GR)

NounPPFrame $311$ $44$ $8, 708$

TransitiveFrame $96$ $101$ $3, 078$

InTransitivePPFrame $143$ $91$ $2, 183$

AdjectivePredicateFrame $28$ $5$ $910$

AdjectiveSuperlativeFrame $21$ $8$ $168$

Total $599$ $249$ $15, 047$

	# Lexical	# Grammar Rule	# Grammar
NounPPFrame	$311$	$44$	$8, 708$
TransitiveFrame	$96$	$101$	$3, 078$
InTransitivePPFrame	$143$	$91$	$2, 183$
AdjectivePredicateFrame	$28$	$5$	$910$
AdjectiveSuperlativeFrame	$21$	$8$	$168$
Total	$599$	$249$	$15, 047$

The table also shows the number of grammar rules that are generated automatically after instantiations of grammar rule templates with lexical entries.

Note that the lexicon presented here includes lexical entries that also cover the training data of QALD-9 benchmarks. The lexicon is evaluated with test data in two modes (as detailed in Section 5): (i) inductive evaluation, and (ii) incremental evaluation.

The time required for creating a lexical entry is approximately $5$ to $10$ minutes depending on the syntactic frame. The time required for creating a sentence template is approximately $10$ to $15$ minutes. However, the majority of sentence templates are similar to one of the templates with minor changes (i.e., inflectional forms, tense, etc.). The creation of these templates took approximately $1$ to $2$ minutes. The total time required to create all sentence templates for entries of type NounPPframe took approximately $1$ hour, sentence templates for entries of type TransitiveFrame took approximately $3$ hours, and sentence templates for entries of type IntransitivePPframe took approximately $5$ hours.

3.2. Porting to Other Languages

Porting the QA system to other languages is straightforward in our approach, which follows two steps: (i) adding new lexical items by providing language-specific properties, and (ii) adding grammar rules for that language. In this section, we detail the lexicon creation for other languages (e.g., German, Italian, and Spanish), focusing on language-specific features and how we accommodate those in our model-based approach.

3.2.1. German Grammar Generation

Unlike in English, German nouns are gender-specific (e.g., either masculine, feminine, or neuter) and the endings of the nouns vary with the corresponding case (e.g., nominative, accusative, dative, and genitive). To add a lexical entry of type NounPPFrame in German, the language expert needs to provide not only the gender but also the written form of the reference in singular and plural forms in all cases. Consider the relative noun ‘Bürgermeister’ (von) (i.e.,‘mayor’ (of)) in German. The semantics is captured by the property dbo:mayor (as shown in Figure 4) and the written forms of the noun are presented in the Lemon dictionary as follows:

The German verb has infinitive forms, conjugated forms depending on the subject, reflexive forms, and separable forms (i.e., verbs with separable prefixes). To add a lexical entry for a German intransitive verb, for example,‘aufhören’, which is captured by the property dbo:routeEnd, the following information needs to be provided: (i) the verb type (i.e., either conjugate or separable or reflexive), (ii) the infinite form, that is,‘aufhören’, (iii) the past form, that is,‘hörte auf’, and (iv) the perfect form, that is,‘hat aufgehört’.

We created a lexicon for the German language and the DBpedia dataset (Release 2016-10; core, links).¹³ As shown in Table 2, there are a total of $301$ GRTs for German. After instantiating these GRTs with $263$ lexical entries of various syntactic frames, a total of $15, 990$ grammar rules were automatically generated.

Table 2.
An Overview Over the Number of Manually Created Lexical Entries, Grammar Rule Templates, and Grammar Rules for Different Frame Types of LexInfo for the German Language.

# Lexical # Grammar Rule # Grammar

Frame Entry Templates (GRT) Rule (GR)

NounPPFrame $101$ $45$ $5, 102$

TransitiveFrame $63$ $113$ $5, 345$

InTransitivePPFrame $74$ $121$ $4, 972$

AdjectivePredicateFrame $15$ $15$ $501$

AdjectiveSuperlativeFrame $10$ $7$ $70$

Total $263$ $301$ $15, 990$

	# Lexical	# Grammar Rule	# Grammar
NounPPFrame	$101$	$45$	$5, 102$
TransitiveFrame	$63$	$113$	$5, 345$
InTransitivePPFrame	$74$	$121$	$4, 972$
AdjectivePredicateFrame	$15$	$15$	$501$
AdjectiveSuperlativeFrame	$10$	$7$	$70$
Total	$263$	$301$	$15, 990$

The table also shows the number of grammar rules that are generated automatically after instantiations of grammar rule templates with lexical entries.

Similar like English, the lexicon presented here includes lexical entries that also cover the training data of QALD-9 benchmarks.

The time required for creating a lexical entry is approximately $10$ to $15$ minutes depending on the syntactic frame. The time required for creating a sentence template is approximately $10$ to $25$ minutes. However, the majority of sentence templates are similar to one of the templates with minor changes in cases and genders. The creation of these templates took approximately $1$ to $2$ minutes. The total time required to create all sentence templates for entries of type NounPPframe took approximately $2$ hours, sentence templates for entries of type TransitiveFrame took approximately $4$ hours, and sentence templates for entries of type IntransitivePPframe took approximately $5$ hours.

Note that to adapt to German, we only added language-specific properties to the lexicon and adapted the sentence templates with respect to the features of German, but the process of grammar generation remains the same.

3.2.2. Italian Grammar Generation

Unlike German, the Italian noun does not depend on the cases but instead on the first letter of the noun they are preceding. To create a Lemon entry for a relative noun in Italian, the language expert needs to provide the gender information of the reference, domain, and range in both singular and plural forms. That is, the language expert needs to provide one of the four forms (i.e., ‘il’ or ‘la’ ‘l” or ‘l’) in the case of a singular noun.

The presence of auxiliary verbs, either ‘avere’ (‘have’) or ‘essere’ (‘be’), in compound tenses, needs to be explicitly specified. The generation of the specific interrogative pronoun is governed by the following rules as (i) ‘chi’ (‘who’) is invariable and refers only to people; (ii) ‘quale’ (‘which’) is used for the masculine and feminine singular, and ‘quali’ is used for masculine and feminine plural; (iii) ‘quanto’ and ‘quanti’ have feminine forms, etc. The use of different prepositions, either simple or articulated, is based on domain/range (e.g., ‘toponyms’ might require different prepositions). The presence of a determiner/articulated preposition based on domain/range semantics (e.g., ‘toponyms’ are preceded by a determiner when the noun refers to a country) also impacts the final choice in lexical elements. In adapting the grammar generation to Italian, we accommodate all these language-specific properties in the sentence template.

We created a lexicon for the Italian language and the DBpedia dataset (Release 2016-10; core, links).¹⁴ As shown in Table 3, there are a total of $266$ GRTs. After instantiating these GRTs with $441$ lexical entries of various syntactic frames, a total of $9, 061$ grammar rules were automatically generated.

Table 3.
An Overview Over the Number of Manually Created Lexical Entries, Grammar Rule Templates, and Sentence Templates for Different Frame Types of LexInfo for the Italian Language.

# Lexical # Grammar Rule # Grammar

Frame Entry Templates (GRT) Rule (GR)

NounPP $291$ $55$ $3, 820$

TransitiveFrame $35$ $89$ $3, 001$

InTransitivePPFrame $37$ $102$ $1, 593$

AdjectivePredicateFrame $74$ $13$ $6, 13$

AdjectiveSuperlativeFrame $4$ $7$ $34$

Total $441$ $266$ $9, 061$

	# Lexical	# Grammar Rule	# Grammar
NounPP	$291$	$55$	$3, 820$
TransitiveFrame	$35$	$89$	$3, 001$
InTransitivePPFrame	$37$	$102$	$1, 593$
AdjectivePredicateFrame	$74$	$13$	$6, 13$
AdjectiveSuperlativeFrame	$4$	$7$	$34$
Total	$441$	$266$	$9, 061$

The table also shows the number of grammar rules that are generated automatically after instantiations of grammar rule templates with lexical entries.

The time required for the creation of a lexical entry is approximately $10$ to $25$ minutes and the time required for creating a sentence template is $20$ to $30$ minutes. However, the majority of sentence templates are similar to one of the templates with minor changes in inflectional forms and tense. The creation of these templates took approximately $1$ to $2$ minutes. The total time required to create all sentence templates for entries of type NounPPframe took approximately $2$ hours, sentence templates for entries of type TransitiveFrame took approximately $3$ hours, and sentence templates for entries of type IntransitivePPframe took approximately $5$ hours.

3.2.3. Spanish Grammar Generation

The properties of Spanish nouns and verbs are not significantly different from those for Italian. Therefore, lexical entries are added in the same way as for Italian, providing a masculine and feminine form for each noun. However, the gradable adjective in Spanish differs from Italian. To add a lexical entry for an adjective such as ‘alto’ (i.e., ‘high’ in English), the language expert needs to provide the written form of superlative in the singular masculine form (i.e., ‘el más alto’), the plural masculine form (i.e., ‘los más altos’), the singular feminine form (i.e., ‘la más alta’), and the plural feminine form (i.e., ‘las más altas’).

We created a lexicon for the Spanish language and the DBpedia dataset (Release 2016-10; core, links).¹⁵ As shown in Table 4, there are a total of $262$ GRTs. After instantiating these GRTs with $470$ lexical entries of various syntactic frames, a total of $8, 548$ grammar rules were automatically generated.

Table 4.
An Overview Over the Number of Manually Created Lexical Entries, Grammar Rule Templates, and Grammar Rules for Different Frame Types of LexInfo for the Spanish Language.

# Lexical # Grammar Rule # Grammar

Frame Entry Templates (GRT) Rule (GR)

NounPPFrame $287$ $57$ $3, 512$

TransitiveFrame $81$ $85$ $2, 987$

InTransitivePPFrame $78$ $97$ $1, 503$

AdjectivePredicateFrame $13$ $14$ $519$

AdjectiveSuperlativeFrame $11$ $9$ $27$

Total $470$ $262$ $8, 548$

	# Lexical	# Grammar Rule	# Grammar
NounPPFrame	$287$	$57$	$3, 512$
TransitiveFrame	$81$	$85$	$2, 987$
InTransitivePPFrame	$78$	$97$	$1, 503$
AdjectivePredicateFrame	$13$	$14$	$519$
AdjectiveSuperlativeFrame	$11$	$9$	$27$
Total	$470$	$262$	$8, 548$

The table also shows the number of grammar rules that are generated automatically after instantiations of grammar rule templates with lexical entries.

The time required for creating a lexical entry is approximately $5$ to $10$ minutes depending on the syntactic frame. The time required for creating a sentence template is approximately $10$ to $25$ minutes. The total time required to create all sentence templates for entries of type NounPPframe took approximately $2$ hours, sentence templates for entries of type TransitiveFrame took approximately $4$ hours, and sentence templates for entries of type IntransitivePPframe took approximately $6$ hours.

The total construction time for the lexicons in four languages was approximately $60$ hours.

4. Implementation

Our approach consists of three components: (i) a grammar generator (as shown on the left side of Figure 3), (ii) a parser, and (iii) a web application (as shown on the right side of Figure 3). The details of all systems and step-by-step instructions for running them from scratch are available here.¹⁶

The core component is the grammar generator tool, which takes a Lemon lexicon and sentence templates as input and automatically creates lexicalized grammar rules. For example, the tool takes the sentence templates (STs) of a syntactic frame (e.g., NounPPFrame) and transforms them into GRTs (as detailed in Section 2.5). These GRTs are independent of the particular lexical entry. After that, these templates are instantiated with lexical entries (e.g., ‘mayor’ (of)) of NounPPFrame, resulting in lexicalized grammar rules. The grammar generation tool is a command-line tool developed in Java 15. Its source code is available here.¹⁷

The second component is the parser tool, which takes a question in natural language and grammar and provides one or more SPARQL queries. The parser is also a command line tool developed in Java 15, and its source code is available here.¹⁸

The third component is the web application (as shown on the right side of Figure 3), which is implemented using the Spring7 framework. Given a question, the application is able to present an answer (or a set of answers) given by the SPARQL endpoint in a comprehensive manner. The previous version of the QA system was limited to English (Elahi et al., 2021) and limited to answering a million questions. We extended the system for German, Spanish, and Italian and also improved the system by implementing the parsing method, thus the capacity of the QA system is extended to cope with an unlimited number of questions as the questions do not need to be indexed. The web-based demonstration of the QA system is available online here,¹⁹ and the source code can be obtained from here.²⁰

5. Evaluation

In this section, we present the evaluation of our approach using the well-known QALD shared task (Usbeck et al., 2023) as a benchmark. We evaluate the system on five editions of QALD (i.e., QALD-9, QALD-7, QALD-6, QALD-5, and QALD-3) and two modes: inductive and incremental. The inductive setting corresponds to the standard evaluation mode in which a model is induced from training data and its generalization ability is evaluated on unseen test data. In our case, the inductive aspects come from the fact that the lexicon entries are created to correspond to lexical elements used in questions in the training dataset.

In our work, we are interested however mainly in an incremental setting, showing that the performance of a QA system can be systematically improved by adding new lexical entries for uncovered questions. In the incremental evaluation mode, we thus add further lexical entries to cover the questions in the test split of the data.

In addition to evaluating our system with respect to QALD datasets, we also discuss how 3rd parties could successfully adapt the system to new datasets. Further, we show the consistency and controllability of our approach by analyzing the impact of removing or adding single lexical items on the performance of the QA system.

5.1. Benchmark

We evaluate our approach using the QALD challenge series (Usbeck et al., 2017) (i.e., QALD-9, QALD-7, QALD-6, QALD-5, and QALD-3), adopting standard evaluation metrics such as precision, recall, and F $_{1}$ for the DBpedia dataset. QALD benchmarks (Usbeck et al., 2019) allow us to measure the performance for multiple languages and over NLQs on DBpedia KB elements. The dataset contains questions in multiple languages and corresponding SPARQL queries and answers from DBpedia.

Table 5 shows the results of the evaluation against QALD-9 (Usbeck et al., 2018) training data for English, German, Italian, and Spanish questions. Figure 9 shows an example of how the F $_{1}$ -measures, the precision measures, and the recall measures improve for each lexical entry added to the lexicon in our evaluation. As can be seen from Table 5: (i) the micro F $_{1}$ -measure is $0.80$ for English; (ii) the score is $0.81$ for German; (iii) the score is $0.66$ for Italian; and (iv) for Spanish, the score is $0.66$ .

Table 5.
Results for QALD-9 Training Dataset.

Language Micro P Micro R Micro F $_{1}$ Macro P Macro R Macro F $_{1}$

English $0.88$ $0.74$ $0.80$ $0.55$ $0.56$ $0.55$

German $0.95$ $0.71$ $0.81$ $0.51$ $0.51$ $0.51$

Italian $0.89$ $0.53$ $0.66$ $0.45$ $0.46$ $0.45$

Spanish $0.90$ $0.52$ $0.66$ $0.43$ $0.43$ $0.43$

Language	Micro P	Micro R	Micro F $_{1}$	Macro P	Macro R	Macro F $_{1}$
English	$0.88$	$0.74$	$0.80$	$0.55$	$0.56$	$0.55$
German	$0.95$	$0.71$	$0.81$	$0.51$	$0.51$	$0.51$
Italian	$0.89$	$0.53$	$0.66$	$0.45$	$0.46$	$0.45$
Spanish	$0.90$	$0.52$	$0.66$	$0.43$	$0.43$	$0.43$

P refers to precision, R refers to recall, and F $_{1}$ refers to F $_{1}$ -measure.

Figure 9.

Results of the incremental evaluation using QALD-9 training data. After each extension of the lexicon, we evaluate precision, recall, and F $_{1}$ -measure.

The evaluation with test data is conducted in two modes: inductive and incremental.

5.2. Inductive Evaluation

The goal of the inductive evaluation mode is to evaluate our approach’s ability to generalize to unseen data or tasks. For the inductive evaluation, we used only the training data to create lexical entries. As explained in Section 3.1.2, we provide samples for each syntactic frame to the language expert, who then creates the sentence templates. These samples were taken from the QALD-9 training data, ensuring that the test data remains unseen by the language expert. Thus, in our inductive evaluation, we follow the same procedure typically used to evaluate machine learning approaches.

Table 6 shows the results for the QALD-9 test dataset for English, German, and Italian. As can be seen from the Table: (i) the micro F $_{1}$ -measure in inductive mode is $0.29$ for English; (ii) the score is $0.23$ for German; (iii) the score is $0.24$ for Italian; and (iv) for Spanish, the score is $0.24$ .

Table 6.
Results of the Inductive and Incremental Evaluation Using the QALD-9 Test Dataset.

Evaluation # Lexical Micro Micro Micro Macro Macro Macro

Language Mode Entries P R F $_{1}$ P R F $_{1}$

English Inductive $61$ $0.72$ $0.18$ $0.29$ $0.22$ $0.21$ $0.21$

Incremental $+ 14$ $0.91$ $0.79$ $0.85$ $0.31$ $0.30$ $0.30$

German Inductive $54$ $0.99$ $0.13$ $0.23$ $0.19$ $0.18$ $0.19$

Incremental $+ 11$ $0.99$ $0.70$ $0.82$ $0.28$ $0.28$ $0.27$

Italian Inductive $51$ $0.98$ $0.14$ $0.24$ $0.21$ $0.20$ $0.21$

Incremental $+ 13$ $0.99$ $0.49$ $0.65$ $0.27$ $0.26$ $0.27$

Spanish Inductive $53$ $0.98$ $0.13$ $0.24$ $0.2$ $0.19$ $0.19$

Incremental + $21$ $0.99$ $0.71$ $0.83$ $0.28$ $0.27$ $0.27$

	Evaluation	# Lexical	Micro	Micro	Micro	Macro	Macro	Macro
English	Inductive	$61$	$0.72$	$0.18$	$0.29$	$0.22$	$0.21$	$0.21$
	Incremental	$+ 14$	$0.91$	$0.79$	$0.85$	$0.31$	$0.30$	$0.30$
German	Inductive	$54$	$0.99$	$0.13$	$0.23$	$0.19$	$0.18$	$0.19$
	Incremental	$+ 11$	$0.99$	$0.70$	$0.82$	$0.28$	$0.28$	$0.27$
Italian	Inductive	$51$	$0.98$	$0.14$	$0.24$	$0.21$	$0.20$	$0.21$
	Incremental	$+ 13$	$0.99$	$0.49$	$0.65$	$0.27$	$0.26$	$0.27$
Spanish	Inductive	$53$	$0.98$	$0.13$	$0.24$	$0.2$	$0.19$	$0.19$
	Incremental	+ $21$	$0.99$	$0.71$	$0.83$	$0.28$	$0.27$	$0.27$

P refers to precision, R refers to recall, and F $_{1}$ refers to F $_{1}$ -measure.

5.3. Incremental Evaluation

In an incremental evaluation, we start with the lexicon created for the inductive evaluation and then add lexical entries incrementally until the questions in the test data are covered. The benefits of the incremental evaluation are that (i) a working version of the QA system for a particular language is ready for use after adding just a few lexical entries, and (ii) each lexical entry added to the lexicon improves the system. The goal of incremental evaluation is to maintain and enhance system performance in a dynamic and efficient manner, ensuring that the system remains robust and effective as new information becomes available.

We argue that our approach is indeed generalizable, as we use only the sentence templates created by the language expert for inductive evaluation, in which the test data remains unseen by the language expert. In the incremental evaluation, we only extend the lexicon at each step for the test data, but we never extend the set of sentence templates, so the grammar remains the same. Nevertheless, the performance improves when we extend the lexicon. Thus, the sentence templates created for the training data allow the approach to generalize and perform well on test data.

An important aspect is that we assume that there is a lexicon so all our results hinge on the availability of such a lexicon. While one could argue that in the incremental evaluation we are relying on test data to create the lexicon (not the sentence templates), we view this as simulating that there is a fairly complete lexicon by incorporating lexical entries relevant for test data. In our view, this is not different from assuming the availability of some other lexical resource such as WordNet (Fellbaum, 1998), FrameNet (Baker et al., 1998), ConceptNet (Speer et al., 2017) etc. The only difference is that we are "simulating" the availability of such a resource. We believe that our approach outperforms ML-based approaches under the assumption that a well-populated lexicon exists. Because, once we add the missing entries to the lexicon, our approach outperforms existing ML-based approaches.

Table 6 shows the results for the QALD-9 test dataset for English, German, and Italian. The table also shows the lexical entries common in both the training and test datasets and additional lexical entries created to cover cases in the test data. For example, for English, $61$ lexical entries are required both for the training dataset and the test dataset and an additional $14$ lexical entries are created to cover the test data.

Figure 10 shows the results of the incremental evaluation that begins with the lexicon created for the inductive evaluation and shows the impact of each additional lexical entry added to the lexicon. Note that, no sentence templates are added, so the grammar remains the same. For English, the evaluation begins with the result of the inductive evaluation (i.e., F $_{1}$ -measure $0.29$ ) and then the result is improved (i.e., F $_{1}$ -measure $0.85$ ) by incrementally adding $14$ lexical entries. Similarly, the F $_{1}$ -measure for German is improved from $0.23$ to $0.82$ by incrementally adding $11$ lexical entries.

Figure 10.

Results of the incremental evaluation using QALD-9 test data for English (top) and German (bottom). The Figure shows precision, recall, and F $_{1}$ -measure after adding a lexical element. (a) English and (b) German.

As can be seen from Table 6: (i) the micro F $_{1}$ -measure in incremental mode is $0.85$ for English; (ii) the score is $0.82$ for German; (iii) the score is $0.65$ for Italian; and (iv) for Spanish, the score is $0.83$ . So far there is no system described in the literature that works for four languages while reaching state-of-the-art performance on all of them.

5.4. Case Analysis

In order to illustrate the workings of our system, we analyze its behaviour in more detail by discussing four types of cases. We sample all questions from the QALD-9 training set and classify them into four cases²¹ to highlight interesting aspects of its behavior.

Case 1 (Exact SPARQL Query): There are many cases for which the parsing result of the question is exactly the same as specified in the QALD-9 dataset. This is the case for $196$ questions (i.e., $45$ %) in QALD-9. An example question is ‘Who developed Skype?’. Given that we model the lexical entry (to) ‘develop’ (i.e., TransitiveFrame) pointing to the property dbo:developer as semantics, specify that the syntactic object maps to domain of the property, the QA system is able to parse the question using the grammar rule (as detailed in Section 2.5.2) Who [VerbPast] <NP $_{map(SyntacticFunction), Property}$ >? and returns the exact SPARQL query as specified in the QALD-9 dataset. Table 7 shows the questions in three languages and the grammar rules for each of these questions.

Table 7.
Case 1: An Example of a Lexical Entry for which $100$ % F $_{1}$ -Measure is Obtained.

Lang. QALD-9 Lexical Entry Grammar Rule Template (GRT)

English ‘Who developed* Skype?’. (to) ‘develop’* Who [VerbPast] <NP $_{map(SyntacticFunction), Property}$ >?

German ‘Wer entwickelt* Skype?’* ‘entwickeln’ Wer [VerbPast] <NP $_{map(SyntacticFunction), Property}$ >?

Italian ‘Chi sviluppato* Skype?’* ‘sviluppare‘ Chi [VerbParticiple] <NP $_{map(SyntacticFunction), Property}$ >?

Spanish ‘Quien desarrollado* Skype?’* ‘desarrollar‘ Quien [VerbParticiple] <NP $_{map(SyntacticFunction), Property}$ >?

Lang.	QALD-9	Lexical Entry	Grammar Rule Template (GRT)
English	‘Who developed* Skype?’*.	(to) ‘develop’	Who [VerbPast] <NP $_{map(SyntacticFunction), Property}$ >?
German	‘Wer entwickelt* Skype?’*	‘entwickeln’	Wer [VerbPast] <NP $_{map(SyntacticFunction), Property}$ >?
Italian	‘Chi sviluppato* Skype?’*	‘sviluppare‘	Chi [VerbParticiple] <NP $_{map(SyntacticFunction), Property}$ >?
Spanish	‘Quien desarrollado* Skype?’*	‘desarrollar‘	Quien [VerbParticiple] <NP $_{map(SyntacticFunction), Property}$ >?

In this case, the parsing result is the same SPARQL query as specified in the QALD-9 dataset.

Case 2 (Similar labels): A second case is one where our system is able to parse a question that is equivalent to the question in QALD-9 apart from the labels used for entities. This is the case for $31$ questions (i.e., $8$ %). An example is the QALD-9 question ‘List all boardgames by GMT.’ The lexical entry involved in generating the grammar rule is ‘boardgame (by)‘. The semantics of the entry is captured with respect to the property dbo:publisher, the subject of the property is dbo:VideoGame, and the object of the property is dbo:Company. The question is parsed using the grammar rule S -> ‘List all boardgames by <NP $_{Range, dbo:publisher}$ >?’. For this grammar rule, the values of the non-terminal <NP $_{Range, dbo:publisher}$ > are retrieved with the following automatically generated SPARQL query:

For matching the entity ‘GMT’ of the question ‘List all boardgames by GMT.’ with these terminals, we used the Jaccard similarity measure (as detailed in Section 2.7) and set a similarity threshold of $0.5$ . Each terminal (e.g., ‘Cambridge University Press’, ‘GMT GAMES’, etc.) is converted into a set of tokens, which is then compared with the set of tokens (i.e., ‘GMT’) from the entity of the question. The Jaccard similarity method finds the similarity by the size of the intersection divided by the size of the union of the two sets. The best match of the token ‘GMT’ is ‘GMT Games’ since the similarity score is greater than/equal to the threshold $0.5$ . The parse result is as follows:

Case 3 (Multiple properties): There are cases where a question in QALD-9 is parsed by our QA system, but the corresponding SPARQL query differs. This is the case for $23$ questions (i.e., $6$ %) in QALD-9. The question ‘Which organizations were founded in 1950?’ is parsed by our system, relying on the grammar rule S -> Which organizations were founded in <NP $_{Range, dbo:formationYear}$ >?’ The lexical entry is ‘founded’ (in), the subject of the property is dbo:Organisation, and the object of the property is xsd:gYear.

The SPARQL query contained in the QALD-9 dataset for the question consists of UNION operations involving multiple properties:

In contrast, the SPARQL query generated by our approach consists of one property. The parse result of the question is as follows:

Case 4 (ASK queries): Our approach can parse boolean questions (i.e., ASK queries that have a Yes/No answer). Consider the question ‘Is Rita Wilson the spouse of Tom Hanks?’ Our approach can parse this question by the grammar rule: Is <NP $_{Range, dbo:spouse}$ > spouse of <NP $_{Domain, dbo:spouse}$ >? The rule is automatically generated by instantiating the GRT (as detailed in Section 2.5.1) using the lexical entry ‘spouse’ (of). The semantics of the entry is captured with respect to the property dbo:spouse, the syntactic subject maps to the object of the property dbo:Person, and the syntactic object maps to the subject of the property dbo:Person. This is the case for $37$ questions (i.e., $9$ %) in QALD-9.

To the best of our knowledge, there is only one knowledge graph based QA system described in the literature that can handle limited forms of boolean questions (i.e., temporal questions) (Ding et al., 2023) In contrast, the automatically generated grammar rules in our approach allows us to parse a wide range of boolean questions from RDF data.

5.5. Error Analysis

In order to better understand the potential errors that our system produces, we manually analysed errors on the QALD-9 training data, yielding nine types of errors:

–
Complex queries: Our approach supports a limited form of composition (as detailed in Section 2.8) by nesting rules that generate noun phrases into the rules for the other frames. $31$ % of errors are due to the limited support of some variations of complex queries. Therefore, the approach can not handle certain type of complex questions such as ‘Give me all people that were born in Vienna and died in Berlin.’ The SPARQL query is as follows:

While in principle the question could be parsed by the grammar rule S -> Give me all people that were born in <NP $_{Range, dbo:birthPlace}$ >., our approach can at the moment not handle conjunctions, neither at the verb phase nor noun phrase level.
–
Abbreviated label: $5 %$ of errors are due to the abbreviated label of entities in the QALD-9 question. Consider the QALD-9 question ‘Where was JFK assassinated?’ and its SPARQL query:

We used the Jaccard similarity measure (as detailed in Section 2.7) to match the entity (of the question) with the label of the DBpedia entity and set a similarity threshold of $0.5$ . Our system fails to parse the question as the similarity score of string ‘JFK’ and ‘John F. Kennedy’ is below $0.5$ .
–
Wrong label: $7 %$ of errors are due to the wrong label of entities in the QALD-9 question. Consider the QALD-9 question ‘Who wrote the book Les Piliers de la terre’ and its SPARQL query:

The question contains the French label (i.e., ‘Les Piliers de la terre?’) of the entity res:The_Pillars_of_the_Earth instead of the English label (i.e., ‘The Pillars of the Earth’).
–
SPARQL query without property: There are cases (i.e., $8$ %) where the SPARQL query of QALD-9 question does not contain any property. Consider the question ‘How many airlines are there?’ and its SPARQL query:

In our approach, however, the semantics of all lexical entries for frames is specified in relation to one property. Thus, in order to successfully parse a question, the question needs to have at least one noun or verb (as detailed in Section 2.3) referring to a property. This issue could be solved by a simple grammar rule of the form ‘How many X are there’ where X refers to all possible classes in DBpedia.
–
Entity-less questions: All our grammar rules assume that at least one of the syntactic positions is filled with a specific entity at subject or object position of the corresponding property. There are however questions in QALD that mention no specific entity. An example is the following question:

‘What is the most frequent cause of death?’ and its SPARQL query:

Our grammar rules generated for the lexical entry ‘death cause’ (of) could instead parse questions involving at least one entity (that died), for example

‘What is the cause of death of Calel Perechodnik?’ as

–
Adjective questions: $7$ % of errors are due to the missing support for different variations of superlative questions (e.g., the question ‘Which Indian company has the most employees?’,‘What is the bridge with the longest span?’ Our approach can only process the canonical forms: ‘What is the highest mountain in Australia?’ or ‘What is the longest river?’, respectively.
–
missing templates: $14$ % of errors are due to missing templates. This is the case for questions such as ‘In which countries can you pay using the West African CFA franc?’ or ‘Which ingredients do I need for carrot cake?’, among others. The sentence templates are created by language expert, so it is unclear why certain templates were not created. To assist in creating sentence templates, we provide a few examples along with a dictionary of syntactic categories (detailed in Section 3.1.2) to the language expert. However, no examples are provided using the first person ‘I’ or second person ‘You’, which may discourage the language expert from creating such templates. For example, the language expert created templates for questions like ‘What are the ingredients of carrot cake?’ and ‘Give me all the ingredients of carrot cake?’ (detailed in Section 2.5.1), but not for ‘Which ingredients do I need for carrot cake?’ or ‘Which ingredients do you need for carrot cake?’.

Another example of a missing template involves temporal questions, such as the question ‘Give me all libraries established earlier than 1400.’. $4$ % of errors are due to the missing support for different variations of temporal questions. Instead, our approach can parse the question ‘Give me all libraries established in 1400’ using the grammar rule: S -> ‘Give me all libraries established in $< {NP}_{Range, dbo:established} >$ ?’ relying on the lexical entry ‘establish’ (in).
–
Spurious matching: $8 %$ of errors are due to the fact that we retrieve the wrong entity using the Jaccard similarity. Consider the QALD-9 question ‘Where was Bach born?’ The QA system parses the question using the grammar rule S -> ‘Where was $< {NP}_{Domain, dbo:birthPlace} > ?^{'}$ relying on the lexical entry ‘born’ (in) The parse is as follows:

But the SPARQL query of the question contained in QALD-9 is:

Note that the entity ‘Bach’ of the question does not match with the entity ‘Johann Sebastian Bach.’ Instead, the highest ranked entity according to the Jaccard similarity score is res:Bach_(actor).

5.6. Comparison to State-of-the-Art English QA Systems

In this section, we discuss question-answering systems evaluated on the QALD benchmark and compare their performance (i.e., F $_{1}$ -Measure) with our approach. To compare our system with these approaches, we evaluate our system both in inductive and incremental modes.

The goal of most QALD systems is to map NLQs to corresponding SPARQL queries. State-of-the-art question-answering systems use machine learning to learn a model to map natural language questions into SPARQL queries. These systems support in most cases only the English language.

5.6.1. Comparision With ML-Based Approach

There is a fundamental difference between how machine learning models are extended and how our model-based approach is extended. While machine learning-based approaches require extensive training data to learn to handle phenomena that are either absent or not sufficiently prominent in the training set, our approach operates independently of training data. Instead, the capabilities of our system can be increased by extending the lexicon, which is a completely different paradigm. This allows us to improve our system step by step (as detailed in Section 5.3).

The incremental improvement of our approach can be compared to experiments on the continuous improvement of machine learning approaches; however, such experiments are typically outside the scope of ML papers, and we lack evidence on how effectively ML methods can be incrementally improved by modifying the training data. There are approaches, such as Barz and Sonntag (2019), that support incremental improvement of QA systems without re-training. However, to the best of our knowledge, this approach has not been applied to linked data, so it remains unclear how machine learning-based QALD systems can be incrementally improved. Therefore, we explain existing ML-based QALD systems and their limitations, highlighting the advantages of our approach for incremental improvements.

In neural network-based QALD systems (Chakraborty et al., 2019; Lukovnikov et al., 2017; Mohammed et al., 2018; Yasunaga et al., 2021), the problem of transforming natural language into a SPARQL query is typically framed as a semantic parsing problem. For example, the transfer learning-based QALD approach (Lukovnikov et al., 2019) uses a fine-tuned BERT model (Vaswani et al., 2017) and decomposes question interpretation into two separate learning tasks: (1) entity span detection, which identifies the spans in the question that mention the entity, and (2) relation classification, which predicts the relation used in the question. In the question ‘Who is the mayor of New York City?’, the entity span would be ‘New York City’, and the relation span would be ‘mayor’. The limitation is that linked data uses a formal structure, with specific ontologies, technical terminologies, and relationships that are less common in natural language text. In the context of incremental improvement, our approach can be easily extended for questions like ‘When was Carlo Giuliani shot?’ by adding a lexical entry such as ‘shot in’ for the property dbo:deathDate (detail in Section 5.3). However, pre-trained models may not have encountered enough examples of these specific patterns during training, even though they are trained on large corpora of text. Moreover, while transfer learning-based approaches are typically employed only for simple questions, our approach supports nesting grammar rules (detail in Section 2.8) and a limited set of compositionality.

There are approaches that translate NLQs into SPARQL queries (Luz & Finger, 2018; Soru et al., 2017, 2018; Yin et al., 2021), which are vocabulary-dependent and require training on samples derived from manually created QA template pairs. Some approaches use a model for joint query interpretation and response ranking (Sawant & Chakrabarti, 2013); others learn semantic parsers (Berant et al., 2013; Kwiatkowski et al., 2013) given a knowledge base and a set of question/answer pairs. Other approaches, such as TeBaQA (Vollmers et al., 2021a), map NLQs to SPARQL queries by learning templates from the benchmark (e.g., the QALD-9 dataset (Usbeck et al., 2018)). It classifies NLQs into isomorphic basic graph patterns and then uses this classification to map a question to a template. However, the training dataset is small, consisting of only $403$ questions, which limits the ability to learn templates. These approaches rely on either manually created training data, which is time-consuming and expensive, or the availability of benchmarks. On the other hand, our model-based approach can be incrementally improved by adding lexical entries to the lexicon; the time required for creating a lexical entry is approximately $5$ to $10$ minutes (detail in Section 3.1.1).

5.6.2. Systems Evaluated on QALD-9

TeBaQA (Vollmers et al., 2021a), map NLQs to SPARQL queries by learning templates from the benchmark. As shown in Table 8, TeBaQA achieves an overall F $_{1}$ -Measure of $0.37$ without any manual effort, whereas our model-based approach achieves $0.85$ with very little manual effort. gAnswer (Zou et al., 2014) is a graph-based approach that maps a NLQ into a semantic query graph containing an edge for each relation mentioned in the question. After that, it simplifies the transformation of natural language into a SPARQL query by converting it into a subgraph matching problem. Similarly, EDGQA (Hu et al., 2021) introduces a graph structure called the Entity Description Graph (EDG), which provides a hierarchical decomposition of a NLQ for generating and composing SPARQL queries. The graph-based approaches achieve an overall F $_{1}$ -Measure ranging from $0.30$ to $0.45$ , which is better than the machine learning approaches. The incremental evaluation shows that our approach achieves a $30$ - $40$ % improvement over this.

Table 8.
Results for Machine Learning and Rule-Based Systems for English in Different QALD Series.

QALD Dataset System Micro P Micro R Micro F $_{1}$

QALD-9 Test Elon (Usbeck et al., 2018) $0.04$ $0.05$ $0.10$

gAnswer (Zou et al., 2014) $0.29$ $0.32$ $0.43$

NSQA (Kapanipathi et al., 2021) $0.31$ $0.32$ $0.45$

WDAqua-core1 (Diefenbach et al., 2020a) $0.26$ $0.26$ $0.28$

QASystem (Usbeck et al., 2018) $0.09$ $0.11$ $0.20$

Zheng and Zhang (2019) $0.45$ $0.47$ $0.46$

Falcon 1.0 (Sakor et al., 2019) $0.23$ $0.23$ $0.23$

SLING (Mihindukulasooriya et al., 2020) $0.39$ $0.50$ $0.44$

GenRL (Rossiello et al., 2021) $0.49$ $0.61$ $0.53$

EDGQA (Hu et al., 2021) $0.31$ $0.40$ $0.32$

KGQAN (Omar et al., 2023) $0.49$ $0.39$ $0.43$

Galactica (Glaese et al., 2022) $0.14$ $0.02$ $0.03$

TeBaQA (Vollmers et al., 2021a) $0.24$ $0.24$ $0.37$

our approach (inductive, L= $61$ ) $0.98$ $0.14$ $0.25$

our approach (incremental (L=+ $14$ ) $0.95$ $0.76$ $0.85$

QALD-7 Train WDAqua-core1 (Diefenbach et al., 2020a) $0.37$ $0.39$ $0.39$

our approach $0.79$ $0.93$ $0.86$

QALD-6 Test AMUSE (Hakimov et al., 2017)²² – – $0.34$

our approach (inductive, L= $53$ ) $0.42$ $0.50$ $0.45$

our approach (incremental (L=+ $47$ ) $0.79$ $0.83$ $0.82$

QALD-5 Test Xser (Xu et al., 2014) $0.74$ $0.72$ $0.73$

UTQA (ebn veyseh, 2016) $0.55$ $0.53$ $0.54$

AskNow (Dubey et al., 2016) $0.32$ $0.34$ $0.33$

SemGraphQA (Beaumont et al., 2015) $0.19$ $0.20$ $0.20$

YodaQA (Baudiš & Šedivý, 2015) $0.18$ $0.17$ $0.18$

our approach (incremental, L= $38$ ) $0.76$ $0.75$ $0.75$

QALD-3 Test gAnswer (Beaumont et al., 2015) $0.40$ $0.40$ $0.40$

CASIA (He et al., 2014) $0.35$ $0.36$ $0.36$

Scalewelis (Guyonvarc’H & Ferré, 2013) $0.33$ $0.33$ $0.33$

RTV (Giannone et al., 2013) $0.32$ $0.34$ $0.33$

Intui2 (Cimiano et al., 2013) $0.32$ $0.32$ $0.32$

SWIP (Pradel et al., 2012) $0.16$ $0.15$ $0.16$

DEANNA (Yahya et al., 2012) $0.21$ $0.21$ $0.21$

our approach (inductive, L= $23$ ) $0.41$ $0.46$ $0.44$

our approach (incremental, L=+ $17$ ) $0 .75$ $0 .83$ $0 .78$

QALD	Dataset	System	Micro P	Micro R	Micro F $_{1}$
QALD-9	Test	Elon (Usbeck et al., 2018)	$0.04$	$0.05$	$0.10$
		gAnswer (Zou et al., 2014)	$0.29$	$0.32$	$0.43$
		NSQA (Kapanipathi et al., 2021)	$0.31$	$0.32$	$0.45$
		WDAqua-core1 (Diefenbach et al., 2020a)	$0.26$	$0.26$	$0.28$
		QASystem (Usbeck et al., 2018)	$0.09$	$0.11$	$0.20$
		Zheng and Zhang (2019)	$0.45$	$0.47$	$0.46$
		Falcon 1.0 (Sakor et al., 2019)	$0.23$	$0.23$	$0.23$
		SLING (Mihindukulasooriya et al., 2020)	$0.39$	$0.50$	$0.44$
		GenRL (Rossiello et al., 2021)	$0.49$	$0.61$	$0.53$
		EDGQA (Hu et al., 2021)	$0.31$	$0.40$	$0.32$
		KGQAN (Omar et al., 2023)	$0.49$	$0.39$	$0.43$
		Galactica (Glaese et al., 2022)	$0.14$	$0.02$	$0.03$
		TeBaQA (Vollmers et al., 2021a)	$0.24$	$0.24$	$0.37$
		our approach (inductive, L= $61$ )	$0.98$	$0.14$	$0.25$
		our approach (incremental (L=+ $14$ )	$0.95$	$0.76$	$0.85$
QALD-7	Train	WDAqua-core1 (Diefenbach et al., 2020a)	$0.37$	$0.39$	$0.39$
		our approach	$0.79$	$0.93$	$0.86$
QALD-6	Test	AMUSE (Hakimov et al., 2017)²²	–	–	$0.34$
		our approach (inductive, L= $53$ )	$0.42$	$0.50$	$0.45$
		our approach (incremental (L=+ $47$ )	$0.79$	$0.83$	$0.82$
QALD-5	Test	Xser (Xu et al., 2014)	$0.74$	$0.72$	$0.73$
		UTQA (ebn veyseh, 2016)	$0.55$	$0.53$	$0.54$
		AskNow (Dubey et al., 2016)	$0.32$	$0.34$	$0.33$
		SemGraphQA (Beaumont et al., 2015)	$0.19$	$0.20$	$0.20$
		YodaQA (Baudiš & Šedivý, 2015)	$0.18$	$0.17$	$0.18$
		our approach (incremental, L= $38$ )	$0.76$	$0.75$	$0.75$
QALD-3	Test	gAnswer (Beaumont et al., 2015)	$0.40$	$0.40$	$0.40$
		CASIA (He et al., 2014)	$0.35$	$0.36$	$0.36$
		Scalewelis (Guyonvarc’H & Ferré, 2013)	$0.33$	$0.33$	$0.33$
		RTV (Giannone et al., 2013)	$0.32$	$0.34$	$0.33$
		Intui2 (Cimiano et al., 2013)	$0.32$	$0.32$	$0.32$
		SWIP (Pradel et al., 2012)	$0.16$	$0.15$	$0.16$
		DEANNA (Yahya et al., 2012)	$0.21$	$0.21$	$0.21$
		our approach (inductive, L= $23$ )	$0.41$	$0.46$	$0.44$
		our approach (incremental, L=+ $17$ )	$0 .75$	$0 .83$	$0 .78$

L refers to the number of lexical entries, P refers to precision, R refers to recall, and F $_{1}$ refers to F $_{1}$ -measure.

SLING (Mihindukulasooriya et al., 2020) and GenRL (Rossiello et al., 2021) are relationship linking frameworks developed for question-answering over linked data. The former approach employs semantic parsing using Abstract Meaning Representation (i.e., a rooted, directed, acyclic graph expressing a question) to extract entities and relations between them, utilizing linguistic cues and information from DBpedia. The latter approach utilizes pre-trained sequence-to-sequence models, using only the question text to generate a sequence of relations based on the question. It also employs knowledge integration and validation mechanisms to address the nuances of the knowledge bases. However, the work does not explain the process of constructing a SPARQL query from the sequence of relations. The relation linking-based approaches achieve the highest overall F $_{1}$ -Measure (i.e., ranging from $0.40$ to $0.55$ ) among all existing systems evaluated on the QALD-9 dataset, whereas our model-based approach outperforms these methods when evaluated in incremental mode.

5.6.3. Systems Evaluated on QALD-7 to QALD-3

We also compare our approach with systems evaluated on older QALD benchmarks, such as QALD-7 (Usbeck et al., 2017) to QALD-3. CASIA (He et al., 2014) is a machine learning-based QA system that translates NLQs into SPARQL by the following pipeline: (i) first, it decomposes the question and detects the candidate phrases; (ii) second, it maps the phrases to semantic items; (iii) finally, it groups the mapped items into semantic triples using a Markov Logic Network (MLN). A similar approach is RTV (Giannone et al., 2013) that selects key ontological information from the question (i.e., predicate, arguments, and properties) using a Hidden Markov Model.

These approaches, together with others (i.e., AMUSE Hakimov et al., 2017, Scalewelis Guyonvarc’H & Ferré, 2013, Intui2 Cimiano et al., 2013, SWIP Pradel et al., 2012, ForecastTKGQuestions Ding et al., 2023, and DEANNA Yahya et al., 2012), were evaluated by inducing a model from training data and testing the model on test data. Table 8 shows the comparison of our approach with these machine learning-based and rule-based systems for English. The incremental evaluation shows that our approach outperforms all machine learning approaches.

Furthermore, Table 8 also shows the results of the evaluation of our system in comparison to rule-based systems such as Xser (Xu et al., 2014), SemGraphQA (Beaumont et al., 2015), and YodaQA (Ruseti et al., 2015) for English. Xser (Xu et al., 2014) takes as input a NLQ in English and retrieves an answer. In this system, the user query is linguistically analyzed to detect predicate-argument structures through a semantic parser. SemGraphQA (Beaumont et al., 2015) transforms NLQs into SPARQL queries. YodaQA (Ruseti et al., 2015) represents the input question as a bag of features and then generates a set of candidate answers by performing a search in the knowledge base for these features. All these rule-based QA systems are evaluated over the QALD-5 test dataset in incremental mode and we see that our system outperforms all systems.

5.7. Performance on Other Languages

A survey (Diefenbach et al., 2018) of the multilingual QALD shows that $10$ % of them were applied to other than the English language and $5$ % of them were applied to more than one RDF dataset. Table 9 lists multilingual QA systems over linked data that were developed in the last 10 years.

Table 9.
The Table Presented in the Multilingual Question Answering System Survey (Perevalov et al., 2022b) Illustrates the Development of Multilingual Question-Answering Systems Over the Last 10 Years.

System Language other than English

QALL-ME (Ferrández et al., 2011) German, Italian, and Spanish

XKnowSearch! (Zhang et al., 2016a) German and Chinese

UTQA (ebn veyseh, 2016) Spanish, Finnish, and Swedish

BreXearch (Zhang et al., 2017) German and Spanish

AMUSE (Hakimov et al., 2017) German and Spanish

DeepPavlov (Burtsev et al., 2018) Russian

Platypus (Tanon et al., 2018) French and Spanish

WDAqua-core1 (Diefenbach et al., 2020a) German, French, Italian, and Spanish

QAnswer (Diefenbach et al., 2020b) English, German, French, Italian, Spanish, Portuguese, Arabic, and Chinese

Zhou Y. (Zhou et al., 2021) Farsi, German, Romanian, Italian, Russian, French, Dutch, Spanish, Hindi, and Portuguese

System	Language other than English
QALL-ME (Ferrández et al., 2011)	German, Italian, and Spanish
XKnowSearch! (Zhang et al., 2016a)	German and Chinese
UTQA (ebn veyseh, 2016)	Spanish, Finnish, and Swedish
BreXearch (Zhang et al., 2017)	German and Spanish
AMUSE (Hakimov et al., 2017)	German and Spanish
DeepPavlov (Burtsev et al., 2018)	Russian
Platypus (Tanon et al., 2018)	French and Spanish
WDAqua-core1 (Diefenbach et al., 2020a)	German, French, Italian, and Spanish
QAnswer (Diefenbach et al., 2020b)	English, German, French, Italian, Spanish, Portuguese, Arabic, and Chinese
Zhou Y. (Zhou et al., 2021)	Farsi, German, Romanian, Italian, Russian, French, Dutch, Spanish, Hindi, and Portuguese

QALL-ME (Ferrández et al., 2011) is a multilingual Question-answering (QA) system that operates on structured data and has been developed for cinema/movie events in the tourism domain. Multilinguality is achieved by an ontology modelling the domain, which also represents a limitation of the framework since an available ontology has to be previously specified. XKnowSearch! (Zhang et al., 2016a) and BreXearch (Zhang et al., 2017) are multilingual semantic search systems that extract knowledge from various media sources, including online news sites, social media, and live TV, and integrate it with additional background knowledge from DBpedia. The approach allows to ask questions not through NLQs but with keyword queries or by using entities or SPARQL queries. These systems are focused on keyword-based searches instead of mapping NLQs to SPARQL queries, and no evaluation is provided.

The UTQA (ebn veyseh, 2016) system uses a QA pipeline of several language-dependent components, such as keyword extraction and type identification, entity linking, and answer extraction. The DeepPavlov (Burtsev et al., 2018) KBQA system works based on a set of large language-dependent deep learning models that perform such tasks as query type prediction, entity detection, relation detection and path ranking. The Platypus system (Tanon et al., 2018) transforms a NLQ into a custom logical representation using a language-dependent grammatical analyzer and a set of predefined rules. From a broad perspective, all these systems are focused on solving the QA problem using language-independent tools, which attracts a wider NLP audience instead of the Question Answering over Linked Data (QALD) community. Therefore, these systems lack evaluations against benchmarks on QALD.

To our knowledge, AMUSE (Hakimov et al., 2017) is the first machine learning-based multilingual QA system developed for languages other than English. The questions are first parsed by a dependency parser and then the model learns mappings between the dependency parse tree for a given question text and RDF nodes in the SPARQL query. AMUSE relies on the one hand on lexicalizations of DBpedia properties as extracted by M-ATOLL (Walter et al., 2014) for multiple languages. In particular for Spanish and German. However, M-ATOLL uses two methods for lexicalization: a label-based approach to extract lexicalizations using ontology labels (i.e., rdfs:label) and a dependency-based approach to extract lexicalizations of ontology properties from an available text corpus (i.e., Wikipedia text). The former method suffers from a lexical gap between natural language terms and the content of KB elements, and the latter method produces every other lexical entry inappropriate. Moreover, the approach is not designed to be deployed in a fully unsupervised mode to a new language.

Table 10 shows the results of the inductive evaluation and the incremental evaluation of our approach. The table also shows the lexical entries common in both the training and test datasets and additional lexical entries created over training data to cover the test. For instance, in the case of German, $41$ lexical entries are required both for the training dataset and the test dataset, and an additional $45$ lexical entries are created for incremental evaluation. Our approach outperforms the AMUSE system for both German and Spanish. The results show that the results measured in the inductive evaluation for German are improved from $0.44$ to $0.75$ by adding $45$ additional lexical entries incrementally. Similarly, for Spanish, it is improved from $0.40$ to $0.83$ by adding $38$ lexical entries.

Table 10.

Results for Machine Learning and Rule-Based Approaches for German, Italian, and Spanish.

QALD	Language	QA System	Micro P	Micro R	Micro F $_{1}$
QALD-6	German	AMUSE (Hakimov et al., 2017)²³	–	–	$0.37$
(Test)		our approach (ind*) (L= $41$ )	$0.41$	$0.46$	$0.44$
		our approach (inc*) (L=+ $45$ )	$0.77$	$0.73$	$0.75$
	Spanish	AMUSE (Hakimov et al., 2017)	–	–	$0.42$
		our approach (ind*) (L=+ $36$ )	$0.40$	$0.41$	$0.40$
		our approach (inc*) (L=+ $41$ )	$0.78$	$0.88$	$0.83$
QALD-7	German	WDAqua-core1 (Diefenbach et al., 2020a)	$0.27$	$0.28$	$0.27$
(Train)		our approach	$0.74$	$0.71$	$0.73$
	Italian	WDAqua-core1 (Diefenbach et al., 2020a)	$0.90$	$0.20$	$0.18$
		Nolano et al. (2022)	$0.48$	$0.22$	$0.30$
		our approach	$0.72$	$0.85$	$0.78$
	Spanish	WDAqua-core1 (Diefenbach et al., 2020a)	$0.31$	$0.32$	$0.31$
		our approach	$0.84$	$0.78$	$0.81$

ind* refers to inductive, inc* refers to incremental, and L refers to the number of lexical entries, P refers to precision, R refers to recall, and F $_{1}$ refers to F $_{1}$ -measure.

WDAqua-core1 (Diefenbach et al., 2020a) is a rule-based approach that translates NLQs to SPARQL queries. The approach is implemented for different languages other than English namely German, French, Italian, and Spanish. As can be seen from Table 10, our approach outperforms the WDAqua-core1 system for all languages.

5.8. Comparsion With ChatGPT

ChatGPT (Ouyang et al., 2022) is a pre-trained large language model that can answer questions, ask for clarification, create dialogues with the user, and deal with follow-up questions. It is trained using reinforcement learning with human feedback, by using a combination of a new dialogue dataset and the InstructGPT dataset. Previous research (Petroni et al., 2019) shows that language models can, to some extent, be considered as knowledge bases (KBs) to support downstream natural language processing (NLP) tasks. This has sparked a growing interest in exploring whether ChatGPT and other large language models (LLMs) can replace traditional Question Answering over Knowledge Graphs (Yih et al., 2016) approaches. Benefiting from extensive training on information derived from Wikipedia and showcasing impressive natural language understanding capabilities, ChatGPT has been demonstrated to be a powerful question-answering system (Faria et al., 2023; Tan et al., 2023).

5.8.1. Prompt

In this section, we compare how ChatGPT (i.e., a language model) performs compared to our approach of question answering over linked data. In particular, we evaluated our approach against ChatGPT using the well-known QALD-9 benchmarks on September 5, 2024. We used ChatGPT 4 (i.e., GPT-4) in a zero-shot and few-shot scenario. In zero-shot scenario, we prompted it with an instruction to generate a SPARQL query for a given question with respect to DBpedia without providing further examples. The prompt for English is shown below:

In a few-shot scenario, we prompted it with an instruction to generate a SPARQL query for a given question in relation to DBpedia, providing one example for each of the five syntactic frames (i.e., NounPPFrame, TransitiveFrame, InTransitivePPFrame, AdjectivePredicateFrame, AdjectiveSuperlativeFrame) in each language. Here is an example for English. The prompts for other languages (German, Italian, and Spanish) are provided in the Appendix.

Table 11 shows the results for the QALD-9 test dataset for four languages. The table further illustrates the evaluation of our approach to the QALD-9 test dataset in both inductive and incremental modes. The table highlights a comparative performance between our approach and ChatGPT across various scenarios, specifically focusing on zero-shot and few-shot learning across different languages. As shown in the table, our approach outperforms ChatGPT in zero-shot scenarios across four languages, in both inductive and incremental modes.

Table 11.
Results for ChatGPT for QALD-9 Test Dataset.

Micro Micro Micro Macro Macro Macro

Language System P R F $_{1}$ P R F $_{1}$

English ChatGPT (zero shot) $0.35$ $0.06$ $0.11$ $0.05$ $0.05$ $0.05$

ChatGPT (few shot) $0.90$ $0.19$ $0.32$ $0.29$ $0.28$ $0.28$

our approach (ind) $0.72$ $0.18$ $0.29$ $0.22$ $0.21$ $0.21$

our approach (inc) $0.91$ $0.79$ $0.85$ $0.31$ $0.30$ $0.30$

German ChatGPT (zero shot) $0.41$ $0.01$ $0.02$ $0.02$ $0.02$ $0.02$

ChatGPT (few shot) $0.92$ $0.13$ $0.23$ $0.22$ $0.21$ $0.21$

our approach (ind) $0.99$ $0.13$ $0.23$ $0.19$ $0.18$ $0.19$

our approach (inc) $0.99$ $0.70$ $0.82$ $0.28$ $0.28$ $0.27$

Italian ChatGPT (zero shot) $0.77$ $0.07$ $0.14$ $0.06$ $0.05$ $0.05$

ChatGPT (few shot) $0.82$ $0.21$ $0.33$ $0.19$ $0.18$ $0.18$

our approach (ind) $0.98$ $0.14$ $0.24$ $0.21$ $0.20$ $0.21$

our approach (inc) $0.99$ $0.49$ $0.65$ $0.27$ $0.26$ $0.27$

Spanish ChatGPT (zero shot) $0.007$ $0.01$ $0.01$ $0.05$ $0.05$ $0.05$

ChatGPT (few shot) $0.93$ $0.17$ $0.29$ $0.19$ $0.19$ $0.18$

our approach (ind) $0.98$ $0.13$ $0.24$ $0.2$ $0.19$ $0.19$

our approach (inc) $0.99$ $0.71$ $0.83$ $0.28$ $0.27$ $0.27$

		Micro	Micro	Micro	Macro	Macro	Macro
English	ChatGPT (zero shot)	$0.35$	$0.06$	$0.11$	$0.05$	$0.05$	$0.05$
	ChatGPT (few shot)	$0.90$	$0.19$	$0.32$	$0.29$	$0.28$	$0.28$
	our approach (ind*)	$0.72$	$0.18$	$0.29$	$0.22$	$0.21$	$0.21$
	our approach (inc*)	$0.91$	$0.79$	$0.85$	$0.31$	$0.30$	$0.30$
German	ChatGPT (zero shot)	$0.41$	$0.01$	$0.02$	$0.02$	$0.02$	$0.02$
	ChatGPT (few shot)	$0.92$	$0.13$	$0.23$	$0.22$	$0.21$	$0.21$
	our approach (ind*)	$0.99$	$0.13$	$0.23$	$0.19$	$0.18$	$0.19$
	our approach (inc*)	$0.99$	$0.70$	$0.82$	$0.28$	$0.28$	$0.27$
Italian	ChatGPT (zero shot)	$0.77$	$0.07$	$0.14$	$0.06$	$0.05$	$0.05$
	ChatGPT (few shot)	$0.82$	$0.21$	$0.33$	$0.19$	$0.18$	$0.18$
	our approach (ind*)	$0.98$	$0.14$	$0.24$	$0.21$	$0.20$	$0.21$
	our approach (inc*)	$0.99$	$0.49$	$0.65$	$0.27$	$0.26$	$0.27$
Spanish	ChatGPT (zero shot)	$0.007$	$0.01$	$0.01$	$0.05$	$0.05$	$0.05$
	ChatGPT (few shot)	$0.93$	$0.17$	$0.29$	$0.19$	$0.19$	$0.18$
	our approach (ind*)	$0.98$	$0.13$	$0.24$	$0.2$	$0.19$	$0.19$
	our approach (inc*)	$0.99$	$0.71$	$0.83$	$0.28$	$0.27$	$0.27$

ind* refers to inductive, inc* refers to incremental, and L refers to the number of lexical entries, P refers to precision, R refers to recall, and F $_{1}$ refers to F $_{1}$ -measure.

However, in the few-shot scenario, ChatGPT outperforms our approach in the inductive mode. ChatGPT is trained on an extensive corpus of diverse, high-quality data from various domains, enabling it to understand a wide range of topics and questions. It is unclear whether the QALD-9 dataset was part of the data used to train ChatGPT. Nevertheless, our approach outperforms ChatGPT in the few-shot scenario in the incremental evaluation mode.

5.8.2. Case Analysis

In order to compare the results of ChatGPT with those produced by our system, we manually analyzed the errors and non-errors in the first $100$ questions of the QALD-9 test data for English in a few-shot scenario, identifying six types²⁴ of cases:

Case 1 (Exact SPARQL query): There are only a few instances (i.e., $21$ %) where the SPARQL query generated by ChatGPT is an exact match with the one specified in the QALD-9 dataset. This holds true for $45$ % of the questions in our approach. An example of such a question is ‘What is the time zone of Salt Lake City?’

Case 2 (Multiple properties): In many cases (i.e., $41$ %), the SPARQL query generated by ChatGPT is different but represents a correct interpretation of the QALD-9 question. This is the case for $6$ % of the questions in our approach. Consider the question ‘Which airports are located in California, USA?’ and its SPARQL query from the QALD-9 dataset and ChatGPT:

As can be seen, the SPARQL query in QALD-9 uses several UNION operations with four properties (i.e., dbo:location, dbo:city, dbo:isPartOf, and dbo:operator), whereas the SPARQL query generated by ChatGPT contains only one property (i.e., dbo:location), does not use any UNION, yet still provides the correct interpretation of the question.

Case 3 (Different but correct property): There are instances (i.e., $16$ %) where the SPARQL query generated by ChatGPT contains a different property than the QALD-9 query. Consider the question ‘What are the nicknames of San Francisco?’ and its SPARQL query from the QALD-9 dataset and ChatGPT:

As can be seen, the SPARQL query generated by ChatGPT contains the property dbo:spokenIn instead of dbo:language.

Case 4 (Invented KB elements): In other cases (i.e., $8$ %), ChatGPT generates a SPARQL query that includes one or more properties, resources, or classes that do not exist in the corresponding RDF data. Consider the question ‘When did Finland join the EU?’ along with its corresponding SPARQL query in the QALD-9 dataset and the query generated by ChatGPT:

The SPARQL query of ChatGPT contains the property (i.e., dbo:euAccession), which does not exist in DBpedia.

Case 5 (Incorrect SPARQL query): In a few instances (i.e., $3$ %), the SPARQL query generated by ChatGPT fails to accurately represent the intended interpretation of the QALD-9 questions. For example, the SPARQL query generated by ChatGPT fails to capture the interpretation of Boolean questions (i.e., ASK queries that require a Yes/No answer) in QALD-9. Consider the question ‘Are there any castles in the United States?’ and its SPARQL query from the QALD-9 dataset and ChatGPT:

In some cases, ChatGPT generates a structurally incorrect SPARQL query. Consider the QALD-9 question ‘Which countries have places with more than two caves?’ and its SPARQL query from the QALD-9 dataset and ChatGPT:

In this example, the SPARQL query generated by ChatGPT includes a nested SELECT statement, which is structurally incorrect.

Case 6 (Incorrect subject/object/property): In some cases (i.e., $2$ %), the SPARQL query generated by ChatGPT places the object in the subject position and vice versa. Consider the question ‘Butch Otter is the governor of which U.S. state?’ and its SPARQL query from the QALD-9 dataset and ChatGPT:

In this example, the entity res:Butch_Otter represents a person and should be in the object position for the property dbo:governor. However, in the SPARQL query generated by ChatGPT, it appears in the subject position, which is incorrect.

There are also instances (i.e., $2$ %) where the SPARQL query generated by ChatGPT contains a different property than the QALD-9 query. Consider the question ‘Who killed Caesar?’ and its SPARQL query from the QALD-9 dataset and ChatGPT:

As can be seen, the SPARQL query generated by ChatGPT contains the property dbo:deathCause instead of the property-object pair (i.e., dct:subject and dbc:Assassins_of_Julius_Caesar) used in QALD-9. None of these cases are present in our approach.

5.9. Portability to New Datasets and Languages

The model-based approach can be easily adapted to a new RDF dataset and languages. To test the portability, we organized a one-week Hackathon on question answering over RDF data²⁵ and let users port our system to different datasets and languages. We provided a detailed guideline for adding a new dataset and language. The participants with different backgrounds tested the portability of our system for four datasets:

ArCo dataset²⁶ (Carriero et al., 2019) aggregates knowledge of Italian Cultural Heritage through the use of several conceptual ontologies and data from the Italian National Catalogue of Cultural properties. The dataset consists of a network of $7$ vocabularies and $169$ million triples about $820$ thousand cultural entities, providing a SPARQL endpoint to access the data.

Wikidata²⁷ contains $2.9$ billion triples and supports some Indian languages such as Bangla and Tamil, which are not available in DBpedia.

AIFB dataset²⁸ contains information on scientific publications such as publication titles, abstracts, authors, professors, date of publication, place of publication, etc.

LexInfo²⁹ is an ontology that defines data categories for the Lemon model.³⁰

The QA system was ported to all these datasets by the participants of the Hackathon. One group of participants created lexical entries for both English and Italian on three frame types for the ArCo dataset. Another group of participants created lexical entries for three languages (i.e., Italian, English, and Bangla) for Wikidata. As per our knowledge, this is the first question-answering system over linked data for the Bangla language. The third group of participants created lexical entries for English and Spanish for the AIFB dataset and LexInfo.

Table 12 shows the results of porting these four new datasets to the QA system for $5$ languages (i.e., English, Italian, German, Spanish, and Bangla).

Table 12.
The Results of Porting Four New Datasets (i.e., ArCo Carriero et al., 2019, Wikidata, AIFB Dataset, and LexInfo) to the QA System for Five Languages (i.e., English, Italian, German, Spanish, and Bangla).

Porting Time Language NounPP Transitive

DataSet (person-hours) Frame Frame Frame InTransitivePP Total Questions

ArCo $9$ hours English $3$ $4$ $4$ $11$ $648$

Italian $3$ $2$ $2$ $7$ $352$

Wikidata $12$ hours English $15$ $12$ $17$ $44$ $8, 203$

Italian $5$ $7$ $3$ $15$ $5, 043$

Bangla $10$ $9$ $2$ $21$ $3, 043$

AIFB $8$ hours English $13$ $2$ $1$ $16$ $7, 087$

German $5$ $2$ $2$ $9$ $7, 022$

LexInfo $4$ hours English $2$ $2$ $1$ $5$ $13$

Spanish $2$ $1$ $1$ $4$ $15$

	Porting Time	Language	NounPP	Transitive
ArCo	$9$ hours	English	$3$	$4$	$4$	$11$	$648$
		Italian	$3$	$2$	$2$	$7$	$352$
Wikidata	$12$ hours	English	$15$	$12$	$17$	$44$	$8, 203$
		Italian	$5$	$7$	$3$	$15$	$5, 043$
		Bangla	$10$	$9$	$2$	$21$	$3, 043$
AIFB	$8$ hours	English	$13$	$2$	$1$	$16$	$7, 087$
		German	$5$	$2$	$2$	$9$	$7, 022$
LexInfo	$4$ hours	English	$2$	$2$	$1$	$5$	$13$
		Spanish	$2$	$1$	$1$	$4$	$15$

As shown in the table, the approach allows non-expert users to port the question-answering system to a new RDF dataset by adding lexical entries for that dataset in only a few hours and with little training.

5.10. Consistency and Controllability

Our QA system is consistent and controllable because if we add a lexical item to the lexicon, it guarantees that all questions involving a syntactic construction supported by the system in which the lexical item can be used will be parsed correctly. Figure 11 shows the number of questions that can be parsed for each lexical entry and the F $_{1}$ -measure for the QALD-9 training dataset. As can be seen from the Figure, each lexical entry added to the lexicon allows the QA system to parse tens of thousands of questions and so the F $_{1}$ -measure is improved incrementally.

Figure 11.

Consistency: Result of the incremental extension of the QA system for $33$ lexical entries. After each extension of the lexicon, we calculate the number of questions in millions (as shown on the left Y-axis) that can be parsed for each lexical entry and evaluate the F $_{1}$ -measure (as shown on the right Y-axis).

Consider the lexical entry ‘born’ (in) in Figure 11, which is an intransitive verb (i.e., IntransitivePPFrame). The semantics of the entry is captured by the property dbo:birthPlace, the domain of the property is dbo:Place, and the range of the property is dbo:Person.

In Section 2.5, we provide details, along with an example, of how the grammar rule templates (GRT) are obtained from sentence templates. The five GRTs for the intransitive verb of type Agent are presented in two groups (i.e., GRT $_{subject}$ and GRT $_{object}$ ), as shown below:

The grammar rules (GRs) are then automatically generated by instantiating these GRTs using the lexical entry ‘born’ (in), as shown below.

When adding the lexical entry ‘born’ (in) to the lexicon (as shown in Figure 11), the number of parseable questions increased from $3.46$ million to $22.18$ million. As seen from this example, the addition of a single lexical entry to the lexicon enables the QA system to parse tens of thousands of questions and significantly increase the F $_{1}$ -measure.

Another example of consistency and controllability is shown in Table 13, where we illustrate the effects of removing a top-ranked lexical entry, a syntactic frame, and a grammar rule from the lexicon and the grammar. We select the top-ranked element by counting the number of questions covered by an element in the QALD series training datasets (QALD-9, QALD-8, QALD-7, QALD-6, and QALD-5) and ranking them in descending order. Following that, we demonstrate the number of questions that cannot be parsed in our approach when removing these elements from the lexicon, along with the resulting decrease in F $_{1}$ -measure.

Table 13.

The Impact of Removing a Top-ranked Lexical Entry, Grammar Rule (GR), Grammar Rule Template (GRT), and syntactic Frame from the English Lexicon (as Detailed in 3.1).

	# Questions	%	Micro F $_{1}$	Macro F $_{1}$	Micro F $_{1}$	Macro F $_{1}$
			(ind*)	(ind*)	(inc*)	(inc*)
English lexicon	$5416.03$ m	$-$	$0.29$	$0.22$	$0.85$	$0.30$
Remove ‘high’ 5.10.1	$5410.61$ m	$0.1 %$	$0.28$	$0.19$	$0.85$	$0.28$
Remove Rule 5.10.2	$5401.75$ m	$0.26 %$	$0.27$	$0.17$	$0.84$	$0.24$
Remove Template 5.10.3	$4978.06$ m	$8.08 %$	$0.27$	$0.16$	$0.84$	$0.21$
Remove NounPPFrame 5.10.4	$4195.95$ m	$22.52 %$	$0.27$	$0.15$	$0.84$	$0.19$

The evaluation is conducted on the QALD-9 test data set. Here, ind* refers to inductive mode, inc* refers to incremental mode of evaluation, Template* refers to grammar rule template (as detailed in Section 5.10.3), Rule* refers to grammar rule (as detailed in Section 5.10.2), % refers to the number questions that cannot be parse after removing these elements, and m refers to millions. P refers to Precision, R refers to Recall, and F $_{1}$ -Measure refers to F $_{1}$ -Measure.

5.10.1. Impact of Lexical Entry Removal

Consider the top-ranked lexical entry ‘high’ (i.e., AdjectiveSuperlativeFrame) in the QALD series, covering a total of $87$ questions (i.e., $25$ questions in QALD-9, $8$ questions in QALD-8, $9$ questions in QALD-7, $23$ questions in QALD-6, and $22$ questions in QALD-5). The lemon entry has four senses dbo:height, dbo:elevation, dbp:height, and dbp:elevation.

In DBpedia, there are $202, 143$ triples for dbo:height, $247, 825$ triples for dbo:elevation, $6, 863$ triples for dbp:height, and $3605$ triples for dbp:elevation. The grammar rule templates (GRT) for gradable adjectives (i.e., AdjectiveSuperlativeFrame) are detailed in Section 2.5.5. As can be seen from Table 13, a total of $5, 421, 864$ questions (i.e., $0.1$ % questions) cannot be parsed in our approach when removing this lexical entry from the English lexicon.

5.10.2. Impact of Grammar Rule (GR) Removal

To demonstrate the impact of removing a grammar rule, we consider the top-ranked grammar rule in the QALD series. The top-ranked grammar rule is as follows:

As can be seen in Table 13, after removing this rule, approximately $14, 284, 166$ questions (i.e., $0.26$ % of questions) cannot be parsed by the QA system.

5.10.3. Impact of Grammar Rule Template Removal

Furthermore, to demonstrate the consistency and controllability of our approach, we remove the top-ranked GRT in the QALD series. The template is shown below:

The grammar rule generated by instantiating the template with the lexical entry of relative (e.g., ‘death date’ (of)) is shown below:

Therefore, the template can be instantiated by a total of $189$ lexical entries (e.g., ’capital’ (of), ’currency’ (of), ’death date’ (of), ’birth date’ (of), etc.) out of $311$ (as shown in Table 1). As seen in Table 13, after removing this template, approximately $437, 978, 324$ questions (i.e., $8.08$ % questions) cannot be parsed by the QA system.

5.10.4. Impact of Syntactic Frame Removal

We repeated the experiment for the top-ranked syntactic frame (i.e., NounPPFrame) in the QALD series. An example of a lexical entry (detailed in Section 2.3.1), GRTs, and grammar rules for NounPPFrame are detailed in Section 2.5.1.

There are a total of $44$ GRTs (as shown in Table 1), and our approach generates a total of $8, 708$ grammar rules by instantiating these templates using lexical entries for this frame.

There are $311$ lexical entries and $31, 284, 166$ triples containing the properties associated with these lexical entries in DBpedia. As seen in Table 13, a total of $1, 220, 082, 474$ questions (i.e., $22.52$ % questions) cannot be parsed in our approach when removing the NounPPFrame.

In contrast, for Machine Learning (ML) systems in general, it is unclear what the impact of adding one example is in terms of the behaviour of the system. Usually, several examples of the same or similar type need to be provided for the ML system to learn a pattern, leading to high redundancy in training data.

5.11. Discussion

Most work on question answering over linked data currently builds on systems that use machine learning to learn a model to map NLQs into SPARQL. Our approach has the following advantages over these systems:

These systems can not be extended to cover new types of questions in a principled way. Our approach overcomes the limitation in a more principled way by simply adding a new syntactic construction to the system, which takes only $10$ to $25$ minutes depending on the language. For example, adding a sentence template³¹ that supports sentences such as ‘What is Angela Merkel’s date of birth?’, etc., will enable the QA system to parse an additional $19.87$ million questions. Furthermore, we demonstrate the consistency and controllability of our approach by removing the top-ranked GRT obtained from the sentence template in the QALD series, as illustrated in Table 13 in Section 5.10.

Another limitation of machine learning approaches is that the addition of new training questions involving a new lexical element does not guarantee that the system can induce the semantics of that lexical element correctly. For example, adding a lexical entry for InTransitivePPFrame, for example,‘born’ (in), will make $18.72$ million questions for queries that involve dbo:birtPlace parsable.

Further, the machine learning-based approaches follow an inductive paradigm to port the system to a new language. The paradigm is costly and time-consuming since it requires adding training data for that language. In our approach, porting the system to a new language involves two steps: (i) creating lexical entries and (ii) creating sentence templates considering the features of that particular language. For example, we ported the system to German by adding lexical entries for all syntactic frames considering features of the German language such as the gender and case information for nouns in Lemon format (as shown in Section 3.2.1). We also ported the system to Italian (Section 3.2.2) and Spanish (Section 3.2.3) and showed that we only needed to add language-specific properties to the lexicon and adapted the sentence templates with respect to the features of these languages, but the process remains the same.

The machine learning-based approaches, for example, Zhang et al., 2016b, rely on algorithms having a large number of learning parameters and require a significant amount of training data. Creating a new training dataset for the new dataset is very expensive (Berant et al., 2013). Finally, our approach operates independently of training data. Instead, the capabilities of our system can be increased by extending the lexicon, which is a completely different paradigm. This allows us to improve our system step by step (as detailed in Section 5.3).

In the ’Hackathon on question answering over RDF data’ event, the participants were able to create a basic version of the QA system for the ArCo dataset (i.e., Italian Cultural Heritage linked data) (Carriero et al., 2019) for English and Italian in $9$ person-hours, for the Wikidata dataset for English, Italian, and Bangla in $12$ person-hours, for the AIFB dataset for English and German in $8$ person-hours, and for the Lexinfo dataset for English and Spanish in $4$ person-hours (as shown in Section 5.9).

Vollmers et al. (2021b) presented a question-answering over knowledge graph (KGQA) approach that requires less effort to create training data compared to the state-of-art machine learning approaches as it uses existing benchmarks as training data. The approach learns templates from existing benchmarks (in particular, QALD-8 and QALD-9) by employing machine learning and feature engineering to learn to classify NLQs into isomorphic basic graph patterns. A disadvantage of the approach is that it depends heavily on benchmarks which limit the system to the language and RDF dataset of the existing benchmarks. Moreover, the research in the field of KGQA suffers from the lack of multilingual benchmarks (Loginova et al., 2021; Neves et al., 2020), and only a few benchmarks (i.e., QALD series³² (Usbeck et al., 2018), LC-QuAD³³ (Dubey et al., 2019), CWQ (Cui et al., 2022), RuBQ (Rybin et al., 2021), etc.) exist that involve multiple languages. In contrast, our approach can be adapted to a new language without the availability of benchmarks. For example, to the best of our knowledge, there are no benchmarks available for the Bangla language, yet the participants of the ’Hackathon on question answering over RDF data’ event were able to create a basic version of the QA system for Wikidata for the Bangla language in $4$ person-hours.

There are also QALD that exploit large language models (Banerjee et al., 2023; Ouyang et al., 2022). One promising system in this approach is ChatGPT (Faria et al., 2023; Tan et al., 2023). Our approach outperforms ChatGPT in zero-shot scenarios across four languages, in both inductive and incremental modes. However, in the few-shot scenario, ChatGPT outperforms our approach in the inductive mode. It is unclear whether the QALD-9 dataset was part of the data used to train ChatGPT. Nevertheless, our approach outperforms ChatGPT in the few-shot scenario in the incremental evaluation mode.

We compare how ChatGPT performs in comparison to our approach and outline the cases where our approach performs better. As an example, when evaluating with the QALD benchmark (i.e., QALD-9 dataset), we found that only a few instances (i.e., $21$ %) have the SPARQL query generated by ChatGPT as an exact match with the one specified in the QALD-9 dataset. This holds for $45$ % of the questions in our approach. Moreover, when using ChatGPT, the generated SPARQL query from a NLQ may contain invented KB elements, such as resources, properties, or classes (as detailed in Section 5.8), that do not exist in the RDF data. We have provided several examples of this case using DBpedia.

Besides systems using machine learning techniques, there are rule-based systems that rely on a set of rules to map natural language (NL) questions into a representation that can be either directly evaluated over the knowledge graph or rely on graph exploration to map the representation of the NL question into a full-fledged SPARQL query. Although no training data is required by the rule-based systems, there are limitations to porting these systems to a new language. The rule-based systems and their limitations compared to our approach are as follows.

To the best of our knowledge, WDAqua-core1 (Diefenbach et al., 2020a) is the only QA system that is implemented for languages other than English such as German, Spanish, French, and Italian and ported for several RDF datasets (Diefenbach et al., 2020b). For lexicalizations, WDAqua-core1 maps n-grams (of natural language sentences) to KB elements by comparing an n-gram with an entity’s rdfs:label, which is limited to the languages available in the dataset. Moreover, there is a well-known lexical gap between content expressed in the form of natural language (NL) texts and content stored in an RDF KB.

Similar to WDAqua-core1, some approaches (Gashkov et al., 2022; Perevalov et al., 2022c) use rdfs:label to bridge the lexical gap between natural language terms and KB elements, enhancing the quality of lexicalization by analyzing SPARQL queries and determining their correctness with respect to the given question. This correction is performed using the VANiLLa Dataset (Magnini et al., 2002), which contains both correct and incorrect answers to questions. However, the dataset is limited to English and Spanish. Furthermore, the approach was not evaluated with respect to the QALD benchmark, leaving its effectiveness in the question-answering system unknown.

In contrast, in our approach, the Lemon model of lexicalization (detailed in Section 2.5) is not limited to any language, and the questions are more natural as the lexicalization is provided by a language expert. For example, consider the lexical entry ‘high’ (as shown in Figure 8); the semantics of this gradable adjective is captured with respect to the property dbo:elevation. However, such lexicalization is not possible when considering the WDAqua-core1 approach, which uses rdfs:label for mapping natural language terms to KB elements.

WDAqua-core1 is based on the observation that many questions can be understood from the semantics of the words but the syntax of the question has less importance. The authors claimed that the system is adaptable to any language by changing only the stop word removal and stemming, but provide no evidence of how language language-specific features (i.e., gender information, grammatical cases, word order, verb types, etc.) are incorporated into their approach. In contrast, we create a lexicon for each language (as detailed in Section 3) by incorporating language-specific features in Lemon format and sentence templates.

Similar to WDAqua-core1, there are pattern-based approaches (Abujabal et al., 2017; Rico et al., 2015; Steinmetz & Ann-Katrin, 2019) that make use of a number of patterns (i.e., a pattern library) that have been defined to map natural language to SPARQL queries. But these works are either in the preliminary stage (Rico et al., 2015) and provide only two patterns as proof of concept or have been developed for the English language (Steinmetz & Ann-Katrin, 2019) only. We adopted the underlying idea of using patterns but we use automatically generated grammar rules, which can be easily extended to another language and RDF dataset.

SemGraphQA (Beaumont et al., 2015), gAnswer (Zou et al., 2014), and Xser (Xu et al., 2014) use external tools (i.e., parts-of-speech taggers, dependency parsers, semantic parsers, etc.) and other approaches such as AskNow (Dubey et al., 2016) and DEANNA (Yahya et al., 2012) use language resources or dictionaries (such as WordNet, BOA, etc.) to map natural language terms to KB elements. Both the language tools and resources are limited to a few languages and it is very difficult (and time-consuming) to adapt them to a new language. In contrast, our approach is independent of any such tool or external dictionary and porting to a new language is straightforward by creating lexical entries (as explained in Section 3) and by writing sentence templates for that language.

Creating a question answering system on top of a new RDF dataset is still challenging and cumbersome (Diefenbach et al., 2018). The main open challenge remains portability (Diefenbach et al., 2020b), that is, the ability to easily apply a QA algorithm to new and previously untested RDF datasets. Qanary (Both et al., 2016) achieved limited portability to a new dataset by integrating a consistent number of tools. Diefenbach et al. (2020b) provided a QA system and argued that the system can be ported to a new RDF dataset with minimal investments, but provide no evidence of how much time it requires to port to a new dataset. Our approach allows us to port our system to a wide range of RDF datasets in hours, not days (as detailed in Section 5.9).

6. Conclusion

We present a multilingual question-answering system that relies on a model of the lexicon-ontology interface in which the meaning of lexical entries is specified with respect to a given vocabulary or ontology. More specifically, our model-based approach automatically generates a lexicalized grammar given a lexicon and sentence templates in a particular language and given an RDF dataset. We present the high-level architecture of our multilingual QA system.

Following this, we describe the automatically generated grammar for five syntactic frames at two levels of abstraction. First, we describe GRTs that are independent of specific lexical entries. Secondly, we provide example instantiations of these templates using lexical entries for illustration. We also include SPARQL queries to retrieve the values of non-terminals in the grammar rules. Further, we demonstrate the methods and implementations of creating a lexicon for English, also adapting the system to other languages by creating lexical entries and sentence templates for these languages.

To compare with machine learning-based approaches, we evaluated our approach in two modes: inductive and incremental, showing that our approach outperforms all state-of-the-art approaches in the incremental evaluation mode. So far there is no system described in the literature that works for four languages (i.e., English, German, Italian, and Spanish) while reaching state-of-the-art performance on all of them for QALD-9 benchmarks and outperforming current QA systems.

A necessary prerequisite for our grammar generation approach is the availability of a Lemon lexicon for each supported language, which is manually created in our approach. Previous work (Elahi et al., 2023) has proposed a method for the fully automatic creation of a Lexicon for QA systems using LexExMachina (Ell et al., 2021) that uses association rules to identify correspondences between lexical elements on the one hand and ontological vocabulary elements on the other hand. Although this method significantly reduces the human effort involved in creating a lexicon for a language and linked data, it causes a $34$ % to $56$ % performance (F $_{1}$ -measure on QALD benchmark) degradation with respect to a manually created lexicon. In future work, we will extend this automatic creation of a lexicon for multilingual settings and enhance the performance in comparison to state-of-the-art QALD systems.

Furthermore, our approach can handle a limited form of composition that allows us to nest some rules into others. In future work, we will investigate how to implement a more principled approach to semantic composition by Flexible Semantic Composition with DUDES (Cimiano, 2009). We also intend to enable the community to contribute to and adapt the grammar generation to other languages. More specifically, we intend to cover Indian and Arabic languages, as there is no QA system for an RDF dataset described in the literature that works for these languages. In this research, we focus on an established task—question answering over linked data (Diefenbach et al., 2018; Unger et al., 2014)—which does not involve semi-structured or unstructured data.This task gained popularity through the QALD challenge, established in 2012, which has undergone 10 iterations. In future work, we will investigate applying the methodology to semi-structured or unstructured data.

Footnotes

Acknowledgments

The work described here is supported by “Linked-data aware NLP services (WG2)” of the Nexus Linguarum () as well as “Lexical and conceptual linking (WG3)” of Prêt-à-LLOD (the European Commission under grant 825182). B. Ell has been partially supported by the SIRIUS centre: Norwegian Research Council project No 237898 and Integreat—Norwegian Centre for knowledge-driven machine learning, project number 332645. Thanks to the participants of the ‘‘Hackathon on question answering over RDF data.’’ Special thanks to participant Japesh Methuku from ADAPT, the world-leading SFI Research Centre for AI-Driven Digital Content Technology in Ireland, for actively participating in the Hackathon and contributing to the system. The authors also extend our appreciation to participants Gennaro Nolano and Maria Pia di Buono from the University of Naples ’L’Orientale,’ Italy, for their valuable contributions to the Italian language. Additionally, they want to acknowledge and thank Wasim Mahmud Surjo from BRAC University, Bangladesh, and Subhana Mahmud Toshi from Arab Open University, Bahrain, for their contributions to the Bangla language.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

ORCID iDs

Mohammad Fazleh Elahi

Basil Ell

Gennaro Nolano

Philipp Cimiano

Notes

Appendix A. SPARQL Queries for Grammar Rules of Lexicon Entries of LexInfo Frames

References

Abujabal

Yahya

Riedewald

Weikum

(2017). Automated template generation for question answering over knowledge graphs. In Proceedings of the 26th world wide web conference (WWW) (pp. 1191–1200). WWW’17. International World Wide Web Conferences Steering Committee. ISBN 9781450349130. https://doi.org/10.1145/3038912.3052583

Baker

C. F.

Fillmore

C. J.

Lowe

J. B.

(1998). The Berkeley Framenet Project. In 36th annual meeting of the association for computational linguistics, and 17th international conference on computational linguistics (Vol. 1, pp. 86–90). Association for Computational Linguistics. https://doi.org/10.3115/980845.980860

Banerjee

Nair

P. A.

Usbeck

Biemann

(2023). Gett-qa: Graph embedding based t2t transformer for knowledge graph question answering. In Proceedings of the 20th European semantic web conference (ESWC) (p. 279–297). Springer-Verlag. https://doi.org/10.1007/978-3-031-33455-9_17

Barz

Sonntag

(2019). Incremental improvement of a question answering system by re-ranking answer candidates using machine learning. In International workshop on spoken dialogue systems technology. https://doi.org/10.48550/arXiv.1908.10149

Baudiš

Šedivý

(2015). Modeling of the question answering task in the YodaQA system. In Proceedings of the 6th international conference on experimental ir meets multilinguality, multimodality, and interaction—Volume 9283 (p. 222–228). CLEF’15. Springer-Verlag. https://doi.org/10.1007/978-3-319-24027-5_20

Beaumont

Grau

Ligozat

A. L.

(2015). SemGraphQA@QALD-5. In LIMSI participation at QALD-5@CLEF. CLEF.

Benz

Cimiano

Elahi

M. F.

Ell

(2020). Generating grammars from lemon lexica for questions answering over linked data: A preliminary analysis. In Proceedings of the 6th natural language interfaces for the web of data workshop (NLIWOD) co-located with the 19th International Semantic Web Conference (ISWC), CEUR Workshop Proceedings (Vol. 2722, pp. 40–55). https://doi.org/10.5281/zenodo.6641631

Berant

Chou

Frostig

Liang

(2013). Semantic parsing on Freebase from question-answer pairs. In D. Yarowsky, T. Baldwin, A. Korhonen, K. Livescu, & S. Bethard (Eds.), Proceedings of the 2013 conference on empirical methods in natural language processing (EMNLP) (pp. 1533–1544). Association for Computational Linguistics.

Berners-Lee

(2006). Linked data—Design issues. W3C (09/20).

10.

Bizer

Heath

Berners-Lee

(2023). Linked data - the story so far, 1 Ed. Association for Computing Machinery.

11.

Bordes

Chopra

Weston

(2014). Question answering with subgraph embeddings. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 615–620). Association for Computational Linguistics. https://doi.org/10.3115/v1/D14-1067

12.

Both

Diefenbach

Singh

Shekarpour

Cherix

Lange

(2016). Qanary—A methodology for vocabulary-driven open question answering systems. In Proceedings of the 13th international conference on the semantic web. Latest advances and new domains (Vol. 9678, pp. 625–641). Springer-Verlag. https://doi.org/10.1007/978-3-319-34129-3_38

13.

Burtsev

Seliverstov

Airapetyan

Arkhipov

Baymurzina

Bushkov

Gureenkova

Khakhulin

Kuratov

Kuznetsov

Litinsky

Logacheva

Lymar

Malykh

Petrov

Polulyakh

Pugachev

Sorokin

Vikhreva

Zaynutdinov

(2018). DeepPavlov: Open-source library for dialogue systems. In F. Liu & T. Solorio (Eds.), Proceedings of the 56th annual meeting of the Association for Computational Linguistics (ACL) (pp. 122–127). Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-4021

14.

Carriero

V. A.

Gangemi

Mancinelli

M. L.

Marinucci

Nuzzolese

A. G.

Presutti

Veninata

(2019). ArCo: The Italian cultural heritage knowledge graph. In Proceedings of the 18th international semantic web conference (ISWC), Lecture Notes in Computer Science (Vol. 11779, pp. 36–52). Springer International Publishing. https://doi.org/10.1007/978-3-030-30796-7_3

15.

Chakraborty

Lukovnikov

Maheshwari

Trivedi

Lehmann

Fischer

(2019). Introduction to neural network based approaches for question answering over knowledge graphs. WIREs Data Mining and Knowledge Discovery, 11(3), e13891–e138925. https://doi.org/10.1002/widm.1389

16.

Cheng

Reddy

Saraswat

Lapata

(2019). Learning an executable neural semantic parser. Computational Linguistics, 45(1), 59–94. https://doi.org/10.1162/coli_a_00342

17.

Cimiano

(2009). Flexible semantic composition with DUDES. In H. Bunt (Ed.), Proceedings of the 8th international conference on computational semantics (IWCS) (pp. 272–276). Association for Computational Linguistics.

18.

Cimiano

Buitelaar

McCrae

J. P.

Sintek

(2011). Lexinfo: A declarative model for the lexicon-ontology interface. Journal of Web Semantics, 9(1), 29–51. https://doi.org/10.1016/j.websem.2010.11.001

19.

Cimiano

Lopez

Unger

Cabrio

Ngomo

A. C. N.

Walter

(2013). Intui2: A prototype system for question answering over linked data. In Proceedings of the question answering over linked data lab (QALD-3) at CLEF.

20.

Cimiano

McCrae

J. P.

Buitelaar

(2016). Lexicon model for ontologies: Community report. In W3C Community group final report.

21.

Conneau

Khandelwal

Goyal

Chaudhary

Wenzek

Guzmán

Grave

Ott

Zettlemoyer

Stoyanov

(2020). Unsupervised cross-lingual representation learning at scale. In D. Jurafsky, J. Chai, N. Schluter, & J. Tetreault (Eds.), Proceedings of the 58th annual meeting of the Association for Computational Linguistics (ACL) (pp. 8440–8451). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.747

22.

Conneau

Lample

(2019). Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’ Alché-Buc, E. Fox, & R. Garnett (Eds.), Proceedings of the advances in neural information processing systems (NeurIPS) (Vol. 32). Curran Associates, Inc.

23.

Cui

Aralikatte

Lent

Hershcovich

(2022). Compositional generalization in multilingual semantic parsing over Wikidata. Transactions of the Association for Computational Linguistics, 10, 937–955. https://doi.org/10.1162/tacl_a_00499

24.

Diefenbach

Both

Singh

Maret

(2020a). Towards a question answering system over the semantic web. Semantic Web, 11(3), 421–439. https://doi.org/10.1007/s10115-017-1100-y

25.

Diefenbach

Giménez-García

Both

Singh

Maret

(2020b). Qanswer kg: Designing a portable question answering system over RDF data. In Proceedings of the 17th European semantic web conference (ESWC) (pp. 429–445). Springer-Verlag. https://doi.org/10.1007/978-3-030-49461-2_25

26.

Diefenbach

López

Singh

K. D.

Maret

(2018). Core techniques of question answering systems over knowledge bases: A survey. Knowledge and Information Systems, 55(3), 529–569. https://doi.org/10.1007/s10115-017-1100-y

27.

Ding

Meng

Chen

Liao

Han

Tresp

(2023). Forecasttkgquestions: A benchmark for temporal question answering and forecasting over temporal knowledge graphs. In: Proceedings of the 22nd international semantic web conference (ISWC) (pp. 541–560). Springer Nature Switzerland.

28.

Dubey

Banerjee

Abdelkawi

Lehmann

(2019). LC-QuAD 2.0: A large dataset for complex question answering over wikidata and dbpedia. In Proceedings of the 18th international semantic web conference (ISWC) (pp. 69–78). Springer.

29.

Dubey

Dasgupta

Sharma

Höffner

Lehmann

(2016). Asknow: A framework for natural language query formalization in SPARQL. In Proceedings of the 13th International conference on the semantic Web: Latest advances and new domains, Lecture Notes in Computer Science (Vol. 9678, pp. 300–316). Springer International Publishing. https://doi.org/10.1007/978-3-319-34129-3_19

30.

ebn veyseh

A. P.

(2016). Cross-lingual question answering using common semantic space. In T. Chakraborty, M. Riedl, & V. Vydiswaran (Eds.), Proceedings of TextGraphs-10: The workshop on graph-based methods for natural language processing (pp. 15–19). Association for Computational Linguistics. https://doi.org/10.18653/v1/W16-1403

31.

Elahi

M. F.

Ell

Cimiano

(2023). LexExMachinaQA: A framework for the automatic induction ofontology lexica for question answering over linked data. In S. Carvalho, A. F. Khan, A. O. Anić, B. Spahiu, J. Gracia, J. P. McCrae, D. Gromann, B. Heinisch, & A. Salgado (Eds.), Proceedings of the 4th conference on language, data and knowledge (LDK) (pp. 207–218). NOVA CLUNL, Portugal.

32.

Elahi

M. F.

Ell

Grimm

Cimiano

(2021). Question answering on RDF data based on grammars automatically generated from lemon models. In I. Tiddi, M. Maleshkova, T. Pellegrini, & V. de Boer (Eds.), Proceedings of the 17th international conference on semantic systems (SEMANTiCS), CEUR Workshop Proceedings (Vol. 2941). CEUR-WS.org.

33.

Ell

Elahi

M. F.

Cimiano

(2021). Bridging the gap between ontology and lexicon via class-specific association rules mined from a loosely-parallel text-data corpus. In D. Gromann, G. Sérasset, T. Declerck, J. P. McCrae, J. Gracia, J. Bosque-Gil, F. Bobillo, & B. Heinisch (Eds.), Proceedings of the 3rd conference on language, data and knowledge (LDK), Open Access Series in Informatics (OASIcs) (Vol. 93, pp. 33:1–33:21). Schloss Dagstuhl – Leibniz-Zentrum für Informatik. https://doi.org/10.4230/OASIcs.LDK.2021.33

34.

Faria

Perdigão

Gonçalo Oliveira

(2023). Question answering over linked data with GPT-3. In A. Simões, M. M. Berón, & F. Portela (Eds.), 12th Symposium on languages, applications and technologies (SLATE 2023), Open Access Series in Informatics (OASIcs) (Vol. 113, pp. 1:1–1:15). Dagstuhl, Germany: Schloss Dagstuhl – Leibniz-Zentrum für Informatik. https://doi.org/10.4230/OASIcs.SLATE.2023.1

35.

Fellbaum

(1998). WordNet: An electronic lexical database. MIT Press.

36.

Ferrández

Ó.

Spurk

Kouylekov

Dornescu

Ferrández

Negri

Izquierdo

Tomás

Orasan

Neumann

Magnini

Vicedo

J. L.

(2011). The QALL-ME framework: A specifiable-domain multilingual question answering architecture. Journal of Web Semantics. https://doi.org/10.2139/ssrn.3199513.

37.

Gangemi

Alam

Asprino

Presutti

Recupero

D. R.

(2016). Framester: A wide coverage linguistic linked data hub. In Proceedings of the 20th international conference on knowledge engineering and knowledge management, Lecture Notes in Computer Science (Vol. 10024, pp. 239–254).

38.

Gashkov

Perevalov

Eltsova

Both

(2022). Improving question answering quality through language feature-based SPARQL query candidate validation. In Proceedings of the 19th European semantic web conference (ESWC) (pp. 217–235).

39.

Giannone

Bellomaria

Basili

(2013). A hmm-based approach to question answering against linked data. In Proceedings of the question answering over linked data lab (QALD-3) at CLEF (2013).

40.

Glaese

McAleese

Trbacz

Aslanides

Firoiu

Ewalds

Rauh

Weidinger

Chadwick

Thacker

(2022). Improving alignment of dialogue agents via targeted human judgements. ArXiv abs/2209.14375.

41.

Gracia

Montiel-Ponsoda

Cimiano

Gómez-Pérez

Buitelaar

McCrae

(2012). Challenges for the multilingual web of data. Journal of Web Semantics, 11, 63–71. https://doi.org/10.1016/j.websem.2011.09.001

42.

Guyonvarc’H

Ferré

(2013). Scalewelis: A Scalable Query-based Faceted Search System on Top of SPARQL Endpoints. In E. Cabrio, P. Cimiano, V. Lopez, A. C. N. Ngomo, C. Unger, & S. Walter (Eds.), Work. Multilingual question answering over linked data (QALD-3).

43.

Hakimov

Jebbara

Cimiano

(2017). AMUSE: Multilingual semantic parsing for question answering over linked data. In Proceedings of the 16th international semantic web conference (ISWC) (pp. 329–346). Springer-Verlag. https://doi.org/10.1007/978-3-319-68288-4_20

44.

Zhang

Liu

Zhao

(2014). Casia@ v2: A mln-based question answering system over linked data. In Conference and labs of the evaluation forum.

45.

Shu

Huang

(2021). Edg-based question decomposition for complex question answering over knowledge bases. In Proceedings of the 20th international semantic web conference (ISWC) (pp. 128–145). Springer International Publishing.

46.

Jebbara

Cimiano

(2019). Zero-shot cross-lingual opinion target extraction. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics: Human language technologies (pp. 2486–2495). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1257

47.

Joshi

A. K.

Schabes

(1997). Tree-adjoining grammars, vol. 3, Pp. 69–123. Springer Berlin Heidelberg.

48.

Jurafsky

Martin

J. H.

(2025). Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition with language models (3rd ed.). Pearson Prentice Hall. Online manuscript released January 12, 2025.

49.

Kapanipathi

Abdelaziz

Ravishankar

Roukos

Gray

Fernandez Astudillo

Chang

Cornelio

Dana

Fokoue

Garg

Gliozzo

Gurajada

Karanam

Khan

Khandelwal

Lee

Y. S.

Luus

..., Yu

(2021). Leveraging Abstract Meaning Representation for knowledge base question answering. In C. Zong, F. Xia, W. Li, & R. Navigli (Eds.), Findings of the association for computational linguistics: ACL-IJCNLP 2021 (pp. 3884–3894). Association for computational linguistics. https://doi.org/10.18653/v1/2021.findings-acl.339

50.

Kipper

Korhonen

Ryant

Palmer

(2008). A large-scale classification of English verbs. Language Resources and Evaluation, 42, 21–40. https://doi.org/10.1007/s10579-007-9048-2

51.

Klinger

Cimiano

(2014). The USAGE review corpus for fine grained multi lingual opinion analysis. In N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, & S. Piperidis (Eds.), Proceedings of the 9th international conference on language resources and evaluation (LREC) (pp. 2211–2218). European Language Resources Association (ELRA).

52.

Klinger

Cimiano

(2015). Instance selection improves cross-lingual model training for fine-grained sentiment analysis. In Proceedings of the 19th conference on computational natural language learning (pp. 153–163). Association for Computational Linguistics. https://doi.org/10.18653/v1/K15-1016

53.

Kwiatkowski

Choi

Artzi

Zettlemoyer

(2013). Scaling semantic parsers with on-the-fly ontology matching. In D. Yarowsky, T. Baldwin, A. Korhonen, K. Livescu, & S. Bethard (Eds.), Proceedings of the 2013 conference on empirical methods in natural language processing (pp. 1545–1556). Association for Computational Linguistics.

54.

Lehmann

Isele

Jakob

Jentzsch

Kontokostas

Mendes

P. N.

Hellmann

Morsey

van Kleef

Auer

Bizer

(2015). DBpediaA large-scale, multilingual knowledge base extracted from Wikipedia. Semantic Web Journal, 6(2), 167–195. https://doi.org/10.3233/SW-140134

55.

Liang

Berant

Forbus

K. D.

Lao

(2017). Neural symbolic machines: Learning semantic parsers on Freebase with weak supervision. In R. Barzilay & M. Y. Kan (Eds.), Proceedings of the 55th annual meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Vol. 1, pp. 23–33). Association for Computational Linguistics. https://doi.org/10.18653/v1/P17-1003

56.

Lichte

Kallmeyer

(2017). Tree-adjoining grammar: A tree-based constructionist grammar framework for natural language understanding. In The AAAI 2017 Spring symposium on computational construction grammar and natural language understanding. AAAI Press.

57.

Loginova

Varanasi

Neumann

(2021). Towards end-to-end multilingual question answering. Information Systems Frontiers, 23(1), 227–241. https://doi.org/10.1007/s10796-020-09996-1

58.

Lukovnikov

Fischer

Lehmann

(2019). Pretrained transformers for simple question answering over knowledge graphs. In C. Ghidini, O. Hartig, M. Maleshkova, V. Svátek, I. F. Cruz, A. Hogan, J. Song, M. Lefrançois, & F. Gandon (Eds.), Proceedings of the 18th international semantic web conference (ISWC), Lecture Notes in Computer Science (Vol. 11778, pp. 470–486). Springer.

59.

Lukovnikov

Fischer

Lehmann

Auer

(2017). Neural network-based question answering over knowledge graphs on word and character level. In Proceedings of the 26th international conference on World Wide Web (WWW), WWW’17 (pp. 1211–1220). International World Wide Web Conferences Steering Committee. https://doi.org/10.1145/3038912.3052675

60.

Luz

F. F.

Finger

(2018). Semantic parsing natural language into SPARQL: Improving target language representation with neural attention. ArXiv abs/1803.04329.

61.

Magnini

Negri

Prevete

Tanev

(2002). Is it the right answer? exploiting web redundancy for answer validation. In P. Isabelle, E. Charniak, & D. Lin (Eds.), Proceedings of the 40th annual meeting of the Association for Computational Linguistics (pp. 425–432). Association for Computational Linguistics. https://doi.org/10.3115/1073083.1073154

62.

McCrae

J. P.

Quattri

Unger

Cimiano

(2014). Modelling the semantics of adjectives in the ontology-lexicon interface. In M. Zock, R. Rapp, & C. R. Huang (Eds.), Proceedings of the 4th workshop on cognitive aspects of the lexicon (CogALex) (pp. 198–209). Association for Computational Linguistics and Dublin City University. https://doi.org/10.3115/v1/W14-4724

63.

McCrae

J. P.

Spohr

Cimiano

(2011). Linking lexical resources and ontologies on the semantic web with lemon. In G. Antoniou, M. Grobelnik, E. Simperl, B. Parsia, D. Plexousakis, P. De Leenheer, & J. Pan (Eds.), Proceedings of the 8th extended semantic web conference on The semantic web: Research and applications (ESWC) (Vol. 6643, pp. 245–259). Springer Berlin Heidelberg.

64.

Mihindukulasooriya

Rossiello

Kapanipathi

Abdelaziz

Ravishankar

Roukos

A. G. S.

Gray

(2020). Leveraging semantic parsing for relation linking over knowledge bases. In Proceedings of the 19th international semantic web conference (ISWC) (pp. 402–419). Springer-Verlag. https://doi.org/10.1007/978-3-030-62419-4_23

65.

Mohammed

Shi

Lin

(2018). Strong baselines for simple question answering over knowledge graphs with and without neural networks. In M. Walker, H. Ji, & A. Stent (Eds.), Proceedings of the 2018 conference of the North American chapter of the Association for Computational Linguistics: Human language technologies (Vol. 2, pp. 291–296). Association for Computational Linguistics. https://doi.org/10.18653/v1/N18-2047

66.

Neelakantan

Q. V.

Abadi

McCallum

Amodei

(2017). Learning a natural language interface with neural programmer. In International conference on learning representations, volume abs/1611.08945.

67.

Neves

Lamúrias

Couto

F. M.

(2020). Biomedical question answering using extreme multi-label classification and ontologies in the multilingual panorama. In Proceedings of the workshop on semantic indexing and information retrieval for health from heterogeneous content types and languages co-located with 42nd European conference on information retrieval, CEUR Workshop Proceedings (Vol. 2619).

68.

Nolano

Elahi

M. F.

di Buono

M. P.

Ell

Cimiano

(2022). An Italian Question Answering System based on grammars automatically generated from ontology lexica. In Proceedings of the 18th italian conference on computational linguistics (CLiC-it).

69.

Omar

Dhall

Kalnis

Mansour

(2023). A universal question-answering platform for knowledge graphs. In Proceedings of the ACM SIGMOD/PODS international conference of management of data (Vol. 1). Association for Computing Machinery. https://doi.org/10.1145/3588911

70.

Ouyang

Jiang

Almeida

Wainwright

Mishkin

Zhang

Agarwal

Slama

Gray

Schulman

Hilton

Kelton

Miller

Simens

Askell

Welinder

Christiano

Leike

Lowe

(2022). Training language models to follow instructions with human feedback. In Proceedings of the 36th international conference on neural information processing systems, NIPS ’22. Curran Associates Inc.

71.

Perevalov

Both

Diefenbach

Ngonga Ngomo

A. C.

(2022a). Can machine translation be a reasonable alternative for multilingual question answering systems over knowledge graphs? In Proceedings of the ACM web conference 2022, WWW ’22 (pp. 977–986). Association for Computing Machinery. https://doi.org/10.1145/3485447.3511940

72.

Perevalov

Both

Diefenbach

Ngonga Ngomo

A. C.

(2022b). Can machine translation be a reasonable alternative for multilingual question answering systems over knowledge graphs? In Proceedings of the ACM web conference 2022, WWW ’22 (pp. 977–986). Association for Computing Machinery. https://doi.org/10.1145/3485447.3511940

73.

Perevalov

Both

Ngomo

A. C. N.

(2018). Multilingual question answering systems for knowledge graphs—A survey. Journal of Web Semantics, 9(1), 29–51. https://doi.org/10.3233/SW-243633

74.

Perevalov

Gashkov

Eltsova

Both

(2022c). Towards knowledge graph-agnostic SPARQL query validation for improving question answering. In Proceedings of the 19th European semantic web conference (ESWC) (pp. 78–82). Springer-Verlag. https://doi.org/10.1007/978-3-031-11609-4_15

75.

Perevalov

Ngonga

A. C.

Both

(2022d). Enhancing the accessibility of knowledge graph question answering systems through multilingualization. In Proceedings of the 16th international conference on semantic computing (ICSC) (pp. 251–256).

76.

Petroni

Rocktäschel

Riedel

Lewis

Bakhtin

Miller

(2019). Language models as knowledge bases? In K. Inui, J. Jiang, V. Ng, & X. Wan (Eds.), Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th International joint conference on natural language processing (EMNLP-IJCNLP) (pp. 2463–2473). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1250

77.

Pradel

Haemmerlé

Hernandez

(2012). A semantic web interface using patterns: The swip system. In M. Croitoru, S. Rudolph, N. Wilson, J. Howse, & O. Corby (Eds.), Graph structures for knowledge representation and reasoning (pp. 172–187). Springer Berlin Heidelberg.

78.

Rico

Unger

Cimiano

(2015). Sorry, i only speak natural language: A pattern-based, data-driven and guided approach to mapping natural language queries to SPARQL. In Proceedings of the 4th international workshop on intelligent exploration of semantic data (IESD) co-located with the 14th International Semantic Web Conference (ISWC).

79.

Rossiello

Mihindukulasooriya

Abdelaziz

Bornea

Gliozzo

Naseem

Kapanipathi

(2021). Generative relation linking for question answering over knowledge bases. In Proceedings of the 20th International semantic web conference (ISWC) (pp. 321–337). Springer International Publishing. https://doi.org/10.1007/978-3-030-88361-4_19

80.

Ruseti

Mirea

Rebedea

Trausan-Matu

(2015). QALD challenge and the ‘YodaQA system’. In Prototype Notes.

81.

Rybin

Korablinov

Efimov

Braslavski

(2021). Rubq 2.0: An innovated Russian question answering dataset. In Proceedings of the 18th European semantic web conference (ESWC) (pp. 532–547). Springer, Springer-Verlag. https://doi.org/10.1007/978-3-030-77385-4_32

82.

Sakor

Mulang’

I. O.

Singh

Shekarpour

Vidal

M. E.

Lehmann

Auer

(2019). Old is gold: Linguistic driven approach for entity and relation linking of short text. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 conference of the north american chapter of the Association for Computational Linguistics: Human language technologies (pp. 2336–2346). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1243

83.

Sawant

Chakrabarti

(2013). Learning joint query interpretation and response ranking. In Proceedings of the 22nd international conference on World Wide Web, WWW ’13 (pp. 1099–1110). Association for Computing Machinery. https://doi.org/10.1145/2488388.2488484

84.

Schabes

Joshi

A. K.

(1988). An Earley-type parsing algorithm for Tree Adjoining Grammars. In Proceedings of the 26th annual meeting of the Association for Computational Linguistics (pp. 258–269). Association for Computational Linguistics.

85.

Shekarpour

Endris

K. M.

Jaya Kumar

Lukovnikov

Singh

Thakkar

Lange

(2016). Question answering on linked data: Challenges and future directions. In Proceedings of the 25th international conference companion on world wide web (WWW) (pp. 693–698). https://doi.org/10.1145/2872518.2890571

86.

Simov

Kouylekov

Simov

(2002). Cascaded regular grammars over XML documents. In Proceedings of the 2nd Workshop on NLP and XML (NLPXML).

87.

Simov

Osenova

(2004). A hybrid strategy for regular grammar parsing. In M. T. Lino, M. F. Xavier, F. Ferreira, R. Costa, & R. Silva (Eds.), Proceedings of the 4th international conference on language resources and evaluation (LREC). European Language Resources Association (ELRA).

88.

Soru

Marx

Moussallem

Publio

Valdestilhas

Esteves

Neto

C. B.

(2017). SPARQL as a foreign language. In Proceedings of the 13th international conference on semantic systems—SEMANTiCS2017 Posters and Demos.

89.

Soru

Marx

Valdestilhas

Esteves

Moussallem

Publio

(2018). Neural machine translation for query construction and composition. In ICML Workshop on neural abstract machines & program induction (NAMPI v2).

90.

Speer

Chin

Havasi

(2017). Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the 31th AAAI conference on Artificial Intelligence, AAAI’17 (pp. 4444–4451). AAAI Press.

91.

Steinmetz

Ann-Katrin

Sattler

K. U.

(2019). From natural language questions to SPARQL queries: A pattern-based approach. In Datenbanksysteme für business, technologie und web (BTW) (pp. 289–308). Gesellschaft für Informatik. https://doi.org/10.18420/btw2019-18

92.

Tan

Min

Chen

(2023). Can ChatGPT replace traditional kbqa models? an in-depth analysis of the question answering performance of the gpt llm family. In International semantic web conference (pp. 348–367). Springer-Verlag. https://doi.org/10.1007/978-3-031-47240-4_19

93.

Tanon

T. P.

de Assuncao

M. D.

Caron

Suchanek

F. M.

(2018). Demoing platypus—A multilingual question answering platform for Wikidata. In Proceedings of the 8th extended semantic web conference (ESWC) (pp. 111–116). Springer International Publishing. https://doi.org/10.1007/978-3-319-98192-5_21

94.

Unger

Freitas

Cimiano

(2014). An introduction to question answering over linked data. In Reasoning web (pp. 100–140). Springer.

95.

Usbeck

Gusmita

R. H.

Ngomo

A. N.

Saleem

(2018). 9th challenge on question answering over linked data (QALD-9). In Joint proceedings of the 4th workshop on semantic deep learning (SemDeep-4) and NLIWoD4: Natural Language Interfaces for the Web of Data (NLIWOD-4) and 9th Question Answering over Linked Data challenge (QALD-9) co-located with 17th International Semantic Web Conference (ISWC) (pp. 58–64).

96.

Usbeck

Ngomo

A. C. N.

Haarmann

Krithara

Röder

Napolitano

(2017). 7th open challenge on question answering over linked data (QALD-7). In Semantic web evaluation challenge (pp. 59–69). Springer International Publishing.

97.

Usbeck

Röder

Hoffmann

Conrads

Huthmann

Ngomo

A. N.

Demmler

Unger

(2019). Benchmarking question answering systems. Semantic Web, 10(2), 293–304. https://doi.org/10.3233/SW-180312

98.

Usbeck

Yan

Perevalov

Jiang

Schulz

Kraft

Möller

Huang

Reineke

Ngomo

Saleem

Both

(2023). QALD-10—The 10th challenge on question answering over linked data. Semantic Web. https://doi.org/10.3233/SW-233471

99.

Vaswani

Shazeer

Parmar

Uszkoreit

Jones

Gomez

A. N.

Kaiser

Polosukhin

(2017). Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Proceedings of the 31st international conference on neural information processing systems, NIPS’17 (Vol. 30, pp. 6000–6010). Curran Associates, Inc.

100.

Vijay-Shanker

(1992). Using descriptions of trees in a tree adjoining grammar. Computational Linguistics, 18(4), 481–518.

101.

Vollmers

Jalota

Moussallem

Topiwala

Ngomo

A. C. N.

Usbeck

(2021a). Knowledge graph question answering using graph-pattern isomorphism. In Proceedings of the 17th international conference on semantic systems (SEMANTiCS).

102.

Vollmers

Jalota

Moussallem

Topiwala

Ngomo

A. C. N.

Usbeck

(2021b). Knowledge graph question answering using graph-pattern isomorphism. In Proceedings of the 17th international conference on semantic systems (Vol. 53, p. 103).

103.

Walter

Unger

Cimiano

(2014). M-atoll: A framework for the lexicalization of ontologies in multiple languages. In P. Mika, T. Tudorache, A. Bernstein, C. Welty, C. Knoblock, D. Vrandečić, P. Groth, N. Noy, K. Janowicz, & C. Goble (Eds.), Proceedings of the 13th international semantic web conference (ISWC) (pp. 472–486). Springer International Publishing.

104.

Wang

Zhang

Yang

Song

Qin

(2022). GNN is a counter? Revisiting GNN for question answering. In International conference on learning representations (ICLR).

105.

Feng

Zhao

(2014). Xser@QALD-4: Answering natural language questions via phrasal semantic parsing. In Communications in computer and information science (Vol. 496).

106.

Yahya

Berberich

Elbassuoni

Ramanath

Tresp

Weikum

(2012). Natural language questions for the web of data. In J. Tsujii, J. Henderson, & M. Paşca (Eds.), Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL) (pp. 379–390). Association for Computational Linguistics.

107.

Yang

Cohen

W. W.

(2017). Differentiable learning of logical rules for knowledge base reasoning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 30). Curran Associates, Inc.

108.

Yasunaga

Ren

Bosselut

Liang

Leskovec

(2021). QA-GNN: Reasoning with language models and knowledge graphs for question answering. In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, & Y. Zhou (Eds.), Proceedings of the 2021 conference of the North American Chapter of the Association for Computational Linguistics: Human language technologies (pp. 535–546). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.naacl-main.45

109.

Yih

W. T.

Richardson

Meek

Chang

W. M.

Suh

(2016). The value of semantic parse labeling for knowledge base question answering. In K. Erk & N. A. Smith (Eds.), Proceedings of the 54th annual meeting of the association for computational linguistics (ACL) (Vol. 2, pp. 201–206). Association for Computational Linguistics.

110.

Yin

Gromann

Rudolph

(2021). Neural machine translating from natural language to SPARQL. Future Generation Computer Systems, 117, 510–519. https://doi.org/10.1016/j.future.2020.12.013

111.

Zhang

Acosta

Färber

Thoma

Rettinger

(2017). Brexearch: Exploring brexit data using cross-lingual and cross-media semantic search. In N. Nikitina, D. Song, A. Fokoue, & P. Haase (Eds.), Proceedings of the 16th international semantic web conference (ISWC), CEUR Workshop Proceedings (Vol. 1963). CEUR-WS.org.

112.

Zhang

Färber

Rettinger

(2016a). Xknowsearch! exploiting knowledge bases for entity-based cross-lingual information retrieval. In Proceedings of the 25th ACM international conference on information and knowledge management, CIKM ’16 (pp. 2425–2428). Association for Computing Machinery. https://doi.org/10.1145/2983323.2983324

113.

Zhang

Liu

Zhao

(2016b). Question answering over knowledge base with neural attention combining global knowledge information. In https://arxiv.org/abs/1606.00979. https://doi.org/10.48550/arXiv.1606.00979

114.

Zheng

Zhang

(2019). Question answering over knowledge graphs via structural query patterns. ArXiv abs/1910.09760. https://doi.org/10.48550/arXiv.1910.09760

115.

Zhou

Geng

Shen

Zhang

Jiang

(2021). Improving zero-shot cross-lingual transfer for multilingual question answering over knowledge graph. In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, & Y. Zhou (Eds.), Proceedings of the 2021 conference of the North American Chapter of the Association for Computational Linguistics: Human language technologies (pp. 5822–5834). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.naacl-main.465

116.

Zou

Huang

Wang

J. X.

Zhao

(2014). Natural language question answering over RDF: A graph data driven approach. In Proceedings of the 2014 ACM SIGMOD international conference on management of data, SIGMOD’14 (pp. 313–324). Association for Computing Machinery. https://doi.org/10.1145/2588555.2610525

	# Lexical	# Grammar Rule	# Grammar
Frame	Entry	Templates (GRT)	Rule (GR)
NounPPFrame	$101$	$45$	$5, 102$
TransitiveFrame	$63$	$113$	$5, 345$
InTransitivePPFrame	$74$	$121$	$4, 972$
AdjectivePredicateFrame	$15$	$15$	$501$
AdjectiveSuperlativeFrame	$10$	$7$	$70$
Total	$263$	$301$	$15, 990$

	Evaluation	# Lexical	Micro	Micro	Micro	Macro	Macro	Macro
Language	Mode	Entries	P	R	F $_{1}$	P	R	F $_{1}$
English	Inductive	$61$	$0.72$	$0.18$	$0.29$	$0.22$	$0.21$	$0.21$
	Incremental	$+ 14$	$0.91$	$0.79$	$0.85$	$0.31$	$0.30$	$0.30$
German	Inductive	$54$	$0.99$	$0.13$	$0.23$	$0.19$	$0.18$	$0.19$
	Incremental	$+ 11$	$0.99$	$0.70$	$0.82$	$0.28$	$0.28$	$0.27$
Italian	Inductive	$51$	$0.98$	$0.14$	$0.24$	$0.21$	$0.20$	$0.21$
	Incremental	$+ 13$	$0.99$	$0.49$	$0.65$	$0.27$	$0.26$	$0.27$
Spanish	Inductive	$53$	$0.98$	$0.13$	$0.24$	$0.2$	$0.19$	$0.19$
	Incremental	+ $21$	$0.99$	$0.71$	$0.83$	$0.28$	$0.27$	$0.27$

Multilingual Question Answering Over Linked Data Building on a Model of the Lexicon-Ontology Interface

Abstract

Keywords

1. Introduction

2.1. OntoLex-Lemon

2.3.1. Nouns

2.5. Generating Grammars

2.5.1. NounPPFrame

2.5.2. TransitiveFrame

2.5.3. IntransitivePPFrame

2.5.4. AdjectivePredicateFrame

2.5.5. AdjectiveSuperlativeFrame

2.6. Parsing

2.7. Simple Question

2.8. Complex Questions

3. Multilingual Grammar Generation

3.1. English Grammar Generation

3.1.1. Creating Lexical Entries

3.1.2. Creating Sentence Templates

3.2.1. German Grammar Generation

5. Evaluation

5.1. Benchmark

Table 5. Results for QALD-9 Training Dataset. Language Micro P Micro R Micro F 1 Macro P Macro R Macro F 1 English 0.88 0.74 0.80 0.55 0.56 0.55 German 0.95 0.71 0.81 0.51 0.51 0.51 Italian 0.89 0.53 0.66 0.45 0.46 0.45 Spanish 0.90 0.52 0.66 0.43 0.43 0.43

5.6.1. Comparision With ML-Based Approach

5.6.2. Systems Evaluated on QALD-9

5.7. Performance on Other Languages

5.8.1. Prompt

5.9. Portability to New Datasets and Languages

5.10.2. Impact of Grammar Rule (GR) Removal

5.10.3. Impact of Grammar Rule Template Removal

5.10.4. Impact of Syntactic Frame Removal

5.11. Discussion

6. Conclusion

Footnotes

Acknowledgments

Funding

Declaration of Conflicting Interests

ORCID iDs

Notes

Appendix A. SPARQL Queries for Grammar Rules of Lexicon Entries of LexInfo Frames

References