Evaluating Large Language Models for RDF Knowledge Graph-Related Tasks

Abstract

Current large language models (LLMs) can work with structured information and even assist developing program code, but can they support working with knowledge graphs (KGs) as well? Which LLM is offering the best capabilities in the field of semantic web and knowledge graph engineering (KGE)? Is it possible to determine this without manually checking many answers? The LLM-KG-Bench framework is designed to answer these questions. It consists of an extensible set of tasks for which the LLM answers are automatically evaluated, and covers different aspects of working with semantic technologies. This article gives a description of the LLM-KG-Bench framework, its main concepts, and the tasks implemented. In a benchmark run, a comprehensive dataset has been generated with it, evaluating more than 40 contemporary open and proprietary LLMs with 26 benchmark tasks, resulting in interaction logs and evaluations of roughly 45 000 LLM task dialogues. Finally, this dataset is used for an analysis of the SPARQL-related capabilities of the LLMs tested.

Keywords

knowledge graph large language model RDF SPARQL LLM-Benchmark LLM-capabilities LLM-RDF-Benchmark

1. Introduction

The combination of large language models (LLMs) and knowledge graphs (KGs) offers great synergetic potential for various knowledge-driven applications and is still gaining more traction within the research community. However, the ongoing rapid development in the field of LLMs makes it difficult to keep up with the latest developments and put them into context of prior work. Several initiatives for automated benchmarking, like BIG-bench (Srivastava et al., 2023), HELM (Liang et al., 2023), or Chatbot-Arena (Chiang et al., 2024), address the need for systematic evaluation of LLM performance. However, the tasks in scope of these evaluations are very general and do not address topics specifically related to KGs.

With the benchmarking framework LLM-KG-Bench, we are particularly focusing on automatically assessing and comparing LLMs regarding their capabilities to cope with Semantic Web technologies. This includes basic capability checks to establish a scientific foundation, as well as more practical tasks. The existing tasks cover already the Knowledge Graph Engineering (KGE) areas, RDF KG serialization, KG construction, SPARQL and KG schema, and the framework is open to support more tasks. Connectors for many open weight and top-ranking commercial LLMs are included.

Our investigation has been guided by the following overarching research questions:

RQ 1: Can KGE-related capabilities of LLMs be evaluated in an automated way?

RQ 2: Should specific LLMs be favored for specific tasks or KG data formats? And if so, to which extent do their capabilities differ and how can it be measured which LLM fits best for which task?

And specific to SPARQL SELECT queries:

RQ 3: Can LLMs follow the syntactic rules of SPARQL SELECT queries?

RQ 4: Can LLMs parse the semantics of SPARQL SELECT queries and act accordingly?

RQ 5: Can LLMs write semantically correct SPARQL SELECT queries for a given question and KG?

We present here an overview on the current state of the LLM-KG-Bench framework together with an evaluation dataset on more than 40 current LLMs and an evaluation of their SPARQL-SELECT-query-related capabilities. The LLM-KG-Bench framework is developed for automated benchmarking of Knowledge-Graph-related capabilities of LLMs.

The main contribution of this article is threefold:

We provide a description of the current state of the framework, including main concepts, its API, supported model connectors and tasks.

We present a dataset on the evaluation of more than 40 LLMs. The dataset includes detailed logs, interaction data and the evaluation result. This enables other researchers to extend this dataset, reproduce our analysis or run additional analyses, for example, by updating the framework code and using the framework’s reevaluation feature.

As a demonstration of the frameworks’ analytical capabilities and the generated dataset, we evaluate the SPARQL-SELECT-query-related capabilities of the selected LLMs.

This is an extended version of previous work by Meyer et al. (2025) with a similar use case as the one presented in Meyer et al. (2024). The added contribution of this paper compared to our previous work consists of:

a unified source of truth for the current state of the LLM-KG-Bench framework, especially based on current conference and workshop articles (Heim et al., 2025; Meyer et al., 2024, 2025)

a refined description of capabilities that an LLM needs to have in order to solve the benchmark tasks

the enrichment of the previous result dataset by executing the benchmark framework against several top-ranked LLMs

grouping and refining of the capability plots together with the computation and analysis of an aggregated score for the SPARQL-related capabilities of the LLMs tested

an improvement of the analysis of using Turtle (TTL) versus JSON-LD RDF serialization format in tasks by including a per-model as well as a per-task summary.

2. Related Work

2.1. LLM Evaluation

Attempts on exploring and navigating the vast amount of LLMs are made in form of several LLM leaderboards, which provide evaluation services for LLMs in form of selections of benchmarks or workloads. We discuss here the most relevant ones that we discovered during our research.

Focusing mainly on commercial models, the Chatbot Arena¹ (Chiang et al., 2024) lists scores for MMLU² and MT-bench (Bai et al., 2024), and also calculates its own arena-score (based on prompts given by users, processed by two models side-by-side and evaluated by the same user voting for his preferred answer). For open models, the Open-LLM-Leaderboard³ provides a list of benchmark results with over 2,000 tested models and several scores plus a carbon dioxide emission estimate. The most exhaustive list is provided by HELM⁴ (Liang et al., 2023) which includes also domain-specific tests like LegalBench and MedQA. While a comparison of the individual benchmark suites would go beyond the scope of this paper, to the best of our knowledge, none of these general leaderboards addresses knowledge graph engineering (KGE) tasks.

Moreover, what is generally amiss is a benchmark execution framework that helps to deal with the particularities of RDF- and KG-related workloads (format parsing, syntax check feedback loops, execution and evaluation of queries toward KGs, etc.). Although early attempts have been made by BIG-bench (Srivastava et al., 2023), it is falling short of building an efficient base for evaluating LLM capabilities in the context of KGE tasks, due to insufficiencies of the provided Task API, along with the focus on multiple choice tasks and scores based on string or document similarities.

In the face of these issues, our LLM-KG-Bench framework aims to reduce complexity and facilitate creating, executing, evaluating, and analyzing KG-related tasks. Its procedures focus on the syntactically and semantically correct generation of RDF (i.e., in Turtle) and SPARQL.

Current efforts in benchmarking coding capabilities seem more closely related to characteristics of KGE capabilities benchmarking, at a conceptual level (e.g., with respect to output format requirements, instruction complexity, and response evaluation strategies). Several leaderboards, like the Big Code Models Leaderboard⁵ for Java, Javascript, and CPP, or the EvalPlus Leaderboard⁶ for Python assess the coding proficiency of LLMs. We included some of the models ranked there into our test set.

2.2. RDF-Related Evaluation

In the current literature⁷ that explores the combination of LLMs and KGs (Meyer et al., 2023b; Pan et al., 2023, 2024), several attempts on evaluating the application of LLMs for KG-related tasks are made. However, these LLM evaluations often focus on details for a very selective problem in a specific task area, like Text to RDF (e.g., Mihindukulasooriya et al., 2023; Zhu et al., 2024), Knowledge Base Extraction from LLMs (e.g., Kalo et al., 2025; Petroni et al., 2019; Singhania et al., 2022), Knowledge Graph Question Answering (KGQA, e.g., Usbeck et al., 2019), Text2SPARQL (e.g., Kovriguina et al., 2023; Marx et al., 2025; Zahera et al., 2024) or generation of RDF mapping language (RML) mappings (e.g., Hofer et al., 2024). Unfortunately, many of these evaluations have been conducted manually, which causes scalability issues with respect to more repetitions and including more or newer models. In case an automated evaluation has been performed, the underlying code usually lacks adaptability to encompass new models or task variations. A benchmarking effort that is related to our interest in studying the JSON-LD capabilities, is StructuredRAG (Shorten et al., 2024). It consists of six tasks designed to assess LLM capabilities in following response format instructions according to JSON templates. Table 1 compares several of the mentioned LLM evaluation approaches with respect to the covered LLMs, addressed KGE tasks and their evaluation mode. Only the LLM-KG-Bench framework combines automatic evaluation for many covered LLMs and several KGE topics.

Table 1.
Comparison of Some of the LLM Evaluation Approaches Mentioned Here.

LLMs Covered KGE Topics Eval. Type

BIG-bench (Srivastava et al., 2023) many (about 15) no automatic

HELM (Liang et al., 2023) many (about 140) no automatic

Chatbot Arena (Chiang et al., 2024) many (about 280) no crowd

ChatGPT KG Experiments (Meyer et al., 2023b) some (2) several (Text2Sparql, KG constr., KG exploration) manual

Text2KgBench (Mihindukulasooriya et al., 2023) some (2) Text2KG automatic

AutoKG (Zhu et al., 2024) some (4) Text2KG, Reasoning manual

LLM-KG-Bench 3 many (about 40) several (RDF, KG constr., SPARQL, KG schema) automatic

	LLMs Covered	KGE Topics	Eval. Type
BIG-bench (Srivastava et al., 2023)	many (about 15)	no	automatic
HELM (Liang et al., 2023)	many (about 140)	no	automatic
Chatbot Arena (Chiang et al., 2024)	many (about 280)	no	crowd
ChatGPT KG Experiments (Meyer et al., 2023b)	some (2)	several (Text2Sparql, KG constr., KG exploration)	manual
Text2KgBench (Mihindukulasooriya et al., 2023)	some (2)	Text2KG	automatic
AutoKG (Zhu et al., 2024)	some (4)	Text2KG, Reasoning	manual
LLM-KG-Bench 3	many (about 40)	several (RDF, KG constr., SPARQL, KG schema)	automatic

Best values are marked with bold font. Only the LLM-KG-Bench framework automatically evaluates several knowledge graph engineering (KGE) tasks and covers many LLMs in a benchmark run.

2.3. Text2Sparql and KBQA

The interpretation and generation of the query language SPARQL, being a core technology for accessing KGs, is one especially relevant example of integrating Semantic Technology with LLMs.

Rangel et al. (2024) introduce a method for fine-tuning OpenLLaMA to generate SPARQL queries for question answering on life science knowledge graphs. Their approach leverages data augmentation techniques, such as the use of meaningful variable names and inline comments, which lead to improved accuracy in SPARQL query generation.

Bustamante and Takeda (2024) focus on enhancing the generation of SPARQL queries from natural language questions. They employ a GPT model to identify the most challenging aspects of the Text2SPARQL task, aiming to apply targeted solutions to these difficulties.

Avila et al. (2024) evaluate ChatGPT’s performance in answering natural language questions on knowledge graphs. Their method, Auto-KGQAGPT, explores translating such questions into SPARQL queries using prompts that include relevant knowledge graph fragments.

Li et al. (2024) address the performance drop seen in real-world conditions where high-quality annotated data are lacking. They propose the FlexKBQA framework, which uses template SPARQL queries that are converted into natural language questions via LLMs to produce synthetic training data. This data is used to fine-tune a lightweight SPARQL generator, further refined with real queries to bridge the gap between synthetic and real-world inputs.

Diallo et al. (2024) provide a thorough evaluation comparing pre-trained language models, non-pre-trained language models, and LLMs, along with various fine-tuning strategies. Their error analysis reveals that incorrect URIs, often caused by hallucinations, are a major source of failure in generated SPARQL queries. Hirigoyen et al. (2022) demonstrate that incorporating a copy mechanism can help mitigate such hallucinations.

Kovriguina et al. (2023) present SPARQLGEN, a one-shot generation technique that uses LLMs to produce SPARQL queries. Their method includes the full context comprising the question, a relevant RDF subgraph, and an example query within a single prompt.

Pliukhin et al. (2023) propose a similar approach to SPARQLGEN for query generation over the ORKG scholarly knowledge graph, enhanced with a more advanced subgraph extraction method. Zahera et al. (2024) extend this by incorporating chain-of-thought prompting and using the GERBIL QA system (Usbeck et al., 2019).

Lehmann et al. (2023) advocate the use of Controlled Natural Language (CNL) as a user interface due to its proximity to natural language. They show that CNL can be reliably and unambiguously translated into formal query languages like SPARQL, significantly reducing the need for large training datasets.

Diefenbach et al. (2017) introduce two datasets for training and benchmarking QA systems on Wikidata: One derived from the SimpleQuestions dataset (Bordes et al., 2015), and another based on user logs and feedback.

Dubey et al. (2019) extend the LC-QuAD dataset and introduce LC-QuAD 2.0 (Trivedi et al., 2017), which is compatible with both Wikidata and DBpedia. The dataset comprises 30,000 questions, including paraphrases and corresponding SPARQL queries.

Banerjee et al. (2022) explore various ways to integrate LLMs into standard knowledge graph workflows. Their focus is on models that can be fine-tuned with consumer-grade hardware, in contrast to approaches focusing on large-scale models used without additional training.

Frey et al. (2025) demonstrate the long-term value of automated benchmarking through the LLM-KG-Bench platform. They compare multiple LLM iterations with a focus on their ability to handle RDF Turtle syntax, preserving responses for future evaluation. Heim et al. (2025) is an example of such a separate evaluation.

Hofer et al. (2024), consistent with the findings by Frey et al. (2023; 2025), investigate LLM-driven RML mapping generation in Turtle. They find that while syntactic errors occur, most LLMs can correct these mistakes. Given that SPARQL is built on Turtle, the authors conduct their experiments as multi-turn dialogues with feedback loops for error correction.

The recently organized ”First International Text2Sparql Challenge” (Marx et al., 2025) shows the potential of best LLM agent-based approaches (Brei et al., 2025; Liu et al., 2024b; Perevalov & Both, 2025) for the Text2Sparql problem and a specific automatic evaluation approach roughly similar to the Text2Sparql evaluation used in LLM-KG-Bench. But it shows as well the open gap, as the best participants reached only scores of about $0.5$ .

Nevertheless, none of these existing works reach the goal of automatic evaluation on many KGE-related tasks as presented here with the LLM-KG-Bench framework in version 3.

3. The LLM-KG-Bench Framework

Benchmarking LLMs involves significant time, financial costs, organizational effort, and the evaluation process can often be imprecise. LLM-KG-Bench is designed to simplify the creation of KG-related assessments while providing a foundational infrastructure for further development. Its main features are:

Modular and Extensible Framework: Supports automated evaluation tasks using a comprehensive set of KG-extraction and evaluation-related helper methods.

Built-in Correction Cycles: Implements dialogue-based correction cycles, enabling LLMs to revise previous mistakes.

Data Security: Supports encryption of task data to prevent test data leakage into LLM training datasets.

Task Management: Manages task configurations, evaluation orchestration, logging, and result persistence.

Result Analysis and Visualization: Provides built-in tools for analyzing and visualizing evaluation results.

Broad Model Support: Includes connectors for many contemporary LLMs.

Open-Source Codebase: The framework is published as open source and welcomes extensions and community contributions.

In the following sections, we describe the basic concepts and infrastructure of the LLM-KG-Bench framework in greater detail.

3.1. Architecture and Main Concepts

Figure 1 gives an overview on the LLM-KG-Bench framework and its execution flow. It is build around some main concepts we want to describe here. We start with the general concept of the Prompt-Answer-Evaluate loop before going further to LLM connectors, tasks, execution scopes, configuration, and scores.

Figure 1.

Overview on the execution flow of the LLM-KG-Bench framework. A execution configuration defines some settings and selects LLM connectors as well as evaluation tasks from a benchmark configuration. In the benchmark execution all given combinations of evaluation tasks and LLM connectors are executed for the number of iterations each as task evaluation iteration. In the task evaluation iteration the task provides a prompt which is answered by the LLM via the LLM connector and the answer is evaluated by the task, maybe resulting in further Prompt-Answer-Evaluate rounds. All results and logs from the task executions is stored for later analysis, visualization or reevaluation. Source: LLM-KG-Bench documentation, updated version of Meyer et al. (2023a).

Prompt-Answer-Evaluate Loop. The evaluation of LLMs is based on dialogues, consisting of prompts and answers. This enables tasks to make use of the chat capability of modern LLMs and their bigger supported context size to get the answer closer to the correct one in an iterative way.

The Prompt-Answer-Evaluate loop starts with the generation of an initial prompt that is sent to an LLM. In the next step, the produced answer is evaluated. Based on the evaluation result, the framework can decide to start a new Prompt-Answer-Evaluate round or stop the dialogue. The process from initial prompt to the stop of the dialogue is named a task evaluation iteration.

The structure of these loops is shown in Figure 2(a) and an example dialogue is given in Figure 4. The prompt and evaluate parts are implemented in evaluation tasks and the answer is organized with the help of LLM connectors.

Figure 2.

Overview of the evaluation workflow and execution scopes. (a) The Prompt-Answer-Evaluate loop for the task – LLM interaction as organized by the framework. Prompting and evaluation is covered by the task, the answer is generated by the LLM, and (b) different execution scopes: task evaluation iterations (includes one to many cycles of the Prompt-Answer-Evaluate loop), task evaluation execution (includes all iterations), benchmark execution (includes all task executions for all combinations of tasks and LLM models defined for all iterations).

Figure 3.

UML class diagram of the Task API and its reference by some example evaluation tasks. All evaluation tasks implement the AbstractLlmKgBenchInterface via an inheritance connection. A task can be described with a TaskInfo object. A TaskExecutionInfo references this TaskInfo object for the documentation.

Figure 4.

Fictive example dialogue for the RdfSyntaxFixList task with a missing dot in Turtle syntax. Some text left out is marked with ”…”. The LLM’s first answer is missing the expected code block with the fixed Turtle which is corrected in the second answer.

LLM Connectors and LLM Connector Classes. The LLM connectors (or model connectors) offer a consistent abstraction layer to interact with a specific LLM (or model). In the benchmark configuration the LLM connectors are defined by parametrizing LLM connector classes for specific LLMs. Several LLM connector classes classes are implemented in the LLM-KG-Bench framework as described in Section 3.3, adding support for many different LLMs.

Evaluation Tasks, Task Classes, and Parametrized Tasks. The evaluation tasks (or benchmark tasks) are the main building block of a benchmark and automatically evaluate the LLM answers. For the Prompt-Answer-Evaluate loop the tasks provide the prompt and evaluation functionality.

Task Classes and Parametrized Tasks. The evaluation tasks can be defined either direct in task classes or as parametrized task classes derived in the benchmark configuration from a task class with a task parameters definition.

Task Parameters. Several task classes support as task parameter a targeted task size or a serialization format parameter. The targeted task size task parameter defines the count of characters expected to be exchanged with the LLM connector in one task iteration.

List Tasks and Task Case Entries. List tasks have a list of task case entries, where each entry defines a distinct exercise resulting in a specific prompt and expected answer. For each task iteration one task case entry is selected from this list. All task case entries are evaluated by the same list task.

Task Evaluation Iterations. We name one task evaluation loop consisting of one or more Prompt-Answer-Evaluate rounds a task evaluation iteration (or a dialog), see also Figure 2(b).

Task Evaluation Executions. Since LLM answers are generated probabilistically, a configurable number of task evaluation iterations is executed, collectively forming a task evaluation execution for a specific task and a particular LLM, see also Figure 2(b).

Benchmark Executions. A benchmark execution consists of all task executions for all combinations of tasks and models defined in the execution configuration, see also Figure 2(b).

Result Reevaluation. LLM answers recorded in a benchmark execution can get reevaluated by repeating the task evaluation code with the LLM connector interactions substituted with the stored interactions. When looking at the Prompt-Answer-Evaluate loop given in Figure 2(a), the evaluate code is reexecuted but prompt and answer is taken from the recorded interaction. This makes it possible to reproduce evaluation results and even offers the chance to run updated evaluation code on given answers without the need of new maybe costly or differing LLM interactions.

Benchmark Configuration and Execution Configuration. A benchmark configuration specifies the tasks and models to be included in a benchmark run together with the number of iterations per task execution. A benchmark configuration can be executed as a whole or with a selection of tasks and models defined by command line parameters.

Evaluation Scores and Infos. An evaluation task should compute for each task evaluation iteration a key value dictionary containing one or more evaluation scores with values in the range of $[0..1]$ . Here $1$ should be the score for an optimal answer. In addition to evaluation scores an evaluation task can return evaluation infos containing more information in the form of a key value dictionary, where the values could be anything useful about the evaluation of the answers.

3.2. Tasks API

Evaluation tasks are implemented following the Task API as a common interface between evaluation tasks and the framework. In LLM-KG-Bench framework Version 3, a major update of the Task API was introduced together with new helper classes. Figure 3 shows a UML class diagram of the new Task API.

All evaluation tasks in LLM-KG-Bench implement the interface AbstractLlmKgBenchTaskInterface. It defines the interface between evaluation tasks and the framework, including the method signatures for Evaluate and Prompt steps as well as task serialization and deserialization.

The following methods defined in the AbstractLlmKgBenchTaskInterface are especially important:

getNextPrompt: combines an evaluation and prompting step. If no new prompt is generated the Prompt-Answer-Evaluate loop ends.

finalizeEvaluation: is called at the end of the Prompt-Answer-Evaluate loop and creates a final evaluation result for this task evaluation iteration.

condenseTaskData: creates a serializable representation of this concrete task case entry. This offers the possibility for later continuation or reevaluation.

createTaskFromCondensedData: initializes a task from the representation given by condenseTaskData

The abstract implementation AbstractLlmKgBenchTaskClass helps to reduce redundant code and eases the concrete task implementation. For the two main variations of tasks, single-prompt tasks and dialogue-tasks, specialized abstract classes are provided. The AbstractDialogTask class helps with handling the score and info values produced by each Prompt-Answer-Evaluate round and computes mean and max aggregate values for scores at the end of a task evaluation iteration. Tasks that store their task data in an encrypted file can benefit from the abstract class AbstractFileListTaskImplementation.

This task API is inspired by the BIG-bench (Srivastava et al., 2023) task classes but has a more detailed abstraction to let the framework orchestrate the tasks Prompt-Answer-Evaluate loop shown in Figure 2(a). Nonetheless the task API should be similar enough with the common generic benchmarking frameworks to keep the path of combining them open for the future. The new task API gives the central framework logic more flexibility in the orchestration, reduces error possibilities and reduces repeated code in tasks.

3.3. LLM Connectors and Supported Models

LLM connectors are responsible for offering standardized APIs to LLMs. They are defined similar to the BIG bench model class. The main method offered is generate_text(inputs, ...)->str, taking a single prompt or dialogue and returning the LLM’s answer.

The LLM-KG-Bench framework offers several model connector classes:

OpenAI / ChatGPT: Connector class for OpenAI-compatible LLMs like GPT-3.5, GPT-4, GPT-4t, GPT-4o and GPT-o1 via the OpenAI python library⁸ and REST API⁹. Many other LLM providers offer a compatible REST endpoint. They can be integrated with this connector as well.

Google / Gemini: Connector class for LLMs from Google like Gemini 1.5 or Gemini 2.0 via the Google python library¹⁰ and REST API¹¹.

Anthropic / Claude: Connector class for LLMs from Anthropic from Claude 1.0 to Claude 3.5. The connector uses the Anthropic REST API¹² using the offered Python library¹³.

vLLM : Runtime for self-hosted LLMs¹⁴ (Kwon et al., 2023). This library is compatible to many open LLMs and enables serving and inferencing with them.

3.4. Evaluation Tasks and Scores

Several tasks are implemented in the LLM-KG-Bench framework as described in several articles (Frey et al., 2023, 2025; Meyer et al., 2024, 2025, 2023a). The tasks range from basic RDF capability checks to establish a clean scientific foundation for further experiments, but include more practical tasks as well.

Here we list all task classes implemented at the moment together with the most important scores. More details are given in the referenced papers and in the tasks folder of the code repository¹⁵. Table 2 shows the relationship between the SPARQL-SELECT-query-related tasks and the research questions 3 to 5.

RDF-related Tasks:

RdfConnectionExplainStatic: This task asks the LLM to find the shortest connection between two nodes in a small RDF graph. The shortest path shall be output as a list of IRIs. Four variations present the graph in different serialization formats: JSON-LD, N-Triples, Turtle, and RDF/XML. See also Frey et al. (2023) and Meyer et al. (2025).

The most important score is listTrimF1: this is an f1 measure calculated by comparison of trimmed given and expected lines.

RdfFriendCount: For a simple RDF graph with, respectively, only one node and one edge type, the LLM is asked to identify the node with the most incoming edges. There are several variations covering on the one hand the different serialization formats JSON-LD, N-Triples, Turtle, and RDF/XML and on the other hand using different values for the other parameters size (targeted task size), knowsCount (number of incoming edges for normal nodes), and specialAddKnowsCount (number of additional incoming edges for the one special node). See also Frey et al. (2023) and Meyer et al. (2025).

The most important score is f1: this is an f1 measure when comparing the given persons with the correct answer.

RdfSyntaxFixList: The LLM is instructed to correct a syntactically invalid RDF graph. The variations cover the serialization formats JSON-LD, N-Triples, and Turtle. For more details, see Meyer et al. (2025).

A former version is the TurtleErrorsStatic task (Frey et al., 2023; Meyer et al., 2023a), which is limited to a single turtle test case but implements an evaluation method optimized for Turtle serialization format.

The most important scores are 0_combined and max_combined: these scores combine for the first (0) or best (max) answer in the dialog a string similarity measure score, a parsable syntax score, and an f1 measure on the KG content. The combined scores are calculated as $0.1 \cdot s t r S i m i l a r i t y + 0.2 \cdot p a r s a b l e S y n t a x + 0.7 \cdot c o n t e n t F 1$ .

RDF-related Tasks on KG-Construction:

FactExtractStatic: The LLM is instructed to extract facts from a given textual fact sheet and create a corresponding Turtle KG. See also Meyer et al. (2023a) and Frey et al. (2023).

The most important score is norm_f-measure: this is an f1 measure comparing triples of the normalized graphs given and expected.

TurtleSampleGeneration: In this task, the LLM shall generate small Turtle knowledge graphs satisfying given requirements. See also Meyer et al. (2023a) and Frey et al. (2023).

The most important score is persons_relative_error: This is a comparison of the number of persons in the generated graph with the expected number.

SPARQL-related Task Classes (for specific tasks see Table 3):

Sparql2AnswerList (S2A): Given a small KG and a SPARQL SELECT query, the LLM shall return the respective result set for the query. Here, the KG is given in the JSON-LD or Turtle format. See also Meyer et al. (2024).

The most important score is combinedF1Score: this is a combination of several F1 measures comparing the expected and given answer.

Text2AnswerList (T2A): In this task, the LLM is instructed to return the result set answering a given textual question on a given KG. While this task is similar to KBQA, we are here mainly interested scientifically in the variation of the Sparql2Answer task. We use the same evaluation logic, similar prompts, and similar task case entries, but replace the SPARQL SELECT query by the corresponding textual question. Again, the variations introduce the KG in either the JSON-LD or Turtle format. For more details, see Meyer et al. (2024).

The most important score is the previously explained combinedF1Score.

Text2SparqlList (T2S): Given a KG and its description, the LLM shall construct a SPARQL SELECT query corresponding to a given natural language query. The KG is either given completely or partly as a subgraph. Alternatively, some variations provide the KG schema instead of the graph itself, thus testing basic schema read capabilities. The variations are described in Table 3 and cover seven different KG datasets. See also Table 3 and Meyer et al. (2024).

The most important scores are 0_combined, max_combined, 0_answerParse and max_answerParse: The answer parse scores evaluate if the first (0) or best (max) given SPARQL SELECT query parses syntactically correct. The combined scores for the first (0) or best (max) given SPARQL SELECT query are based on the answer parse score and an F1 score comparing the result sets of given and optimal query. The value is computed as $0.2 \cdot a n s w e r P a r s e + 0.8 \cdot f 1 m e a s u r e$ .

SparqlSyntaxFixingList (SSF): Similar to the RdfSyntaxFixList task, the LLM gets a SPARQL SELECT query with syntax errors and shall return a corrected query. See also Meyer et al. (2024).

The most important scores are the previously explained 0_combined, max_combined, 0_answerParse and max_answerParse.

The prompts used by the tasks are designed in a way that keeps ambiguity as low as possible. All requirements that we expect the LLM to respect are stated explicitly, for example, “stick with the original formatting”, or “answer with just one markdown fenced code block … no other text”. At the same time, we avoid LLM-specific prompt optimization for a fair comparison across different models. An example dialog is shown in Figure 4.

Table 2.
Mapping of the Task Types to the Different Task Aspects and Research Questions Covered. Adopted from Meyer et al. (2024).

Research Question:	RQ 3	RQ 3	RQ 4	RQ 5
Task Aspect:	Syntax Read	Syntax Create	Semantic Read	Semantic Create
SPARQL Syntax Fixing (SSF)	x	x	–	–
SPARQL to Answer (S2A)	x	–	x	–
Text to SPARQL (T2S)	–	x	–	x
Text to Answer (T2A)	–	–	–	–

Table 3.

Overview on the Different SPARQL-Related Tasks Implemented within LLM-KG-Bench Framework.

Task Type	Dataset Subset	Task	KG Info Type	KG Info Format
SSF	Syntax-Errors	SSF-LC-QuAD	not needed here	not needed here
S2A	Organizational	S2A-Orga	full KG	Turtle or JSON-LD
T2S	Organizational	T2S-Orga	full KG	Turtle
	Organizational-Numeric	T2S-OrgaNum	full KG + IRIs + Labels	Turtle + table
	Beastiary	T2S-Beast-Graph	KG subset	Turtle
		T2S-Beast-Schema	schema	Turtle
		T2S-Beast-Subschema	schema subset	Turtle
		T2S-Beast-Iris	IRIs	list
	CoyPu-Mini	T2S-CoyPu-Graph	full KG	Turtle or JSON-LD
		T2S-Coypu-Schema	schema	Turtle or JSON-LD
		T2S-Coypu-Iris	IRIs	list
T2A	Organizational	T2A-Orga	full KG	Turtle or JSON-LD

3.5. Benchmark Datasets and KGs Used for SPARQL Task Implementations

There are several benchmark datasets available which contain pairs of SPARQL SELECT queries and textual natural language questions for specific knowledge graphs. We selected and implemented benchmark tasks for a couple of current datasets for smaller and bigger knowledge graphs. Only English textual questions were used as we focus on SPARQL here and not language capabilities. A total of five tuples, each consisting of a question and corresponding SPARQL SELECT query, was manually selected from each dataset. This allows to rerun the tasks more often to reduce the random noise in the results.

Organizational dataset and KG: The smallest KG used is an organizational KG (Meyer et al., 2023b). We use it here together with a corresponding dataset¹⁶ of question and SPARQL pairs created by Brei et al. (2024).

Triples: $29$ . Identifiers: human readable.

Organizational Numeric: We created an additional variant of the organizational dataset and KG with numerical IRIs (first 3 digits of hash) and same questions.

Triples: $29$ . Identifiers: numeric.

CoyPu-Mini dataset and KG: Brei et al. (2024) published another dataset based on a small subset of the CoyPu KG¹⁷. This sub graph is small enough to fit into the context size of LLMs evaluated here. We added lists of relevant IRIs and schema information.

Triples: $283$ . Identifiers: human readable.

Beastiary dataset and KG: The Beastiary dataset and KG¹⁸ was presented by Kovriguina et al. (2023). It offers for each question a relevant sub graph and a list of IRIs. We derived relevant schema information from these IRI lists.

Triples: $98070$ . Identifiers: human readable.

SPARQL SELECT Query Syntax Errors: We took one question and SPARQL pair from LC-QuAD and derived 5 tests from it by inserting different kinds of syntax errors, one error per test.

Triples: $\sim 1 billion$ . Identifiers: numeric.

The list of SPARQL-related benchmark tasks created based on these resources in the LLM-KG-Bench framework is given in Table 3.

3.6. General Task Characterization

For a structured task characterization, we draw from frameworks from cognitive psychology and educational research. Bloom’s Taxonomy (Bloom, 1956) was originally developed to classify educational objectives based on the level of cognitive complexity required. It is based on behavioral observations of learning processes and was later revised to better fit modern views of cognitive psychology (Anderson & Krathwohl, 2001). Part of that revision includes the separation between what is known from what is done with it, leading to a Knowledge Dimensions framework (Anderson & Krathwohl, 2001). It distinguishes between types of knowledge required for different tasks. Relational Complexity Theory (Halford et al., 1998) was originally developed in developmental and comparative psychology and draws the notion of relational arity from formal systems in logic and computer science, including relational database theory. It models task difficulty as a function of the number of simultaneous relations that must be processed. Together, these models provide a basis for analyzing the cognitive-inspired and structural demands that the benchmark tasks impose.

The values in the characterization are assigned based on the minimal cognitive and structural requirements expected for successful task completion. Cognitive processes such as Understand are assigned when tasks require interpreting given information, while Apply corresponds to the execution of known procedures. Create is used when tasks involve generating new content and Analyze when tasks require decomposing and interpreting relations between elements. For knowledge dimensions, Factual knowledge is assigned when specific syntactic or terminological information is needed, Conceptual when structural or relational understanding is necessary and Procedural when correct application of methods is required. Relational complexity is rated as Low when tasks involve isolated or simple binary relations, and as Medium when multiple entities and relations must be coordinated simultaneously. Note that not all categories across the frameworks are represented, as the current set of tasks does not span the full theoretical space. The characterization is shown in Table 4.

Table 4.
Task Characterization According to Bloom’s Taxonomy (Bloom, 1956), the Knowledge Dimensions Framework (Anderson & Krathwohl, 2001) and Relational Complexity Theory (Halford et al., 1998).

Task Cognitive Process Knowledge Dimension Relational Complexity

RDF-related:

FactExtractStatic Understand, Create Conceptual, Procedural Medium

RdfConnectionExplainState Understand, Analyze Conceptual Medium

RdfFriendCount Apply Procedural Low

RdfSyntaxFixList Understand, Apply Factual, Procedural Low

TurtleSampleGeneration Understand, Create Conceptual, Procedural Medium

SPARQL-related:

Sparql2AnswerList Understand, Apply Conceptual, Procedural Low

Text2AnswerList Understand, Apply Conceptual, Procedural Low

Text2SparqlList Understand, Create Conceptual, Procedural Low

SparqlSyntaxFixingList Understand, Apply Factual, Procedural Low

Task	Cognitive Process	Knowledge Dimension	Relational Complexity
RDF-related:
FactExtractStatic	Understand, Create	Conceptual, Procedural	Medium
RdfConnectionExplainState	Understand, Analyze	Conceptual	Medium
RdfFriendCount	Apply	Procedural	Low
RdfSyntaxFixList	Understand, Apply	Factual, Procedural	Low
TurtleSampleGeneration	Understand, Create	Conceptual, Procedural	Medium
SPARQL-related:
Sparql2AnswerList	Understand, Apply	Conceptual, Procedural	Low
Text2AnswerList	Understand, Apply	Conceptual, Procedural	Low
Text2SparqlList	Understand, Create	Conceptual, Procedural	Low
SparqlSyntaxFixingList	Understand, Apply	Factual, Procedural	Low

4. Experiment

We utilized the LLM-KG-Bench framework to evaluate a list of 41 LLMs with a selection of 26 tasks. The results containing the full dialog and evaluation data for all this about 45 000 LLM task dialogues (task Evaluation Iterations) is documented and publicly available for further analysis including reevaluation. Whenever new LLMs are of interest, their data can get added to the initial dataset (Meyer et al., 2025). An example for a statistical analysis on existing data generated is presented by Heim et al. (2025).

In the following section, we outline the experiment setup and some analysis results from this evaluation.

4.1. Experimental Setup

We adopted the default configuration to define the tasks and models included in this evaluation. As a trade-off between resource usage and confidence, we have decided to conduct 20 iterations for the proprietary LLMs and 50 iterations for the open LLMs. For the 41 LLMs selected and 26 selected evaluation tasks this get multiplied to 45 500 task evaluation iterations.

4.1.1. Selected Tasks

The benchmark was executed on the following 26 tasks that are described in Section 3.4 and in the code repository:

RdfSyntaxFixList (3 variations): For Turtle, JSON-LD and N-Triples as graph format

RdfConnectionExplainStatic (4 variations): For Turtle, JSON-LD, RDF/XML and N-Triples as graph format

RdfFriendCount (8 variations): For Turtle, JSON-LD, RDF/XML and N-Triples as graph format; 1 and 2 as additional link count

SparqlSyntaxFixingList (1 variation): For LC-QuAD / Wikidata

Sparql2AnswerList (1 variation): For Organizational graph

Text2SparqlList (9 variations): For Organizational, Organizational-Numeric, Coypu-Mini and Beastiary

4.1.2. Selection of Proprietary LLMs

To get an overview on the current state-of-the-art proprietary models we selected three long-term high-ranked model families from the Chatbot Arena Leaderboard: OpenAI GPT, Google Gemini and Anthropic Claude. From these families, we selected the current models in various sizes and also included the latest GPT-3.5 for comparability with other results. The selected models together with their context size are shown in Table 5.

Table 5.
Details for the Models Selected for the Experiment Presented Here and the Iterations Evaluated Per Model and Task Combination.

Family Model Parameter Count Context Iterations

OpenAI-GPT ChatGPT 3.5 turbo ? 16k 20

ChatGPT 4o ? 128k 20

ChatGPT 4o-mini ? 128k 20

ChatGPT o1 ? 128k 20

ChatGPT o1-mini ? 128k 20

Google-Gemini Gemini 2.0 Flash ? 128k–1M 20

Gemini 1.5 Pro ? 128k–2M 20

Gemini 1.5 Flash ? 128k–1M 20

Anthropic-Claude Claude 3.5 Sonnet ? 200k 20

Claude 3.5 Haiku ? 200k 20

Qwen (Hui et al., 2024; Yang et al., 2024) Qwen2-0.5B-Instruct 0.5B 32k 50

Qwen2-1.5B-Instruct 1.5B 32k 50

Qwen2-7B-Instruct 7B 128k 50

Qwen2-57B-A14B-Instruct 57B (active: 14B) 64k 50

Qwen2-72B-Instruct 72B 128k 50

Qwen2.5-0.5B-Instruct 0.5B 32k 50

Qwen2.5-1.5B-Instruct 1.5B 32k 50

Qwen2.5-3B-Instruct 3B 32k 50

Qwen2.5-7B-Instruct 7B 128k 50

Qwen2.5-14B-Instruct 14B 128k 50

Qwen2.5-32B-Instruct 32B 128k 50

Qwen2.5-72B-Instruct 72B 128k 50

Qwen2.5-Coder-32B-Instruct 32B 128k 50

Qwen3.0-235b-a22b 235B (active 22B) 41k 50

Meta-Llama (Grattafiori et al., 2024) Meta-Llama-3-8B-Instruct 8B 8k 50

Meta-Llama-3-70B-Instruct 70B 8k 50

Llama-3.1-8B-Instruct 8B 128K 50

Llama-3.1-70B-Instruct 70B 128K 50

Llama-3.2-1B-Instruct 1B 128K 50

Llama-3.2-3B-Instruct 3B 128K 50

Llama-3.3-70B-Instruct 70B 128K 50

Llama-4.0-Maveric 400B (active: 17B) 1M 50

Microsoft-Phi (Abdin et al., 2024) Phi-3-mini-128k-instruct 3.8B 128k 50

Phi-3-small-128k-instruct 7B 128k 50

Phi-3-medium-128k-instruct 14B 128k 50

Phi-3.5-mini-instruct 3.8B 128k 50

Phi-3.5-MoE-instruct 42B (active: 6.6B) 128k 50

Infly-OpenCoder (Huang et al., 2024) OpenCoder-8B-Instruct 8B 8k 50

Deepseek-ai (Guo et al., 2024) deepseek-coder-33b-instruct 33B 16k 50

deepseek-chat-v3-0324 685B (active: 37B) 64K–164K 50

deepseek-r1 671B (active: 37B) 64K–164K 50

Family	Model	Parameter Count	Context	Iterations
OpenAI-GPT	ChatGPT 3.5 turbo	?	16k	20
	ChatGPT 4o	?	128k	20
	ChatGPT 4o-mini	?	128k	20
	ChatGPT o1	?	128k	20
	ChatGPT o1-mini	?	128k	20
Google-Gemini	Gemini 2.0 Flash	?	128k–1M	20
	Gemini 1.5 Pro	?	128k–2M	20
	Gemini 1.5 Flash	?	128k–1M	20
Anthropic-Claude	Claude 3.5 Sonnet	?	200k	20
	Claude 3.5 Haiku	?	200k	20
Qwen (Hui et al., 2024; Yang et al., 2024)	Qwen2-0.5B-Instruct	0.5B	32k	50
	Qwen2-1.5B-Instruct	1.5B	32k	50
	Qwen2-7B-Instruct	7B	128k	50
	Qwen2-57B-A14B-Instruct	57B (active: 14B)	64k	50
	Qwen2-72B-Instruct	72B	128k	50
	Qwen2.5-0.5B-Instruct	0.5B	32k	50
	Qwen2.5-1.5B-Instruct	1.5B	32k	50
	Qwen2.5-3B-Instruct	3B	32k	50
	Qwen2.5-7B-Instruct	7B	128k	50
	Qwen2.5-14B-Instruct	14B	128k	50
	Qwen2.5-32B-Instruct	32B	128k	50
	Qwen2.5-72B-Instruct	72B	128k	50
	Qwen2.5-Coder-32B-Instruct	32B	128k	50
	Qwen3.0-235b-a22b	235B (active 22B)	41k	50
Meta-Llama (Grattafiori et al., 2024)	Meta-Llama-3-8B-Instruct	8B	8k	50
	Meta-Llama-3-70B-Instruct	70B	8k	50
	Llama-3.1-8B-Instruct	8B	128K	50
	Llama-3.1-70B-Instruct	70B	128K	50
	Llama-3.2-1B-Instruct	1B	128K	50
	Llama-3.2-3B-Instruct	3B	128K	50
	Llama-3.3-70B-Instruct	70B	128K	50
	Llama-4.0-Maveric	400B (active: 17B)	1M	50
Microsoft-Phi (Abdin et al., 2024)	Phi-3-mini-128k-instruct	3.8B	128k	50
	Phi-3-small-128k-instruct	7B	128k	50
	Phi-3-medium-128k-instruct	14B	128k	50
	Phi-3.5-mini-instruct	3.8B	128k	50
	Phi-3.5-MoE-instruct	42B (active: 6.6B)	128k	50
Infly-OpenCoder (Huang et al., 2024)	OpenCoder-8B-Instruct	8B	8k	50
Deepseek-ai (Guo et al., 2024)	deepseek-coder-33b-instruct	33B	16k	50
	deepseek-chat-v3-0324	685B (active: 37B)	64K–164K	50
	deepseek-r1	671B (active: 37B)	64K–164K	50

The parameter count of proprietary models is not documented and marked with a question mark (?) Here. proprietary model families are OpenAI-GPT, Google-Gemini and Anthropic-Claude.

4.1.3. Selection of Open LLMs

We based our selection of state-of-the-art open LLMs on the Open LLM Leaderboard (Fourrier et al., 2024) and used the average score over all included benchmarks as our reference value. The selection criteria were that the model is instruction-finetuned, as required by the task construction of the LLM-KG-Bench framework, has less than 80B parameters because of a limited amount of available hardware resources and is a base model, that is, not a fine-tuned version of another model. With the latter requirement, we wanted to stick to mature and popularly used LLMs that are not just optimized to achieve a slightly higher score on one or few benchmarks than a base model.

The models fulfilling all our criteria and were among the TOP-4 models based on the average benchmark scores, disregarding models of the same family with lower scores, were Qwen2-72B-Instruct, Meta-Llama-3.1-70B-Instruct, solar-pro-preview-instruct and Phi-3.5-MoE-instruct. Here, we excluded solar-pro-preview-instruct from our selection since it only supports a context length of up to 4096 Tokens and not all prompts of tasks included in the run fitted within this limit. For the remaining three models, we also included all models of their larger model families that matched our requirements, that is, we also tested all models of the Llama3 (Grattafiori et al., 2024), Qwen2 (Hui et al., 2024; Yang et al., 2024), and Phi3 (Abdin et al., 2024) families fulfilling our requirements.

In addition, we wanted to test open LLMs that are fine-tuned or explicitly optimized on code since they could potentially better understand and produce structured data as required for the tasks included in the LLM-KG-Bench. Here, we consulted the EvalPlus Leaderboard (Liu et al., 2024a) and used the reported models’ Mostly Basic Python Programming (MBPP) Benchmark score as our reference value to assess the code-producing quality of the models. Again, we excluded models that were only finetuned versions of a code-finetuned or -optimized base model, had more than 80B parameters or were not instruction-finetuned. Moreover, we only searched for models that are fine-tuned or explicitly optimized on code. Finally, we included the Top-3 models satisfying the criteria in our runs, namely Qwen2.5-Coder-32B-Instruct (Hui et al., 2024), DeepSeek-Coder-33B-Instruct (Guo et al., 2024) and OpenCoder-8B-Instruct (Huang et al., 2024).

After this selection of 37 open LLMs and a first evaluation round end of 2024, we extended the list by the following 4 high ranking LLMs from this families with later release dates until April 2025: deepseek-R1, deepseek-chat-v3-0324, Llama-4.0-Maveric and Qwen3.0-235b-a22b.

4.2. Experimental Results

With the configuration described we generated a dataset containing the detailed conversations and the evaluations for 20 to 50 iterations per task and model. In the following section we show some analysis on tasks, models and the preference on input KG format.

4.2.1. Task Result Examples

For each task, a box plot was generated, containing the individual values for the most important score for each model tested. Due to limited space, we show a selection for three tasks in Figure 5. The rest is available in the dataset repository.

Figure 5.

Selection of box plots for task score values of the LLMs evaluated. The arithmetic mean value is indicated with a circle, individual score values for each task iteration are indicated with crosses. (a) SparqlSyntaxFixing (SSF): max_combined score and (b) Sparql2Answer (S2A): combinedF1 score and (c) Text2Sparql (T2S) for Organizational-Numerical dataset: max_combined score.

Our evaluation of SPARQL-related capabilities is organized around the task types SparqlSyntaxFixing (SSF), Sparql2Answer (S2A) and Text2Sparql (T2S). For SSF and T2S we selected the max_combined score, which is $0.2$ for syntactically correct but wrong queries. Figures 5(a) and 5(c) show several LLMs that seem to have no problem answering with a syntactically correct SPARQL query with scores $\geq 0.2$ . As Figure 5(b) shows, several LLMs answered correct with the expected result set for the given SPARQL SELECT queries. In Figure 5(c), a box plot for the difficult task T2S for the Organizational-Numerical dataset is shown. Several semantically wrong answers were encountered, but Claude 3.5 Haiku & Sonnet, GPT 4o and Gemini 1.5 Pro provide good answers in this special case.

4.2.2. Capability Plots

To get a quick overview on LLM models, the LLM-KG-Bench framework can create capability plots. They aggregate result scores for each model into categories according to a configuration file. For the analysis of SPARQL capabilities we use the following categories:

Syn-1: Working with SPARQL SELECT queries with first answer. This includes the 0_answerParse scores (first answer per iteration) from task types SparqlSyntaxFixing (SSF) and Text2Sparql (T2S).

Syn-Max: Working with SPARQL SELECT queries with correction possibility in dialog. This includes the max_answerParse scores (best answer per iteration) from the same task types as Syn-1.

Sem-R: Working according to semantics of SPARQL SELECT queries. This includes the combinedF1 score of the Sparql2Answer (S2A) task type.

Sem-W-1: Creating semantically correct SPARQL SELECT queries with first answer. This includes the 0_combined scores (first answer per iteration) from Text2Sparql (T2S) task types.

Sem-W-Max: Creating semantically correct SPARQL SELECT queries with correction possibility in dialog. This includes the max_combined scores (best answer per iteration) from Text2Sparql (T2S) task types.

Figure 6 shows a selection of typical capability plots for LLMs. Plots for all LLMs evaluated can be found in the supplemental material.

Figure 6.

Selection of typical capability plots for SPARQL SELECT query-related capabilities of the evaluated LLMs. The scores of the models are aggregated along the following capabilities: Syn-1 = SPARQL Syntax of first answer; Syn-max = SPARQL Syntax of best answer per iteration; Sem-R = SPARQL Semantic Read; Sem-W-1 = SPARQL Semantic Write of first answer; Sem-W-max = SPARQL Semantic Write of best answer per iteration. The black line shows the mean value, the blue area the standard deviation. Scores range from 0 to best value 1. (a) very good: Claude-3.5-Sonnet, and similar, for example, Qwen-2.5-Coder, GPT 4o, Gemini-1.5-Pro, Deepseek-R1, (b) some problems: Qwen-2.5-7B, and similar, for example, GPT-3.5, Phi-3.5 MoE, (c) more problems: Phi-3-Medium, and similar, for example, Phi-3-small, Llama-3.1-8B, Qwen-2.5-3B, (d) some problems with S2A task: DeepSeek-Coder and (e) high gain from dialog: Llama-3-8B, and similar Llama-3.1-8B, (f) Only Syntax good: Qwen-2.5-1.5B, and similar, for example, Qwen-2.0-1.5B and Phi-3-Mini.

About a quarter of the models have plots similar to Figure 6(a) showing an almost perfect result with only slight problems with the writing of semantically correct SPARQL SELECT queries. For these models the framework would benefit from the inclusion of more complex benchmark datasets.

The rest of the models had more problems with answering the tasks resulting in plots like Figures 6(b) and 6(c). Moreover, some very small models from Llama and Qwen family had even worse plots.

Two models, DeepSeek-Coder and OpenCoder revealed strong problems with the Sparql2Answer Task type resulting in low values for the Semantic Read capabilities. This is especially interesting, as their results in the rest of the categories are very good.

Figure 6(e) shows Llama-3-8B benefits from the correction infos in the dialog. The first answer was often improved during the dialog. Other LLMs evaluated here seem to have fewer benefit from the dialog.

Understanding the syntax of SPARQL SELECT queries seems to be much easier for LLMs than the other capabilities. Most models show syntax understanding capability scores that are higher than the rest of the capability scores. Especially small models have a capability plot similar to Figure 6(f), showing a good syntax understanding but big problems with the other capabilities.

Aside from the capability compasses, we also aggregated all scores in Table 6. Although the values behind these scores have a high variance, the scores can give a first impression of the SPARQL capabilities. Since the results are very close together near the top, it is hard to make out a clear winner. For these runs of experiments, Claude 3.5 Haiku has scored the best results, closely followed by other large commercial models. In order to distill a list of highest scoring models, we chose Claude 3.5 Haiku as our anchor point and conducted pairwise t-tests with each of the models below it to find the point at which a significant difference in performance could be observed. In other words, the first test was conducted between Claude 3.5 Haiku and Claude 3.5 Sonnet, the next test on Claude 3.5 Haiku and Qwen-2.5-Coder-32B, and so forth. We chose $α = 0.05$ for these tests as this is a common default. For this series of experiments, the list of highest scoring models ends with Gemini 1.5 Pro. It is interesting to see that GPTo1-pre 2024/09 and Qwen-2.5-72B did not make the list, while Gemini 1.5 Pro did. This can be attributed to the fact that we chose a smaller sample size for propietary models, and the higher variance in scores for Gemini 1.5 Pro compared to the other two makes it harder to differentiate it from Claude 3.5 Haiku. In a similar vein, Gemini 1.5 Flash only barely fails this test with a $p$ -value of $0.039$ , so depending on the $α$ , it could be included or not. However, starting at GPTo1-mini 2024/09 and below, the $p$ value drops down to $0.0001$ and lower, meaning that below Gemini 1.5 Flash none of the models would make it into the leader group. The language models that show no significant difference in performance compared to Claude 3.5 Haiku with our chosen $α$ of $0.05$ are highlighted in bold and prefixed with a star (*) in Table 6.

Table 6.

Combined Scores of the SPARQL-related Capabilities of the LLMs Tested.

ModelName	Score
*Claude 3.5 Haiku	0.94 $(\pm 0.20)$
*Claude 3.5 Sonnet	0.94 $(\pm 0.21)$
*Qwen-2.5-Coder-32B	0.93 $(\pm 0.22)$
*GPT4o 2024/11	0.91 $(\pm 0.22)$
GPTo1-pre 2024/09	0.91 $(\pm 0.25)$
*Gemini 1.5 Pro	0.90 $(\pm 0.27)$
Gemini 1.5 Flash	0.90 $(\pm 0.28)$
GPTo1-mini 2024/09	0.89 $(\pm 0.27)$
Llama-3.1-70B	0.89 $(\pm 0.27)$
Deepseek-R1	0.90 $(\pm 0.27)$
Qwen-3-235B	0.89 $(\pm 0.28)$
Llama-4-Maverick	0.87 $(\pm 0.29)$
Deepseek-Chat-v3	0.87 $(\pm 0.29)$
GPT4o-mini 2024/07	0.87 $(\pm 0.29)$
Qwen-2.0-72B	0.85 $(\pm 0.30)$
Qwen-2.5-32B	0.81 $(\pm 0.35)$
GPT3.5 2024/01	0.80 $(\pm 0.34)$
Phi-3.5-MoE	0.74 $(\pm 0.35)$
Qwen-2.5-7B	0.73 $(\pm 0.39)$
Deepseek-Coder-33B	0.73 $(\pm 0.38)$
Qwen-2.5-72B	0.91 $(\pm 0.25)$	Qwen-2.0-57B-A14B	0.70 $(\pm 0.41)$
Qwen-2.0-7B	0.58 $(\pm 0.46)$
Qwen-2.5-3B	0.57 $(\pm 0.44)$
OpenCoder-8B	0.55 $(\pm 0.45)$
Phi-3.0-medium-128k	0.50 $(\pm 0.44)$
Phi-3.5-mini	0.49 $(\pm 0.44)$
Llama-3.1-8B	0.42 $(\pm 0.44)$
Llama-3.0-8B	0.41 $(\pm 0.44)$
Phi-3.0-mini-128k	0.39 $(\pm 0.41)$
Phi-3.0-small-128k	0.39 $(\pm 0.43)$
Llama-3.0-70B	0.86 $(\pm 0.30)$	Qwen-2.5-1.5B	0.32 $(\pm 0.41)$
Llama-3.3-70B	0.85 $(\pm 0.31)$	Llama-3.2-3B	0.23 $(\pm 0.36)$
Qwen-2.0-1.5B	0.20 $(\pm 0.34)$
Gemini 2.0 Flash Exp	0.85 $(\pm 0.31)$	Qwen-2.5-0.5B	0.11 $(\pm 0.27)$
Qwen-2.5-14B	0.81 $(\pm 0.34)$	Llama-3.2-1B	0.03 $(\pm 0.14)$
Qwen-2.0-0.5B	0.01 $(\pm 0.10)$

The categories combined into the single score are described in section 4.2.2. Models prefixed with a star (*) show no significant difference in performance compared to Claude 3.5 Haiku.

4.2.3. Model Preferences on Input Format

After running each task both on Turtle (TTL) and JSON-LD serializations of the same knowledge graph, we wanted to find out whether there is a preference for a certain format (i.e., a higher score when using one format over the other). To that end, we conducted several t-tests; both on the task level and the model level. The latter helps us understand if any given model has an overall preference for a certain input format, while the former gives us a better understanding on a per-task basis.

Looking at Table 7, we can see that seven models have an overall preference for TTL, while eight models performed overall better on JSON-LD serializations. In contrast to that, 25 models do not have any kind of preference, leaving us with the conclusion that there is no universally preferred format.

Table 7.
Result of Two-Sided t-Tests Checking for a Preference for Turtle (TTL) vs. JSON-LD Serialization.

AllTasks Rdf Connection ExplainStatic Rdf Friend Count-1 Rdf Friend Count-2 Rdf Syntax FixList Sparql2 Answer List Orga Text2 Answer List Orga

Claude 3.5 Haiku TTL JSON TTL TTL – – –

Claude 3.5 Sonnet – – – – – – –

Deepseek-Coder-33B JSON – JSON JSON JSON – –

deepseek-chat-v3-0324 – JSON TTL TTL – – –

deepseek-r1 – – – – – – –

GPT3.5 2024/01 – – JSON JSON – – TTL

GPT4o 2024/11 – – – – JSON – –

GPT4o-mini 2024/07 TTL – – TTL – – TTL

GPTo1-mini 2024/09 – – – – – – –

GPTo1-pre 2024/09 – – – – – – –

Gemini 1.5 Flash TTL – TTL – – – –

Gemini 1.5 Pro – – – – – – –

Gemini 2.0 Flash Exp – – – – – – –

Llama-3.3-70B JSON – JSON JSON JSON – –

Meta-Llama-3-70B JSON – JSON JSON JSON – –

Meta-Llama-3-8B – TTL – – – – –

Meta-Llama-3.1-70B – – – – JSON TTL –

Meta-Llama-3.1-8B JSON – JSON JSON JSON – –

Meta-Llama-3.2-1B JSON – – – JSON – –

Meta-Llama-3.2-3B – JSON TTL TTL JSON – –

llama-4-maverick JSON – JSON JSON – – –

OpenCoder-8B – – – – – TTL –

Phi-3-medium-128k TTL TTL – JSON – – TTL

Phi-3-mini-128k – – – JSON JSON – –

Phi-3-small-128k – JSON – – TTL – –

Phi-3.5-MoE JSON – JSON JSON JSON TTL –

Phi-3.5-mini – TTL JSON – JSON TTL –

Qwen2-0.5B – – – JSON – – –

Qwen2-1.5B JSON TTL JSON JSON JSON – –

Qwen2-7B – JSON – – – – TTL

Qwen2-57B-A14B TTL – – – JSON TTL TTL

Qwen2-72B – – – – JSON – –

Qwen2.5-0.5B – – – – – – –

Qwen2.5-1.5B – – – – JSON – –

Qwen2.5-14B – TTL – – – – TTL

Qwen2.5-32B TTL – TTL TTL JSON JSON –

Qwen2.5-3B – JSON – – JSON – –

Qwen2.5-7B TTL JSON – – – TTL TTL

Qwen2.5-72B – JSON JSON JSON – – TTL

Qwen2.5-Coder-32B TTL – TTL TTL – – –

Qwen-3-235B – – – – – – –

All Models – JSON JSON JSON TTL TTL

	AllTasks	Rdf Connection ExplainStatic	Rdf Friend Count-1	Rdf Friend Count-2	Rdf Syntax FixList	Sparql2 Answer List Orga	Text2 Answer List Orga
Claude 3.5 Haiku	TTL	JSON	TTL	TTL	–	–	–
Claude 3.5 Sonnet	–	–	–	–	–	–	–
Deepseek-Coder-33B	JSON	–	JSON	JSON	JSON	–	–
deepseek-chat-v3-0324	–	JSON	TTL	TTL	–	–	–
deepseek-r1	–	–	–	–	–	–	–
GPT3.5 2024/01	–	–	JSON	JSON	–	–	TTL
GPT4o 2024/11	–	–	–	–	JSON	–	–
GPT4o-mini 2024/07	TTL	–	–	TTL	–	–	TTL
GPTo1-mini 2024/09	–	–	–	–	–	–	–
GPTo1-pre 2024/09	–	–	–	–	–	–	–
Gemini 1.5 Flash	TTL	–	TTL	–	–	–	–
Gemini 1.5 Pro	–	–	–	–	–	–	–
Gemini 2.0 Flash Exp	–	–	–	–	–	–	–
Llama-3.3-70B	JSON	–	JSON	JSON	JSON	–	–
Meta-Llama-3-70B	JSON	–	JSON	JSON	JSON	–	–
Meta-Llama-3-8B	–	TTL	–	–	–	–	–
Meta-Llama-3.1-70B	–	–	–	–	JSON	TTL	–
Meta-Llama-3.1-8B	JSON	–	JSON	JSON	JSON	–	–
Meta-Llama-3.2-1B	JSON	–	–	–	JSON	–	–
Meta-Llama-3.2-3B	–	JSON	TTL	TTL	JSON	–	–
llama-4-maverick	JSON	–	JSON	JSON	–	–	–
OpenCoder-8B	–	–	–	–	–	TTL	–
Phi-3-medium-128k	TTL	TTL	–	JSON	–	–	TTL
Phi-3-mini-128k	–	–	–	JSON	JSON	–	–
Phi-3-small-128k	–	JSON	–	–	TTL	–	–
Phi-3.5-MoE	JSON	–	JSON	JSON	JSON	TTL	–
Phi-3.5-mini	–	TTL	JSON	–	JSON	TTL	–
Qwen2-0.5B	–	–	–	JSON	–	–	–
Qwen2-1.5B	JSON	TTL	JSON	JSON	JSON	–	–
Qwen2-7B	–	JSON	–	–	–	–	TTL
Qwen2-57B-A14B	TTL	–	–	–	JSON	TTL	TTL
Qwen2-72B	–	–	–	–	JSON	–	–
Qwen2.5-0.5B	–	–	–	–	–	–	–
Qwen2.5-1.5B	–	–	–	–	JSON	–	–
Qwen2.5-14B	–	TTL	–	–	–	–	TTL
Qwen2.5-32B	TTL	–	TTL	TTL	JSON	JSON	–
Qwen2.5-3B	–	JSON	–	–	JSON	–	–
Qwen2.5-7B	TTL	JSON	–	–	–	TTL	TTL
Qwen2.5-72B	–	JSON	JSON	JSON	–	–	TTL
Qwen2.5-Coder-32B	TTL	–	TTL	TTL	–	–	–
Qwen-3-235B	–	–	–	–	–	–	–
All Models		–	JSON	JSON	JSON	TTL	TTL

Preferences are expected if the confidence interval is at least 95%, Bold font indicates 99% or better.

Looking at the Table 7 as a whole, we can see that preferences for one format over the other appear roughly equally distributed. However, there are some clusters like the Llama family of models preferring JSON, or Qwen leaning more toward TTL.

This might have to do with the amount of training data that covers TTL versus that which covers JSON. Unfortunately, as not all training data is publicly available, we cannot investigate this further.

When looking at the global preferences per task, we find that the Text2Answer and Sparql2Answer tasks appear to be easier to solve when operating on a TTL-formatted KG. In contrast to that, counting friends seems to work better for JSON KGs. This could come from the fact that, in order to find the person with the most incoming connections, the model needs to be able to parse the knowledge graph and have some notion of what an array is and how to count. In contrast, extracting facts (or answers in that case) may profit from having the desired answers close to the question subjects , which is a strength of TTL.

Looking at the table, it is important to note that a model having no preference could mean one of these two things:

The model performs equally well on both formats and has no problem solving the task, no matter what. This is the case for Gemini 1.5 Pro, GPTo1 and some others.

The model has big problems solving the task. This is, for instance, the case for Qwen2 0.5B.

The former is a desired state, as this relieves the user of the burden of having to choose a serialization that maximizes the quality of the results, while models having preferences in certain areas indicate that they are still growing toward that state. To find out whether a model falls into the former or the latter category, one should cross-reference the table with the corresponding capability compass.

5. Discussion and Outlook

Coming back to the research questions, the LLM-KG-Bench framework helps answering them.

RQ1. We showed here and in previous works (Frey et al., 2023; Meyer et al., 2024) the possibilities for automated evaluation of KGE-related capabilities. The LLM-KG-Bench framework is open for further development to support additional KGE task areas.

RQ2. In the analysis of the experiment result and in previous work (Frey et al., 2023, 2025; Heim et al., 2025; Meyer et al., 2024) we could identify ways to distinguish the KGE-related capabilities of different LLMs. For SPARQL SELECT queries we propose here the scores calculated in Table 6 as guidance.

RQ3. At least for the tasks evaluating the understanding of SPARQL SELECT queries syntax seems to be possible most of the time for most evaluated LLMs as shown in Figure 5(a) and in Meyer et al. (2024). Further work is needed to check this claim for more complex queries and errors.

RQ4. Figure 5(b) shows at least some LLMs seem to have no problems with working according to the semantics of SPARQL SELECT queries.

RQ5. At least for the evaluated tasks, the creation of semantically correct SPARQL SELECT queries seems to be possible most of the time for some LLMs, when looking at Figure 5(c) and the good capability plot type in Figure 6(a). However, this also needs to be checked for other more complex queries. Moreover, it is important to consider that the evaluation is currently only done based on the answer set, which does not necessarily mean that the semantic of the SPARQL SELECT query is correct.

Still There are Some Things to Keep in Mind. Some of the tasks, like Sparql2Answer or RdfFriendCount, can be answered with graph databases much more efficiently and reliable. But the measuring of this RDF-related capabilities is of a more fundamental scientific interest. On the other side, the focus on this basis tasks induce a risk of overestimating the KGE-related capabilities of LLMs, as real world scenarios are often of a more complex nature.

The answer quality of LLMs can vary even on small changes in prompt or task. We tested only with a limited amount of tasks and task entries.

The automated evaluation could miss important aspects of the answer. For example, the SPARQL SELECT queries generated were tested only extensionally on the result set generated, which does not always imply that the query was well formulated to match the expected semantics.

Although we tried to hide the benchmark data, we can not guarantee no LLM is trained with them, as we want to keep the benchmark and its results open and reproducible.

Further Work. is needed to include more datasets into the framework, extend the list of tasks to cover more aspects related to Knowledge Graphs and extend the analytical capabilities of the framework. The framework and the evaluation data generated is publicly available to enable contribution.

The framework would benefit from more datasets and especially more evaluation task classes. We see here on the one hand additional topics like OWL, RML, SHACL, and on the other hand more complex tasks and analyses like nested SPARQL queries or even the interaction with real graph databases.

As the number of LLMs published and the size of datasets generated with the framework grow, new ways to check the results are needed. Maybe one could utilize another level of LLMs to check on the automated evaluation results for details missed and highlight these findings for manual inspection. This way, the evaluation could remain deterministic and reproducible, but we gain another level of checks and maybe even insight.

Supplemental Material

sj-zip-1-swj-10.1177_22104968261461351 - Supplemental material for Evaluating Large Language Models for RDF Knowledge Graph-Related Tasks—The LLM-KG-Bench-Framework 3

Supplemental material, sj-zip-1-swj-10.1177_22104968261461351 for Evaluating Large Language Models for RDF Knowledge Graph-Related Tasks—The LLM-KG-Bench-Framework 3 by Lars-Peter Meyer, Johannes Frey, Felix Brei, Desiree Heim, Sabine Gründer-Fahrer, Sara Todorovikj, Claus Stadler, Markus Schröder, Natanael Arndt and Michael Martin in Semantic Web

Supplemental Material

sj-zip-2-swj-10.1177_22104968261461351 - Supplemental material for Evaluating Large Language Models for RDF Knowledge Graph-Related Tasks—The LLM-KG-Bench-Framework 3

Supplemental material, sj-zip-2-swj-10.1177_22104968261461351 for Evaluating Large Language Models for RDF Knowledge Graph-Related Tasks—The LLM-KG-Bench-Framework 3 by Lars-Peter Meyer, Johannes Frey, Felix Brei, Desiree Heim, Sabine Gründer-Fahrer, Sara Todorovikj, Claus Stadler, Markus Schröder, Natanael Arndt and Michael Martin in Semantic Web

Footnotes

Funding

The authors thank DFKI for running the experiments involving open LLMs on their GPU cluster. This work was partially supported by grants from the German Federal Ministry of Research, Technology and Space (BMFTR) to the projects StahlDigital (13XP5116B), ScaleTrust (16DTM312D), KupferDigital2 (13XP5230L) and PROVIDER (13N17700) as well as from the German Federal Ministry for Economic Affairs and Climate Action (BMWK) to the KISS project (01MK22001A) and CoyPu project (01MK21007A) and from the German Federal Ministry of Transport and Digital Infrastructure (BMDV) to the project ADA (19F2190B).

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Supplemental Material

Supplemental material for this article is available online.

ORCID iDs

Lars-Peter Meyer

Johannes Frey

Felix Brei

Desiree Heim

Sabine Gründer-Fahrer

Sara Todorovikj

Claus Stadler

Markus Schröder

Natanael Arndt

Michael Martin

Notes

References

Abdin

Aneja

Awadalla

Awadallah

Awan

A. A.

Bach

Bahree

Bakhtiari

Bao

Behl

Benhaim

Bilenko

Bjorck

Bubeck

Cai

Chaudhary

Chen

, ... & Zhou

(2024). Phi-3 technical report: A highly capable language model locally on your phone. https://arxiv.org/abs/2404.14219.

Anderson

L. W.

Krathwohl

D. R.

(2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Addison Wesley Longman, Inc.

Avila

C. V. S.

Vidal

V. M. P.

Franco

Casanova

M. A.

(2024). Experiments with text-to-sparql based on chatgpt. In 18th IEEE International conference on semantic computing, ICSC 2024, Laguna Hills, CA, USA, February 5-7, 2024 (pp. 277–284). IEEE. https://doi.org/10.1109/ICSC59802.2024.00050

Bai

Liu

Zhou

Lin

Zheng

Ouyang

(2024). Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In Proceedings of the 62nd annual meeting of the association for computational linguistics (Volume 1: Long Papers) (p. 7421–7454). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.acl-long.401.

Banerjee

Nair

P. A.

Kaur

J. N.

Usbeck

Biemann

(2022). Modern baselines for sparql semantic parsing. In Proceedings of the 45th international ACM sigir conference on research and development in information retrieval, SIGIR ’22. ACM. https://doi.org/10.1145/3477495.3531841

Bloom

B. S.

(1956). Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain. New York: Longman.

Bordes

Usunier

Chopra

Weston

(2015). Large-scale simple question answering with memory networks.

Brei

Bühmann

Frey

Gerber

Meyer

L. P.

Stadler

Bulert

(2025). ARUQULA—an LLM based Text2SPARQL approach using ReAct and knowledge graph exploration utilities. In S. Tramp, M. Gôlo, E. Marx, & P. V. do Carmo (Eds.), Proceedings of the first international TEXT2SPARQL challenge, Co-Located with Text2KG at ESWC25, June 01, 2025, Portorož, Slovenia, CEUR Workshop Proceedings (Vol. 4094, pp. 49–63). https://doi.org/10.48550/arXiv.2510.02200

Brei

Frey

Meyer

L. P.

(2024). Leveraging small language models for Text2SPARQL tasks to improve the resilience of AI assistance. In J. Holze, S. Tramp, M. Martin, S. Auer, R. Usbeck, & N. Krdzavac (Eds.), Proceedings of the third international workshop on linked data-driven resilience research 2024 (D2R2’24), colocated with ESWC 2024, CEUR Workshop Proceedings (Vol. 3707). CEUR-WS.org. https://ceur-ws.org/Vol-3707/D2R224_paper_5.pdf.

10.

Bustamante

Takeda

(2024). Sparql generation with entity pre-trained gpt for kg question answering.

11.

Chiang

W. L.

Zheng

Sheng

Angelopoulos

A. N.

Zhu

Zhang

Jordan

Gonzalez

J. E.

Stoica

(2024). Chatbot arena: An open platform for evaluating LLMs by human preference. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, & F. Berkenkamp (Eds.), Proceedings of the 41st international conference on machine learning, Proceedings of Machine Learning Research (Vol. 235, pp. 8359–8388). PMLR.

12.

Diallo

P. A. K. K.

Reyd

Zouaq

(2024). A comprehensive evaluation of neural sparql query generation from natural language questions.

13.

Diefenbach

Tanon

T. P.

Singh

K. D.

Maret

(2017). Question answering benchmarks for wikidata. In N. Nikitina, D. Song, A. Fokoue, & P. Haase (Eds.), Proceedings of the ISWC 2017 Posters & demonstrations and industry tracks co-located with 16th international semantic web conference (ISWC 2017), Vienna, Austria, October 23–25, 2017, CEUR Workshop Proceedings (Vol. 1963). CEUR-WS.org. https://ceur-ws.org/Vol-1963/paper555.pdf.

14.

Dubey

Banerjee

Abdelkawi

Lehmann

(2019). Lc-quad 2.0: a large dataset for complex question answering over wikidata and dbpedia. In Proceedings of the 18th international semantic web conference (ISWC). Springer. https://doi.org/10.1007/978-3-030-30796-7_5

15.

Fourrier

Habib

Lozovskaya

Szafer

Wolf

(2024). Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard.

16.

Frey

Meyer

Arndt

Brei

Bulert

(2023). Benchmarking the abilities of large language models for RDF knowledge graph creation and comprehension: How well do LLMs speak Turtle? In M. Alam & M. Cochez (Eds.), Proceedings of the workshop on deep learning for knowledge graphs (DL4KG 2023) co-located with the 21th International Semantic Web Conference (ISWC 2023), Athens, November 6–10, 2023, CEUR Workshop Proceedings (Vol. 3559). CEUR-WS.org. https://ceur-ws.org/Vol-3559/paper-3.pdf.

17.

Frey

Meyer

L. P.

Brei

Gruender

Martin

(2025). Assessing the evolution of LLM capabilities for knowledge graph engineering in 2023. In The Semantic Web: ESWC 2024 satellite events (pp. 51–60). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-78952-6_5

18.

Grattafiori

Dubey

Jauhri

Pandey

Kadian

Al-Dahle

Letman

Mathur

Schelten

Vaughan

Yang

Fan

Goyal

Hartshorn

Yang

Mitra

Sravankumar

Korenev

Hinsvark

Rao

,...& Ma

(2024). The llama 3 herd of models. https://arxiv.org/abs/2407.21783.

19.

Guo

Zhu

Yang

Xie

Dong

Zhang

Chen

Y. K.

Luo

Xiong

Liang

(2024). Deepseek-coder: When the large language model meets programming—The rise of code intelligence. CoRR abs/2401.14196. https://doi.org/10.48550/arXiv.2401.14196.

20.

Halford

G. S.

Wilson

W. H.

Phillips

(1998). Processing capacity defined by relational complexity: Implications for comparative, developmental, and cognitive psychology. Behavioral and Brain Sciences, 21(6), 803–831. 10.1017/S0140525X98001769

21.

Heim

Meyer

L. P.

Schröder

Frey

Dengel

(2025). How do scaling laws apply to knowledge graph engineering tasks? The impact of model size on large language model performance. In ESWC 2025 Workshops and tutorials joint proceedings, CEUR Workshop Proceedings (Vol. 3977). https://ceur-ws.org/Vol-3977/elmke-3.pdf.

22.

Hirigoyen

Zouaq

Reyd

(2022). A copy mechanism for handling knowledge base elements in SPARQL neural machine translation. In Y. He, H. Ji, S. Li, Y. Liu, & C. H. Chang (Eds.), Findings of the association for computational linguistics: AACL-IJCNLP 2022 (pp. 226–236). Online only: Association for Computational Linguistics. https://aclanthology.org/2022.findings-aacl.22.

23.

Hofer

Frey

Rahm

(2024). Towards self-configuring knowledge graph construction pipelines using LLMS—A case study with RML. In Fifth international workshop on knowledge graph construction @ ESWC2024, CEUR Workshop Proceedings (Vol. 3718). CEUR-WS.org. https://ceur-ws.org/Vol-3718/paper6.pdf.

24.

Huang

Cheng

Liu

J. K.

Hao

Song

Yang

Liu

J. H.

Zhang

Chai

Yuan

Zhang

Liu

Zhang

Wang

Chu

(2024). Opencoder: The open cookbook for top-tier code large language models. CoRR abs/2411.04905. https://doi.org/10.48550/arXiv.2411.04905

25.

Hui

Yang

Cui

Yang

Liu

Zhang

Liu

Zhang

Dang

Fan

Zhang

Yang

Men

Huang

Zheng

Miao

Quan

Feng

Ren

Zhou

Lin

(2024). Qwen2.5-coder technical report. https://arxiv.org/abs/2409.12186.

26.

Kalo

J. C.

Razniewski

Zhang

Nguyen

T. P.

(2025). Lm-kbc 2025: 4th challenge on knowledge base construction from pre-trained language models. In S. Razniewski, J. C. Kalo, D. Islakoglu, T. P. Nguyen, & B. Zhang (Eds.), Joint Proceedings of the 3rd workshop on knowledge base construction from pre-trained language models and the 4th challenge on language models for knowledge base construction (KBC-LM+LM-KBC 2025)co-located with the 24th International Semantic Web Conference (ISWC 2025), CEUR Workshop Proceedings (Vol. 4041). https://ceur-ws.org/Vol-4041/paper7.pdf.

27.

Kovriguina

Teucher

Radyush

Mouromtsev

(2023). SPARQLGEN: One-shot prompt-based approach for SPARQL query generation. In International conference on semantic systems, CEUR Workshop Proceedings (Vol. 3526). CEUR-WS.org. https://ceur-ws.org/Vol-3526/paper-08.pdf.

28.

Kwon

Zhuang

Sheng

Zheng

C. H.

Gonzalez

J. E.

Zhang

Stoica

(2023). Efficient memory management for large language model serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th symposium on operating systems principles.

29.

Lehmann

Ferré

Vahdati

(2023). Language models as controlled natural language semantic parsers for knowledge graph question answering. In K. Gal, A. Nowé, G. J. Nalepa, R. Fairstein, & R. Radulescu (Eds.), ECAI 2023—26th European conference on artificial intelligence, september 30–October 4, 2023, Kraków, Poland—Including 12th Conference on Prestigious Applications of Intelligent Systems (PAIS 2023), Frontiers in Artificial Intelligence and Applications (Vol. 372, pp. 1348–1356). IOS Press. https://doi.org/10.3233/FAIA230411

30.

Fan

Duan

Dong

Liu

Wang

(2024). Flexkbqa: A flexible llm-powered framework for few-shot knowledge base question answering. https://doi.org/10.1609/aaai.v38i17.29823

31.

Liang

Bommasani

Lee

Tsipras

Soylu

Yasunaga

Zhang

Narayanan

Kumar

Newman

Yuan

Yan

Zhang

Cosgrove

Manning

C. D.

Ré

Acosta-Navas

Hudson

D. A.

Zelikman

,...& Koreeda

(2023). Holistic evaluation of language models. Transactions on Machine Learning Research. https://arxiv.org/abs/2211.09110 .

32.

Liu

Xia

C. S.

Wang

Zhang

(2024a). Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36.

33.

Liu

Semnani

Triedman

Zhao

I. D.

Lam

(2024b). Spinach: Sparql-based information navigation for challenging real-world questions. In Findings of the association for computational linguistics: EMNLP 2024 (pp. 15977–16001). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-emnlp.938

34.

Marx

do Carmo

P. V.

Gôlo

Tramp

(2025). Preface of the first international text2sparql challenge (text2sparql’25). In S. Tramp, M. Gôlo, E. Marx, & P. V. do Carmo (Eds.), Proceedings of the first international TEXT2SPARQL challenge, co-located with Text2KG at ESWC25, June 1, 2025, Portorož, Slovenia, CEUR Workshop Proceedings (Vol. 4094). https://ceur-ws.org/Vol-4094/preface.pdf.

35.

Meyer

L. P.

Frey

Brei

Arndt

(2024). Assessing SPARQL capabilities of large language models. In E. Vakaj, S. Iranmanesh, R. Stamartina, N. Mihindukulasooriya, S. Tiwari , F. Ortiz-Rodríguez, & R. Mcgranaghan (Eds.), Proceedings of the 3rd International workshop on natural language processing for knowledge graph creation co-located with 20th international conference on semantic systems (SEMANTiCS 2024), CEUR Workshop Proceedings (Vol. 3874, pp. 35–53). https://ceur-ws.org/Vol-3874/paper3.pdf.

36.

Meyer

L. P.

Frey

Heim

Brei

Stadler

Junghanns

Martin

(2025). LLM-KG-bench 3.0: A compass for semanticTechnology capabilities in the ocean of LLMs. The Semantic Web. ESWC 2025, 15719, 280–296. https://doi.org/10.1007/978-3-031-94578-6_16

37.

Meyer

L. P.

Frey

Junghanns

Brei

Bulert

Gründer-Fahrer

Martin

(2023a). Developing a scalable benchmark for assessing large language models in knowledge graph engineering. In N. Keshan, S. Neumaier, A. L. Gentile, & S. Vahdati (Eds.), Proceedings of the posters and demo track of the 19th international conference on semantic systems (SEMANTICS 2023), CEUR Workshop Proceedings (Vol. 3526). CEUR-WS.org. https://ceur-ws.org/Vol-3526/paper-04.pdf.

38.

Meyer

L. P.

Stadler

Frey

Radtke

Junghanns

Meissner

Dziwis

Bulert

Martin

(2023b). LLM-assisted knowledge graph engineering: Experiments with ChatGPT. In C. Zinke-Wehlmann & J. Friedrich (Eds.), First Working conference on artificial intelligence development for a resilient and sustainable tomorrow (AITomorrow) 2023 (pp. 101–112). Informatik aktuell. https://doi.org/10.1007/978-3-658-43705-3_8

39.

Mihindukulasooriya

Tiwari

Enguix

C. F.

Lata

(2023). Text2KGBench: A benchmark for ontology-driven knowledge graph generation from text. In T. R. Payne, V. Presutti, G. Qi, M. Poveda-Villalón, G. Stoilos, L. Hollink, Z. Kaoudi, G. Cheng, & J. Li (Eds.), The Semantic Web—ISWC 2023 (pp. 247–265). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-47243-5_14

40.

Pan

J. Z.

Razniewski

Kalo

J. C.

Singhania

Chen

Dietze

Jabeen

Omeliyanenko

Zhang

Lissandrini

Biswas

de Melo

Bonifati

Vakaj

Dragoni

Graux

(2023). Large language models and knowledge graphs: Opportunities and challenges. Transactions on Graph Data and Knowledge, 1(1), 2:1–2:38. https://doi.org/10.4230/TGDK.1.1.2

41.

Pan

Luo

Wang

Chen

Wang

(2024). Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering (TKDE), 36(7), 3580–3599. https://doi.org/10.1109/TKDE.2024.3352100

42.

Perevalov

Both

(2025). Text-to-sparql goes beyond English: Multilingual question answering over knowledge graphs through human-inspired reasoning. In S. Tramp, M. Gôlo, E. Marx, & P. V. do Carmo (Eds.), Proceedings of the first international TEXT2SPARQL challenge, co-located with Text2KG at ESWC25, June 1, 2025, Portorož, Slovenia, CEUR Workshop Proceedings (Vol. 4094, pp. 77–93). https://doi.org/10.48550/arXiv.2507.16971

43.

Petroni

Rocktäschel

Riedel

Lewis

Bakhtin

Miller

(2019). Language models as knowledge bases? In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). Association for Computational Linguistics. https://doi.org/10.18653/v1/d19-1250

44.

Pliukhin

Radyush

Kovriguina

Mouromtsev

(2023). Improving subgraph extraction algorihtms for one-shot sparql query generation with large language models. In 1st Scholarly QALD Challenge 2023 and 4th semantic answer type, relation and entity prediction tasks challenge 2023, CEUR Workshop Proceedings (Vol. 3592). CEUR-WS.org. https://ceur-ws.org/Vol-3592/paper6.pdf.

45.

Rangel

J. C.

de Farias

T. M.

Sima

A. C.

Kobayashi

(2024). Sparql generation: An analysis on fine-tuning openllama for question answering over a life science knowledge graph. To appear in Proceedings of SWAT4HCLS 2024: Semantic Web Tools and Applications for Healthcare and Life Sciences.

46.

Shorten

Pierse

Smith

T. B.

Cardenas

Sharma

Trengrove

van Luijt

(2024). Structuredrag: Json response formatting with large language models. https://arxiv.org/abs/2408.11061.

47.

Singhania

Nguyen

T. P.

Razniewski

(2022). Lm-kbc: Knowledge base construction from pre-trained language models. In S. Singhania, T. P. Nguyen, & S. Razniewski (Eds.), Proceedings of the semantic web challenge on knowledge base construction from pre-trained language models 2022co-located with the 21st International Semantic Web Conference (ISWC2022), CEUR Workshop Proceedings (Vol. 3274, pp. 1–10). https://ceur-ws.org/Vol-3274/paper1.pdf.

48.

Srivastava

Rastogi

Rao

Shoeb

A. A. M.

Abid

Fisch

Brown

A. R.

Santoro

Gupta

Garriga-Alonso

Kluska

Lewkowycz

Agarwal

Power

Ray

Warstadt

Kocurek

A. W.

Safaya

Tazarv

Xiang

,...& Wu

(2023). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.

49.

Trivedi

Maheshwari

Dubey

Lehmann

(2017). Lc-quad: A corpus for complex question answering over knowledge graphs. In C. d’Amato, M. Fernández, V. A. M. Tamma, F. Lécué, P. Cudré-Mauroux, J. F. Sequeda, C. Lange, & J. Heflin (Eds.), The Semantic Web—ISWC 2017—16th international semantic web conference, Vienna, Austria, October 21–25, 2017, Proceedings, Part II, Lecture Notes in Computer Science (Vol. 10588, pp. 210–218). Springer. https://doi.org/10.1007/978-3-319-68204-4_22

50.

Usbeck

Röder

Hoffmann

Conrads

Huthmann

Ngonga-Ngomo

A. C.

Demmler

Unger

(2019). Benchmarking question answering systems. Semantic Web, 10(2), 293–304. https://doi.org/10.3233/sw-180312

51.

Yang

Hui

Zheng

Zhou

Liu

Huang

Dong

Wei

Lin

Tang

Wang

Yang

Zhang

Yang

,... Fan

(2024). Qwen2 technical report. https://arxiv.org/abs/2407.10671.

52.

Zahera

H. M.

Ali

Sherif

M. A.

Moussallem

Ngonga Ngomo

A. C.

(2024). Generating sparql from natural language using chain-of-thoughts prompting. In SEMANTiCS 2024: Knowledge graphs in the age of language models and neuro-symbolic AI. Amsterdam, Netherlands. https://doi.org/10.3233/ssw240028

53.

Zhu

Wang

Chen

Qiao

Yao

Deng

Chen

Zhang

(2024). LLMs for knowledge graph construction and reasoning: Recent capabilities and future opportunities. World Wide Web, 27(5) https://doi.org/10.1007/s11280-024-01297-w

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

353.14 MB

13.06 MB

Evaluating Large Language Models for RDF Knowledge Graph-Related Tasks—The LLM-KG-Bench-Framework 3

Abstract

Keywords

1. Introduction

2. Related Work

2.1. LLM Evaluation

2.2. RDF-Related Evaluation

3. The LLM-KG-Bench Framework

3.1. Architecture and Main Concepts

3.3. LLM Connectors and Supported Models

3.4. Evaluation Tasks and Scores

Table 2. Mapping of the Task Types to the Different Task Aspects and Research Questions Covered. Adopted from Meyer et al. (2024).

3.6. General Task Characterization

4.1. Experimental Setup

4.1.1. Selected Tasks

4.1.2. Selection of Proprietary LLMs

4.2. Experimental Results

4.2.1. Task Result Examples

Supplemental Material

sj-zip-1-swj-10.1177_22104968261461351 - Supplemental material for Evaluating Large Language Models for RDF Knowledge Graph-Related Tasks—The LLM-KG-Bench-Framework 3

Supplemental Material

sj-zip-2-swj-10.1177_22104968261461351 - Supplemental material for Evaluating Large Language Models for RDF Knowledge Graph-Related Tasks—The LLM-KG-Bench-Framework 3

Footnotes

Funding

Declaration of Conflicting Interests

Supplemental Material

ORCID iDs

Notes

References

Supplementary Material

Table 2.
Mapping of the Task Types to the Different Task Aspects and Research Questions Covered. Adopted from Meyer et al. (2024).