A comparison of object-triple mapping libraries

Let’s consider a fictive national civil aviation authority (CAA), which is a public governmental body and is obliged to publish its data as open data. The CAA decides to develop a safety management system for the oversight of aviation organizations under its jurisdiction, involving safety occurrence reporting, safety issues definition, and aviation organization performance dashboard. The CAA can then concentrate on suspicious patterns identified by the system (e.g., from frequently occurring events) during its inspections at the aviation organizations or during safety guidelines preparation. To establish data and knowledge sharing between the CAA and the aviation organizations, CAA decides to share the safety data as 5-star Linked Data [81]. To achieve this, the system is designed on top of a proper ontological conceptualization of the domain. This allows capturing safety occurrence reports with deep insight into the actual safety situation, allowing to model causal dependencies between safety occurrences, describe event participants, use rich typologies of occurrences, aircraft types, etc. Furthermore, integration of other available Linked Data sources, e.g., a register of aviation companies (airlines, airports etc.) and aircraft helps to reveal problematic aviation organizations. A subset of the described functionality has been implemented in the SISel system [51]. Another system with related functionality aimed at the spacecraft accident investigation domain was built at NASA by Carvalho et al. [12,13].

To be usable by non-experts, the system is domain-specific, not a generic ontology editor/browser. No matter how the user interface is implemented, it is supposed to communicate with the backend through a well-designed domain-specific API. Business logic implementing such API requires efficient access to the underlying data. Using common Semantic Web libraries like Jena [11] or RDF4J [9] for such access ends up with a large amount of boilerplate code, as these libraries work on the level of triples/statements/axioms and do not provide frame-based knowledge structures. The solution is to allow developers to use the well-known object-oriented paradigm, which is well supported in IDEs and which provides compile-time checking of the object model structure and usage. However, to match objects to triples, one needs an object-triple mapping (OTM) that defines the contract between the application and the underlying knowledge structure. Many libraries trying to offer such a contract exist. However, these libraries differ a lot in their capabilities as well as efficiency, neither of which is systematically documented. In this survey paper, we provide a framework for comparing various OTM solutions in terms of features and performance. In addition, we use this framework and try to systematize the existing OTM approaches with the aim of helping developers to choose the most suitable one for their use case.

1.2. Contribution

The fundamental question of this paper is: “Which object-triple mapping solution is suitable for creating and managing object-oriented applications backed by an RDF triple store?”. The actual contribution of this paper can be split into three particular goals:

The remainder of this paper is structured as follows. Section 2 presents the necessary background on the RDF language, OTM systems, and Java performance benchmarks. Section 3 reviews existing approaches to comparing OTM libraries and benchmarking in the Semantic Web world in general. Section 4 presents the framework designed in this work. Section 5 introduces the OTM libraries chosen for the comparison and discusses the reasons for their selection. Sections 6 and 7 actually compare the selected libraries using the framework designed in Section 4. The paper is concluded in Section 8. Appendices A.1 and A.2 contain full reports of time performance, resp. scalability comparison.

Ontological data are typically stored in dedicated semantic databases or triple stores, such as RDF4J [9], GraphDB [5] or Virtuoso [22]. To be able to make use of these data, an application needs means of accessing the underlying triple store. Unfortunately, there is no common standard for accessing a semantic database (like ODBC [54] for relational database access). This lack of standardization gave rise to multiple approaches and libraries that can be basically split into two categories (as discussed in [46]):

Domain-specific libraries are more suitable for creating object-oriented applications. Such applications express the domain model in the form of object classes, and their properties (attributes or relationships). OTM, in this case, allows to automatically map the notions from the object-oriented realm to the semantic data model and vice versa. Domain-independent libraries perform no such thing. They treat the underlying data at the same level as the RDF language, i.e., at the level of statements, without any assumption about their meaning. Therefore, a domain-specific application attempting to use such libraries would either need to treat its data in the same manner, which makes it extremely difficult to understand, develop, and maintain; or the developer would end up writing their own version of an object-triple mapping library. Either way, it is more favorable to leverage dedicated object-triple mapping solutions. In addition, some OTM libraries allow accessing various data sources in a unified way, support transactional processing and compile-time checking of correct model usage. Table 1 shows a simplified mapping between RDF(S) terms and object model artifacts considered by the majority of the OTM solutions. We shall use these notions throughout this paper.

An analogous division where indirect and direct models correspond to the domain-independent and domain-specific approaches respectively is used in [61]. The authors argue that direct models are more suitable for domain-specific applications and propose a hybrid approach where parts of the application model are domain independent and parts are domain specific.

The difference between domain-independent and domain-specific libraries is similar to the difference between JDBC [40] and JPA [39] in the relational database world. JDBC is a row-based access technology, which requires a lot of boilerplate code. In addition, not all databases fully adhere to the SQL6

There exists a number of OTM libraries. However, their user base is still relatively small, which is reflected in the small amount of information available about them. Some libraries are tied to a specific underlying storage, others’ APIs do not reflect the specifics of semantic data access. Ultimately, one would like to be able to determine which of the libraries is the most suitable for his/her object-oriented application.

As was already stated, the developed comparison framework consists of a set of qualitative criteria and a performance benchmark. The benchmark is written in Java, which is hardly a surprise since Java is the most prominent language in the Semantic Web community. Therefore, approaches to performance testing in Java were also reviewed.

Java-based performance testing has certain specifics which are given by the nature of the Java platform where each program is compiled into an intermediate language – the bytecode – and interpreted at runtime by the Java virtual machine (JVM). The JVM is able to optimize frequently executed parts of a program. These optimizations occur during execution of the application. The JVM also automatically manages application memory via a mechanism called garbage collection which automatically frees memory occupied by objects that are no longer referenced. Of course, if the application forgets to discard objects it does not use anymore – it has a so-called memory leak – it may eventually run out of free memory. Although garbage collection is nowadays extremely efficient, it still requires CPU time and may influence the application performance. Java benchmarks thus usually take into account application throughput and its dependence on the amount of available memory (heap size). The best known benchmarks in this area are SPEC [71] and DaCapo [7]. DaCapo measures application throughput w.r.t. heap size. In addition, it samples heap occupation and composition and new object allocation rate. The authors of [29] count garbage collection events and the time JVM spends executing them.

Generally, when benchmarking Java applications, it is necessary to execute a warm-up period which allows the JVM to perform code optimizations. Otherwise, the results could register a significant speedup after the initial unoptimized phase. Georges et al. [28] discuss further practices for executing Java application benchmarks and interpreting their results, introducing a statistically rigorous methodology for Java benchmarking. Their methodology was used in the experiments in this work.

4.1. Feature criteria

We have designed a set of criteria, which can be used to evaluate an OTM library’s suitability in a particular use case. The criteria can be split into three main categories: General (GC)

General criteria are based on the principles known from application development and relevant object-relational mapping features.

GC1 – transactions Transactional processing, i.e., splitting work into individual, indivisible operations, is one of the fundamental paradigms of computer science [79]. It originated in databases but, due to its universally applicable principles, it is used throughout information systems with various levels of granularity. Nevertheless, support for transactions is one of the main features an OTM library has to provide, be it using an internal transaction engine or relying on the underlying storage.

Transactions are characterized by the ACID acronym, which stands for Atomicity of the transaction, eventual Consistency of data, Isolation of concurrent transactions and Durability of the transaction results. Given the complex nature of transaction isolation where several isolation levels exist,8

Fully satisfied if: The library supports ACD user-controlled transactions.

GC2 – storage access variability Storage access variability refers to the library’s ability to connect to semantic storages of various vendors. While application access to relational databases is standardized in two ways – the Open Database Connectivity (ODBC) [54] is a platform-independent database access API; the Java Database Connectivity (JDBC) [40] is a Java-based database access API (a JDBC driver can use an ODBC driver internally) – the semantic world lacks a corresponding standard. For instance, although RDF4J-based access is supported by most of the industry’s biggest repository developers, it is not possible to use it to connect to a Jena SDB store, which is backed by a relational database. To allow access to different storages, the OTM library has to account for this either by having modules for access to various storages or by defining an API whose implementations would provide access to the repositories. Another possibility, although probably not very efficient, could be accessing the storage using a SPARQL [23] or Linked Data Platform [69] endpoint.

Fully satisfied if: The library supports connecting to a triple store by means of at least two different APIs (e.g., Jena, RDF4J) or via a generic SPARQL endpoint API.

GC3 – query result mapping Besides reading individual instances with known identifiers, it is often necessary to read a set of instances corresponding to some search criteria (for example, read all reports with the given severity assessment which document events that occurred during the last month). Naturally, mapping of SPARQL query results to objects is expected, as SPARQL is the standard query language for the Semantic Web, but other languages like SeRQL (a Sesame query language) can be supported. A similar feature is supported by JPA [39] where SQL results can be mapped to entities or dedicated Java objects.

Fully satisfied if: SPARQL query results can be declaratively mapped to the object model (entities).

GC4 – object-level query language Related to GC3 is the question of query languages supported by the OTM library. While SPARQL is adequate in most situations, it can be cumbersome to work with, especially given the fact that one has to deal with IRIs and prefixes. When querying the data from an object-oriented application, it is convenient to be able to leverage the object model in the queries. This is again supported by JPA [39], where one can use the Java Persistence Query Language (JPQL) – a query language with SQL-like syntax but exploiting classes, attributes, and relationships instead of tables and columns. Furthermore, it also defines the Criteria API which allows one to construct queries dynamically at runtime using builder objects. Criteria API is especially suitable when a large number of optional filtering criteria can be used in the query and juggling with string concatenation would be difficult and error-prone. In addition, Criteria API provides the benefit of compile-time syntax checking, because it uses regular Java classes and methods.

Prototypical solutions already exist in the Semantic Web community. For instance, Hillairet et al. [36] use the Hibernate Query Language, an implementation of the standard Object Query Language [14] (OQL)9

Fully satisfied if: The library provides an object model-based query language which can be used to access the data (e.g., like JPQL).

GC5 – detached objects As pointed out in [37], applications need to be able to detach persistent objects from their connection to the storage in order to work with them, for instance, when the entity is passed up through web application layers and transferred over the network in response to a client request or stored in an application-level cache. The opposite (an object which is always managed) can lead to an excessive amount of retained connections to the storage, blocking the resources of the machine. The difference between detached and managed entities is indeed in that managed entities are tracked by the persistence library which watches for changes in these objects.

Fully satisfied if: The persistence library allows to attach/detach objects to/from the repository connection.

GC6 – code/ontology generator Setting up a database schema or an object model may seem like a one-time work but as the application evolves, so does often the schema. Keeping both the ontology schema and the object model in sync automatically can spare the developer a decent amount of time and bugs (e.g., a typo in an IRI). For this reason, JPA allows one to generate database schema from an object model [39]. In addition, many JPA implementations support also the converse – generating an object model from database tables. There also exist libraries like Owl2Java [83] which allow generating an object model from an ontology. To this end, three approaches to object model vs ontology schema transformation are discussed in [63] – manual mapping of an object model, its automatic generation from an ontological schema and a hybrid approach, which the authors deem most useful based on its correctness/efficiency ratio. In the hybrid scenario, a basic model is generated automatically and the developer then fine tunes it manually. Conversely, one could start with an object model, generate an ontology schema from it and continue extending the schema further.

Fully satisfied if: The library provides a generator to synchronize the object model with the ontology.

OC1 – explicit inference treatment An RDFS repository may respond to a query with asserted and inferred statements. While they are indistinguishable from the formal point of view [2], this holds only for the retrieval scenario. Statement removal (possibly as part of an update) works differently: asserted triples can be directly deleted but the existence of inferred statements is only implied by the presence of other statements. Consider triples and an application which maps values of rdfs:label. If an OTM library does not differentiate asserted and inferred statements, it will let the application update ex:falcon-1’s label. However, since the application works with rdfs:label and its value is inferred, updating it will actually not remove the asserted skos:prefLabel value. Instead, it will just assert a value of rdfs:label, leading to two labels asserted for the same object:

This is hardly the expected behavior. An OTM library has to deal with such situations, e.g., by marking inferred attributes as read-only or by keeping track of inferred values and indicating an error whenever the application attempts to remove or modify such a value. This issue would be even more important if the repository supported SPIN [43] rules or OWL, which allow richer inference.

Fully satisfied if: The library gives its users explicit control over whether they are working with asserted or inferred statements.

OC2 – named graphs Named graphs [10] are an important feature of RDF, as they allow to split RDF data into meaningful and identifiable parts inside an RDF dataset. Although there exist different semantics for treatment of named graphs, triple stores mostly adopt the one where the dataset’s default graph represents a union or merge of its named graphs [82]. This strategy makes named graphs suitable for logical structuring of data. Consider a company repository containing information about projects, business contracts and employees. It is sensible to let the employees occupy a different named graph than projects, possibly together with a different degree of access control.

Fully satisfied if: The OTM library provides a way to access different graphs (default graph as well as named graphs) of an RDF dataset.

OC3 – automatic provenance management RDF allows to record provenance information about the data using the RDF reification vocabulary [8] (in addition, an OWL 2-compliant PROV-O ontology [48] has been introduced by W3C). This approach, although not without flaws [65], provides an interesting alternative to auditing in JPA, which is not standardized and is done in an ad hoc manner. Quasthoff and Meinel [62] introduce a prototype able to generate provenance data in RDF processing. More specifically, it connects newly created statements to triples upon which they are based. An example of benefits of automatic provenance tracking can be formulated in terms of the benchmark model introduced later in Section 4.2.1 – instead of manually assigning the last editor or author of a report, the OTM could set it based on user session information available in the application. It should be possible to configure for which entities or which attributes the provenance data should be generated and how to retrieve relevant information (e.g., user).

Fully satisfied if: The library provides a way to automatically generate provenance statements about the data it created using a standard vocabulary. The granularity of the provenance data and its selection may be configurable, e.g., at entity or statement level.

MC1 – inheritance mapping Class hierarchies represent a major component of every domain conceptualization. Although RDFS does not support advanced class declaration expressions like disjointness, intersection, or union, which are part of OWL [20], it allows building class hierarchies using the RDFS subClassOf property. When mapping class hierarchies to an object model, the developer may encounter one technical issue: some programming languages, including Java, do not support multiple inheritance on the class level, whereas RDFS does. This is often resolved using interfaces, which do not have this restriction.

Mapping ontological inheritance to an object model may bring subtle conceptual issues. For example, take an ontology containing the following statements:

and a corresponding object model, i.e., entity classes Person,10

Fully satisfied if: The library supports both single-class and multiple-class inheritance mapping.

MC2 – unmapped data access An application’s object model may represent only a part of the domain ontology. Similarly, one may start building an application using a simplified object model and gradually extend its coverage of the underlying ontology. Yet, there exist situations in which access to the portion of the ontology not mapped by the object model may be required. Consider an aviation safety occurrence reporting application (will be discussed in the next section) based on an aviation safety occurrence reporting ontology which extends a generic safety reporting ontology [45]. The generic safety ontology can be used for other high-risk industries such as power engineering, railroad transportation, etc. There may exist an application used, for example, by a national safety board, which allows browsing safety occurrence reports from different industries using an object model based on the generic ontology. Yet, the application could provide access to the industry-specific property values without them being mapped by the object model.

Another example of the benefits of accessing unmapped ontological data is ontology-based classification. In the aviation safety reporting tool, events can be classified using RDFS classes from a predefined vocabulary [64]. However, the application does not need to map these classes in order to use them if the OTM library allows access to these unmapped types.

Fully satisfied if: The library allows to access data which are not mapped by the OTM.

MC3 – RDF collections and containers By default, all RDF properties are plural. In addition, RDF does not impose any kind of ordering restriction on their values, giving them effectively the semantics of a mathematical set. RDF defines the notions of containers and collections, which allow to represent groups of values with different kinds of semantics. For instance, an RDF container rdf:Seq represents an ordered unbounded sequence of elements [8]. On the other hand, an RDF collection rdf:List is a sequence of elements with a known length [8]. Lists (or sequences of data in general) are ubiquitous in programming, so it is important to support mapping of such RDF structures.

Fully satisfied if: The library allows to manage both RDF collections and containers.

To provide a full picture of the object-triple mapping landscape, we complement the (qualitative) feature comparison with a (quantitative) performance comparison. To make the comparison more usable and repeatable, it was decided to design a benchmark framework which could be used to compare as large variety of OTM libraries as possible.

As [33] and [57] point out, a benchmark should aim at simulating real-world conditions as much as possible, especially avoiding microbenchmarks, which are easy to get wrong. Luckily, we had an experience of developing ontology-based safety information solutions for the aviation domain in the INBAS12

4.2.1. Benchmark model

We took an excerpt of the RT model and modified it by removing some of the attributes which use features already present in other parts of the model. It is built upon a conceptualization of the aviation domain called the Aviation Safety Ontology [44] which is based on the Unified Foundational Ontology (UFO) [31]. RDFS serialization of the model ontology is visualized in Fig. 2. The model primarily concerns occurrence reports which document safety occurrences. Each occurrence may consist of several events structured in a part-of hierarchy. Events may be factors of other events, for example, one event may cause or mitigate another event. Reports can reference resources, which are typically file and document attachments. A question-answer tree represents a dynamically generated form structure. Basic provenance data (author, last editor, date of creation and last edit) are stored for each report as well.

OPEN IN VIEWER

Fig. 2.

Benchmark ontology visualization. Rectangles with solid line represent RDFS classes, rectangles with dotted line denote literal datatypes and ellipses are RDF properties used as RDF triple subjects/object. Each edge represents an RDF triple with a source/target node label representing the triple subject/object and the edge label representing the triple predicate. Unlabeled edges ended with a hollow arrow denote triples with the rdfs:subClassOf predicate. prefixes ufo, as, doc, xsd, rdf and rdfs have been introduced in previous listings. Prefix dc denotes the Dublin Core namespace http://purl.org/dc/terms/, prefix foaf denotes the FOAF namespace http://xmlns.com/foaf/0.1/ and rt denotes an application specific namespace http://onto.fel.cvut.cz/ontologies/reporting-tool.

The goal of the benchmark is to exercise both common mapping features known from JPA and features specific to OTM revealed in our experience with building ontology-based applications. Therefore, the model contains both singular and plural relationships, properties with literal values, inheritance, and references to resources not mapped by the object model (the ufo:has_event_type property references an external vocabulary, a doc:logical_record references an unmapped doc:question-doc:answer tree). The last point illustrates how the object model can be connected to a larger domain model which exceeds the application’s area of operation. The corresponding object model is illustrated in Fig. 3.

OPEN IN VIEWER

Fig. 3.

UML class diagram of the benchmark model.

We conclude this part with a remark regarding instance identifiers. The identifier attribute was not included in declarations of the entity classes in the model because various libraries use various types for the identifier, e.g., String, URI, or a vendor-specific class. Moreover, since some of the libraries do not make a mapped object’s identifier (the RDF resource) accessible at all, a key property was added which explicitly specifies a unique identifier of the instance. The notion of a unique key identifying an instance was formalized in the OWL 2 specification [56] in the form of owl:hasKey axioms. In contrast to them, our key has no formal semantics and its uniqueness is purely conventional. It is used to find matching objects when verifying operation results.

ActiveRDF [59] is an object-oriented API for Semantic Web data written in Ruby. It is built around the premise that dynamically-typed languages like Ruby are more suitable for semantic data representation because, like RDF, they do not require objects to conform to a specific class prescription. ActiveRDF represents resources by object proxies, attaching to them methods representing properties. Invocations of these methods translate to read/write queries to the storage. In addition, it supports an object-oriented query syntax based on PathLog [26] and is able to actually generate convenience filter methods automatically.

In fact, ActiveRDF is not a typical OTM library like AliBaba or Empire where RDFS classes are mapped to entity classes. Instead, it treats RDF resources as objects to which it dynamically attaches methods corresponding to their properties. One caveat of the highly dynamic nature of ActiveRDF is pointed out in [72] – it does not offer type correctness and typographical error verification present in libraries based on statically-typed languages such as Java.

5.2. AliBaba

AliBaba [52] is a Java library for developing complex RDF-based applications. It is an evolution of the Elmo [55] library. AliBaba is an extension to the original Sesame API, so it supports only storages accessible via a Sesame/RDF4J connector. Its API is centered around the ObjectConnection class, which allows to persist and retrieve objects mapped by AliBaba’s OTM. It uses dynamically generated proxy objects to track updates. AliBaba also allows to use SPARQL queries to support more complex strategies of mapping values to Java attributes. On the other hand, the entities managed by AliBaba are not able to store IRIs of the resources they represent, which is a severe limitation requiring the developer to maintain a map of objects and identifiers externally.

In addition to OTM, AliBaba contains an HTTP server which can automatically make resources accessible as REST services, providing querying, inserting and deleting capabilities.

5.3. AutoRDF

AutoRDF [15] is a library facilitating handling of RDF data written in C + + . The fact that it is written in C + + allows it to run, for instance, on embedded devices – actually, one of its intended usages is in ID document verification. It is built on top of the Redland RDF library.17

Empire [30] is an RDF-based implementation of the JPA standard [39]. Therefore, its API should be familiar to developers used to working with relational databases in Java. However, since parts of JPA deal with the physical model of the relational database under consideration (e.g., the @Table and @Column annotations), Empire is not fully JPA-compliant. On the other hand, it does support the basic EntityManager API, query API, and entity lifecycle. Unfortunately, Empire’s documentation is limited and it is often unclear which parts of JPA are implemented.

It adds a set of RDF-specific annotations which are used to express the mapping of Java classes and attributes to RDFS classes and properties (see Listing 3). It does not support anonymous resources, so all instances it works with must have an IRI.

5.5. JAOB

The Java Architecture for OWL Binding (JAOB) [78], as its name suggests, is primarily intended for OWL ontologies. However, with minor adjustments, RDF(S) ontologies can be manipulated by it. First, the annotation used for declaring class mapping is called OWLClass, but an RDFS class IRI can be used as its value. Then, in contrast to RDF(S), OWL defines three kinds of properties – object properties, datatype properties and annotation properties [20]. These properties have equivalent annotations in JAOB. To use them with RDF(S), it is necessary to use OWLObjectProperty for attributes whose value is expected to be a reference to another object (RDF resource), OWLDataProperty can be used for other attributes.

JAOB is built on top of OWL API, so it supports only ontologies stored in files. It basically works as (un)marshaller, so it is able to load or save collections of mapped objects. However, there is, for example, no direct way to update an entity in JAOB.

JAOB also contains a generator which allows generating Java classes based on an ontology schema and vice versa.

5.6. JOPA

The Java OWL Persistence API (JOPA) [46,49] is a persistence library primarily designed for accessing OWL ontologies. However, it can be used to work with RDF data as well. Its API is designed so that it resembles the JPA as much as possible, but it does take semantic data specifics like named graphs or inference into account. JOPA tries to bridge the gap between the object-oriented application world, which uses the closed-world assumption (missing knowledge is assumed false), and the ontological world, which is based on the open-world assumption (missing knowledge can be true in some worlds and false in others), by splitting the model into a static and dynamic part. The static part is mapped to the object model and its validity is guarded by integrity constraints with closed-world semantics [75]. The dynamic part is not directly mapped (although it is accessible to a limited extent) and may evolve freely without affecting the object model.

Using JOPA with RDF(S) ontologies requires minor adjustments because JOPA is OWL-based. These adjustments are similar to those described for JAOB.

5.7. KOMMA

The Knowledge Management and Modeling Architecture (KOMMA) [80] is a Java library for building applications based on semantic data. A part of this system is an object-triple mapping module, but the library itself has much richer functionality, including support for building graphical ontology editors based on the Eclipse Modeling Framework.18

RDFBeans [1] is another OTM library built on top of RDF4J. It allows two forms of an object model: 1) the object model may consist of Java classes representing the RDFS classes; 2) the object model may consist of interface declarations forming the mapping, RDFBeans will generate appropriate implementations at runtime using the dynamic proxy mechanism. RDFBeans supports, besides the basic features like transactions and inheritance, also mapping of Java collections to RDF containers, or operation cascading, e.g., when an updated entity is merged into the storage, objects referenced by this entity are merged as well. Instances of entity classes without an explicitly declared identifier attribute are mapped to RDF blank nodes.

5.9. RDFReactor

RDFReactor [77] is yet another Java OTM library. It supports code generation and uses proxy objects to access RDF data directly, instead of storing attribute values in entities. In addition to the mapped properties, RDFReactor entity classes also contain an API for type information manipulation – methods getAll, add and remove allow to retrieve and update types of an instance at runtime. Besides RDFS, RDFReactor also supports selected OWL features like cardinality constraints, inverse properties.

All entity classes mapped by RDFReactor extend the ReactorBase class which acts as a generic representation of an RDF resource, allowing untyped access to all its properties. Subclasses can then explicitly map these properties to provide type-safe access.

5.10. The Semantic Framework

The Semantic Framework (SF) [41] is a Java library for object-oriented access to ontologies. Similar to JAOB or JOPA, the SF is primarily built for OWL, but it can, again, be used with RDFS as well. It internally uses OWL API to process RDF files and its core is an extension of the Jenabean library [17]. The combination of Jena (internally used by Jenabean) and OWL API is, in our opinion, rather odd given the fact that Jena is also able to process RDF files. The mapping is realized using annotations in entity classes.

5.11. Spira

Spira [47] is a library for viewing RDF data as model objects written in Ruby. It allows to access RDF data both as domain objects (entities) and as RDF statements. In particular, one may view and edit statements representing an entity directly from its object representation. On the other hand, it is also possible to create RDF statements directly, without any object representation of the resource. Spira allows to ‘view’ resources as instances of various classes without requiring them to explicitly have the corresponding RDF type. It also allows to map one Ruby class to multiple ontological types, to work both with named and anonymous resources, or to use localized property values, i.e., a single-valued property can have a value in multiple languages.

Spira uses RDF.rb19

SuRF [3] is a Python object RDF mapper. SuRF is built on top of the RDFLib20

SuRF contains a search API which can be used to retrieve instances of the mapped classes and filter them by their attribute values. One subtle feature of SuRF is that it allows to retrieve resources regardless of whether they are actually present in the triple store. To check for their existence, a dedicated is_present method has to be called.

There are several OTM libraries which were omitted from the selection. Elmo [55], Jastor [74], Jenabean [17], Owl2Java [83] or Sommer [73] are obsolete, with their last versions published more than five years ago.

We also wanted to include a purely SPARQL endpoint-based solution which would allow us to compare the performance of a storage agnostic library to libraries exploiting vendor-specific optimizations. However, RAN-Map [76] is not published online (neither sources nor binaries) and TRIOO [24] is buggy and we were not able to adapt it to the benchmark application without significant changes to its code base (for example, it does not insert resource type statements on persist).

Libraries like JOINT-DE [37] (and the original JOINT), Sapphire [72] and the Semantic Object Framework [16] were excluded because their source code is not publicly available and the articles describing them do not provide enough details to evaluate the comparison criteria.

6.1. GC1 – transactions

AutoRDF, JAOB and the Semantic Framework do not support transactions likely due to their lack of support for triple stores in general (more details in Section 6.2). While ActiveRDF, RDFReactor and Spira do support access to regular triple stores and, for example, RDFReactor internally makes use of the store’s transactional mechanism, they do not allow the programmer to control the transactions externally. Empire’s API hints towards support for transactional processing but its implementation is rather strange. As will be discussed in Section 7, operations which insert/update/remove data are automatically committed without any way of preventing this behavior. So, for example, it is not possible to bind updates to multiple entities into one transaction.

AliBaba, KOMMA, and RDFBeans support transactions by relying on the underlying storage’s transaction management, i.e., starting an OTM-level transaction begins a database-level transaction and its end causes the database-level transaction to end as well. SuRF, on the other hand, tracks the changes to objects by marking their updated attributes dirty and allows the programmer to commit the changes manually. JOPA handles transactions by creating clones of the manipulated objects and tracking their changes locally, pushing them into the storage on commit. In addition, the changes are stored in transaction-local models and are used to enhance results of subsequent operations in the same transaction, e.g., when an entity is removed, it will not be in the results of a find all query executed later in the same transaction.

6.2. GC2 – storage access variability

AutoRDF, JAOB and the Semantic Framework support only access to RDF data stored in text files, albeit with various serializations. AliBaba and RDFBeans are tightly bound to the Sesame API, so they can access exclusively Sesame/RDF4J storages.

While KOMMA does not have any other implementation than RDF4J storage access, its internal APIs are designed so that new storage connector implementations can be added.

ActiveRDF supports access to various storages, including Jena, Sesame or a generic SPARQL endpoint. Empire contains storage access modules which can implement the required access API. For instance, it provides a connector to the Stardog database.21

In contrast to JAOB, which does not provide any query API at all, AutoRDF, RDFBeans, the Semantic Framework and Spira do not provide a query API either but contain at least a find all method, which allows to retrieve all instances of the specified type. In case of RDFBeans the situation is peculiar because its RDFBeanManager wraps an instance of RDF4J RepositoryConnection which supports SPARQL query execution, yet, RDFBeanManager does not expose this functionality. SuRF allows to execute arbitrary SPARQL queries, but cannot map their results to entities.

While ActiveRDF does not support mapping SPARQL query results to entities, its advanced query API obviates this issue by supporting almost complete SPARQL grammar [59].

Finally, AliBaba, Empire, JOPA and KOMMA support mapping SPARQL query results to entities by allowing to specify target entity class when retrieving query results.

6.4. GC4 – object-level query language

ActiveRDF contains a variation of a criteria query API containing methods representing query operations like selection, filtering and projection. Additionally, it automatically generates methods for filtering objects by their properties, e.g., it will provide a method Person.find_by_name for a class Person with an attribute name. A similar API exists in SuRF.

None of the other libraries support any object-level query language, so their users have to resort to SPARQL queries, if available.

6.5. GC5 – detached objects

ActiveRDF, AliBaba, AutoRDF, RDFReactor, Spira, and SuRF do not support detached objects because their entities act as proxies which load attribute values from the repository when they are accessed for the first time, or on each access, depending on the internal implementation. Conversely, setting attribute values causes the changes to be written into the storage immediately (except for SuRF which tracks changes locally). Therefore, they have to hold onto a connection to the storage in order to provide basic data access functions. Similarly, although it would appear that KOMMA does support detached objects because it is possible to close a KOMMA IEntityManager and still access attributes of an object read by the closed manager, the contrary is true. As pointed out in [37], the generated proxy objects retain a connection to the underlying storage. Closing an IEntityManager only closes a delegate object.

Empire, JAOB, JOPA, RDFBeans and the Semantic Framework, on the other hand, allow working with the entities completely independently of the persistence context from which they were retrieved because they store the data in the actual objects.

6.6. GC6 – code/ontology generator

Empire, RDFBeans, the Semantic Framework and Spira do not contain any generator capable of creating entity classes from ontology schema or vice versa. ActiveRDF and SuRF do not contain code generators either, but they do not actually use any static object model and all entity classes are generated on demand at runtime.

AliBaba, AutoRDF, JAOB, JOPA and KOMMA contain generators able to build an object model from the ontology. The AliBaba generator supports both RDFS and OWL, but it is less configurable. AutoRDF supports several OWL constructs, but its core is RDFS-based. On the other hand, the generators in JAOB and JOPA are OWL-based and expect the ontology to contain declarations of OWL classes and object, data and annotation properties. KOMMA reuses the AliBaba code generator but the feature is undocumented.

JAOB is the only library with a generator capable of creating an ontology schema based on an object model. This generator is able to create classes and their hierarchies, plus OWL data and object property declarations with domains and ranges and information about whether the property is functional or not [20].

6.7. OC1 – explicit inference treatment

JOPA allows marking entity attributes as possibly containing inferred data. Currently, this makes them effectively read-only. However, in the future, the implementation may permit additive changes. AliBaba, KOMMA and RDFReactor allow marking attributes as read-only, a mechanism which could be used to prevent attempts at modification of inferred statements.

6.8. OC2 – named graphs

Alibaba, AutoRDF, JAOB, RDFBeans, the Semantic Framework, Spira, and SuRF are not aware of RDF named graphs and work only with the default graph.

Empire supports named graphs to a limited extent. Named graph access is specified via the @NamedGraph annotation, which has two modes of operation. The first mode leads to each instance of the annotated class being stored in its own named graph. The second mode then allows to specify an IRI of a named graph into which all instances of the annotated class are saved. JOPA and ActiveRDF, on the other hand, allow one to specify a named graph per instance and for individual attributes. RDFReactor uses model instances which represent individual named graphs of the repository. Each entity is then associated with a single model.

6.9. OC3 – automatic provenance management

None of the evaluated libraries fully supports automatic provenance management. However, AliBaba and KOMMA support the notion of behaviors which could be used to generate provenance data. A behavior adds custom functionality to entities and may be shared by multiple entity classes. One such implementation could be invoked by entity setters, extract user information and insert corresponding provenance statements into the repository. In Spira, Active Model24

Inheritance mapping is probably the most complex feature in the comparison framework, with many subtle issues. Despite this, several evaluated libraries take a straightforward approach which can often lead to unexpected results.

Consider Empire which does support multiple inheritance in that it is able to work with classes which inherit mapped getters/setters from multiple interfaces. However, it is unable to persist more than one type for an entity. So if each interface declares a mapped type, only one of them gets persisted. KOMMA and RDFBeans suffer from the same issue, i.e., they correctly interpret inherited attributes, but always persist only a single type. To illustrate the issue, let us have interfaces Student and Employee, which are mapped to RDFS classes ex:Student and ex:Employee respectively. Then, we declare an entity class WorkingStudent, which implements both Student and Employee. Persisting an object c of type WorkingStudent into the repository would result in either ex:c rdf:type ex:Student or ex:c rdf:type ex:Employee being inserted, depending on the order of declarations in the implements clause of WorkingStudent, where c is mapped to ex:c. Conversely, a straightforward implementation of loading an instance of WorkingStudent would require the corresponding resource to have types ex:Student and ex:Employee, mapping WorkingStudent to an unnamed intersection of the two. However, this is not formalized in any way. A reasonable solution may be to require mapping of concrete entity classes to a single RDFS class only.

The Semantic Framework appears to also support multiple inheritance thanks to its use of Java classes and interfaces. However, the implementation extracts only attributes declared in a particular class, without regard for any fields inherited from its superclass. This in effect means that any values of fields declared in a superclass are ignored when persisting an instance.

JAOB does not support inheritance mapping. SuRF allows to specify multiple superclasses when loading a resource but they do not represent mapped ontological classes, they are regular Python classes adding custom behavior to the instance. On the other hand, similarly to ActiveRDF, since loaded instances contain attributes corresponding to all properties found on the resource, the concept of inheritance mapping does not really apply in this case. Appropriate type assertion presence on a resource can be of importance w.r.t. the type hierarchy, but such an issue is more closely related to the way libraries handle explicit and inferred statements.

JOPA currently supports only class-based inheritance, so mapping classes with multiple supertypes is not possible. However, the semantics of its treatment of inheritance is precisely defined – it requires an explicit type assertion and supports polymorphic instance access, i.e., an inferred supertype may be specified for loading as long as an unambiguous asserted subtype can be found for a resource. Similarly, RDFReactor and Spira support only single type inheritance. On the other hand, Spira allows an entity to specify multiple types, so multiple ontological classes can be combined into a single mapped Ruby class, which can be used as another class’s parent.

AliBaba and AutoRDF are thus the only libraries fully supporting class hierarchy mapping. Since AliBaba is a Java library, it relies on interfaces to support multiple inheritance. AutoRDF is able to exploit the built-in support for multiple inheritance in C + + .

6.11. MC2 – unmapped data access

Providing access to unmapped properties is more challenging in statically typed languages like Java or C + + . AutoRDF, Empire, JAOB, RDFBeans and the Semantic Framework do not support such a feature.

OPEN IN VIEWER

Fig. 4.

UML object diagram of the object model used by the benchmark.

AliBaba, JOPA, KOMMA and RDFReactor, although being Java libraries, do attempt to supply this access. In JOPA, there are two relevant attribute annotations – a @Types attribute allows to read and modify an instance’s ontological types and @Properties denotes a map of property-set of values pairs, representing the properties not mapped by the rest of the entity attributes. AliBaba allows to retrieve the RDFS resource represented by an entity and, since it is an extension of the Sesame API, to access any data related to the resource via regular Sesame API calls. KOMMA contains the IResource interface which represents RDFS resources and has utility methods for manipulating statements related to particular resources. RDFReactor solves the issue by making all entity classes implement the ReactorBase interface (actually, the classes usually extend the ReactorBaseImpl abstract class) whose API contains generic methods for setting and retrieving arbitrary properties attached to the instance. In fact, the mapped attributes make use of this API internally.

ActiveRDF and SuRF, due to their lack of a static model, naturally allow to access all the properties of a resource. In Spira, all model objects can be manipulated also at the RDF statement level thanks to the integration with RDF.rb.

Support for RDF collections and containers is scarce among the evaluated libraries. The only libraries which fully implement this feature are AliBaba, KOMMA and RDFBeans. AliBaba supports RDF lists and RDF containers. Both are mapped to instances of java.util.List. If the list root has the type rdf:List, it is treated correspondingly, otherwise, it is assumed to be a RDF container. This root type management, however, has to be done on RDF statement level, no dedicated API is present for it in AliBaba. KOMMA’s support for RDF lists and containers is virtually the same as in AliBaba. RDFBeans also allows to work with both RDF containers and lists. The type of the target RDF construct is specified via Java annotations.

The approaches of the Semantic Framework and JOPA may be regarded as partial support. The Semantic Framework automatically stores array and java.util.List attributes as RDF sequences, without any way to configure this behavior. JOPA supports the OWL design pattern of linked lists and linked lists with content nodes [21], but it does not support any of the RDF containers. The type of the list is specified via an annotation.

7.1. Experiment setup

The experiments were conducted on a machine with the following setup:

OS: Linux Mint 17.3 64-bit

After publishing the initial performance results, we received optimization suggestions from the authors of KOMMA. They provided a significant performance boost for KOMMA and were incorporated into the benchmark setup. However, these optimizations are undocumented.

Let us now delve into the results of the experiments and attempt to interpret them. First, performance results are detailed, followed by a discussion of scalability of the evaluated libraries. Finally, memory utilization experiments are examined. Complete results can be found in the appendix and also – together with the underlying data – online.28

7.2.1. Performance

In this part, relative performance differences between the evaluated libraries are discussed. As a representative example, results for benchmark with heap size set to 1 GB are visualized in a plot in Fig. 5. Table 4 complements the plot visualization with information on execution time mean, standard deviation, the 95% confidence interval and the application average throughput. Throughput illustrates how many reports was the OTM library able to process per second. A report in this context denotes the object graph shown in Fig. 4. More specifically, this means: OP1, OP2

Each persisted report consists of 11 entities mapped to 68 triples (Person instances are pre-persisted in the set up phase and thus not counted).

OP3, OP4

Each loaded report consists of 13 entities mapped to 90 triples (Person instances are loaded together with the report).

OP5

Each updated report consists of 90 triples loaded for the merge and 9 triples updated.

OP6

Each removed report represents 90 triples loaded and 68 of them removed (Person instances were not removed).

Performance on other heap sizes is discussed in Section 7.2.2, Appendix A.1 contains complete results.

OPEN IN VIEWER

Fig. 5.

Performance of the individual libraries on a 1 GB heap. The plots are grouped by the respective operations. Lower is better.

All the libraries performed relatively consistently, with low deviation and the confidence interval size within 2% of the execution time mean, as suggested by [28]. Only Empire exhibited larger standard deviation for the retrieval operations (OP3 and OP4). KOMMA displayed large standard deviation and many outliers when running with 32 MB heap but this is most likely due to the high stress put on the garbage collector which has to pause the application relatively often. The following paragraphs elaborate on individual operation results.

Create (OP1) and batch create (OP2) The character of OP1 and OP2 suggests that the libraries should perform better in OP2 because they have to execute only a single transaction compared to a linear number of transactions in OP1. The only library that defied this expectation and performed virtually the same in both OP1 and OP2 was Empire. This is due to the fact that Empire actually commits the transaction after each data modifying operation of its EntityManager, more precisely, after merge, persist and remove. Not only is this highly inefficient in the batch scenario, but it can also lead to incorrect behavior because it effectively prohibits a rollback of transactional changes or grouping multiple operations into one transaction. The rather poor performance of Empire is partially due to the fact that it iterates over all attributes of the object being persisted using Java reflection instead of relying on a metamodel (model of entity classes and their attributes) [39] built during application persistence initialization. Reflection is a powerful mechanism which allows a program to examine and modify itself and the environment in which it operates [25] but it represents a performance overhead. While middleware libraries in Java typically use reflection, it is important to be aware of its performance impacts and to avoid them when possible. The other factor in Empire’s create performance is its strategy for checking that an instance does not already exist in the repository. It uses a SPARQL SELECT query with unbound subject, predicate and object, filtering results by the subject being equal to the persisted individual’s identifier. First, our experience with the RDF4J API indicates that executing SPARQL queries is in most cases slower than using the statement filtering API. Second, a simpler ASK query could have been used.

OPEN IN VIEWER

Table 4

Results of the performance benchmark with all libraries running on a 1 GB heap. ( T ¯ ) denotes mean execution time, (σ) standard deviation, CI 95 95% confidence interval, and TPS  represents average throughput per second (details in Section 7.2.1). O stands for the operation executed and L for the evaluated library. For each operation, the fastest library is underlined

The performance of KOMMA depends on whether a data change tracker is enabled. If it is enabled, it attempts to remove old attribute values on change. This leads to significantly worse performance. When it is disabled, KOMMA’s performance is comparable to that of Empire and RDFBeans. The data change tracker relevance for persist is given by the fact that KOMMA does not implement a dedicated persist operation, instead, it allows to register an identifier to a particular type. Persisting attribute values then requires calling setters on the managed instance returned by the registration method. Whenever an attribute is set, the value is inserted into the repository and the object state and KOMMA’s internal attribute cache are refreshed.

RDFBeans performed slightly better than Empire and KOMMA. Two factors of the implementation of its add method are of interest. One is that the method is synchronized. This incurs a slight performance overhead because Java has to manage the synchronization mechanism even for the single-threaded benchmark. And it is somewhat unexpected, given that the documentation of RDF4J specifically declares the RepositoryConnection, around which RDFBeans BeanManager is built, not to be thread-safe and recommends multi-threaded applications to obtain separate connections for each thread.29

²⁹

See https://tinyurl.com/repository-connection, accessed 2018-11-21.

Second, RDFBeans uses the RDF4J API hasStatement method with a bound subject to check for a resource’s existence in the repository. In contrast, JOPA uses the same method, but specifies the whole triple, checking whether the subject is an instance of the specified RDFS class. This is not only arguably faster, but it also allows to persist the same entity as an instance of different RDFS classes.

It can be seen that AliBaba and JOPA have very similar performance in OP1. In the batch scenario (OP2), on the other hand, AliBaba was able to perform a round in under one second, an order of magnitude faster than in OP1. The large difference is given by the transaction commit mechanism, which synchronizes a binary object store with its persistent version, potentially stored on the system disk.

Retrieve (OP3) and retrieve all (OP4) JOPA performed the best in both retrieval scenarios. Looking through the source code, all the libraries use different strategies for loading entity attributes.

AliBaba uses a single SPARQL SELECT query to get the subject and all the properties mapped by the entity class. To avoid failed joins, all the property patterns are wrapped in OPTIONAL. However, OPTIONAL increases the query complexity to PSPACE [67].

Empire adopts a different strategy. A SPARQL CONSTRUCT query is issued, retrieving a graph containing all statements concerning the specified subject. Then, it processes the attributes mapped by the target entity class and filters values for the corresponding properties from the constructed graph.

JOPA does not use SPARQL to retrieve entities for two reasons. One is our experience showing that filtering using the RDF4J API is more efficient. The other, and more important, is that a SPARQL query does not allow to specify whether inferred statements should be included or not and the result does not indicate the origin of the resulting statements (see OC1 in Section 4). Instead, JOPA uses a single call to get all statements concerning the specified subject and then processes these statements, populating the resulting object with appropriate values. While this strategy may turn out inefficient in cases where the object model represents only a tiny portion of the graph around the loaded subject, it does sufficiently well in most cases. Indeed, the results show that it outperforms all the other strategies by a considerable margin.

KOMMA supports two strategies for loading an entity. One relies on a dedicated find method which returns initially an empty object. When a getter is called on this object, KOMMA loads the corresponding attribute value using a SPARQL SELECT query. Since KOMMA has an internal cache of values, it may happen that the value is already cached and no query has to be performed. The other strategy, employed in the benchmark, uses an explicit SPARQL CONSTRUCT query, which fetches all relevant property values in one call and significantly speeds up instance retrieval. However, the query could not be used in OP4, so KOMMA’s performance in this operation is considerably worse than in OP3.

RDFBeans returns a fully initialized object, but it loads its attributes one by one using the RDF4J filtering API. Moreover, similarly to persist, it first checks for the subject’s existence in the repository, which involves a rather costly statement filter with a bound subject. Thus, it was the slowest for both OP3 and OP4.

Update (OP5) OP5 showed comparable performance by AliBaba, JOPA, KOMMA and RDFBeans, with JOPA outperforming the others. Empire performed the worst. The update is also interesting in terms of the internal implementation in the particular libraries.

AliBaba, because it does not support detached objects, requires the user to first load the required entity and then update it manually, essentially corresponding to the behavior Empire and JOPA implement internally. However, this can become extremely cumbersome – for instance, consider an object coming to the REST API of an application in a PUT request, signifying a data update. Then the application would either have to determine the changes by scanning the incoming object and comparing it to an existing object in the database, or use the incoming object as the new state and replace the existing data completely by iterating over all its attributes and merging their values into the repository. The approach of JOPA (see below) is much more flexible because it allows, for example, to specify whether the merge operation should be cascaded to objects referenced by the entity being merged.

Empire internally loads all the data concerning the subject which is being merged, removes them and then inserts the new, updated, data. While efficient, it is hardly the correct behavior. Consider a situation when the application is using a legacy RDF triple store and the object model covers only a part of the data schema. Invoking merge in Empire will not only delete the mapped data but also all the other statements which have the same subject as the updated instance. The automatic commit of changes in Empire, which, given the setup of OP5, did not influence the performance but which can be harmful from the transactional behavior point of view nonetheless, should also be stressed.

JOPA internally loads the instance into which updates will be merged (the same behavior is expected in JPA [39]). It then compares the merged instance with the loaded one and propagates the changes into the repository. The semantics of update in JOPA is supposed to correspond to JPA as much as possible. For instance, JPA specifies that when merging an entity with references to other objects, these references should not be merged and, actually, if they contain changes, these changes should be overwritten with the data loaded from the repository [39]. Of the benchmarked libraries, only JOPA behaves in this way.

Since KOMMA does not support detached objects, updating an object requires loading a corresponding instance from the repository and updating the required attributes in the same manner as with AliBaba. The optimized instance loading query from OP3 implementation was used to retrieve the instance to update.

RDFBeans works similarly to Empire in that it simply removes all statements concerning the updated subject and inserts new statements corresponding to the updated object. This, on the one hand, gives RDFBeans a significant performance boost, on the other hand, it can lead to incorrect behavior. As mentioned when discussing Empire, it can lead to the loss of data not mapped by the object model. RDFBeans performed better because it does not load the data to be removed before the actual removal, it directly issues the delete command.

Delete (OP6) Except for Empire, all the libraries exhibited comparable performance for OP6, with KOMMA winning by a narrow margin.

AliBaba does not have a dedicated remove method, so entity removal in its case is a little complicated. It consists of loading the object to be removed, setting all its attributes to null and then removing the class designation, i.e., the assertion of the instance’s type.

The performance of Empire suffers from the fact that it uses the same strategy for loading statements as in the case of OP3, i.e., a SPARQL CONSTRUCT query. This query retrieves all statements with the specified subject. Empire then removes these statements from the storage. Similar to update, this has the potentially harmful effect of removing statements which are not a part of the object model.

To avoid this issue, JOPA employs the epistemic remove strategy. Epistemic remove deletes only statements asserting properties mapped by the object model. Although less efficient than removing all statements concerning an individual, it is safer in terms of data consistency.

Unfortunately, AliBaba, Empire and JOPA do not deal with the situation where the removed resource is an object of other statements, i.e., it could be referenced by other instances (whether mapped by the object model or not). Relational databases resolve this problem with referential integrity which does not allow removal of a certain record as long as there are references to it in the form of foreign keys. In the open world of ontologies, a reference to an individual completely suffices to prove its existence. However, for an object model, this may not be (and often is not) the case. Therefore, OTM libraries should face this issue. In this regard, KOMMA and RDFBeans come with a solution.

Both KOMMA and RDFBeans perform entity removal by removing all statements whose subject or object is the resource being removed. This sufficiently deals with the issue of referential integrity. On the other hand, this solution can lead to unintended data removal. For example, one may remove an instance without knowing by how many other resources (mapped or not) it is referenced. Another user can then load an object which formerly referenced the removed one and find out that the connection has been severed for reasons unknown. We believe that this issue deserves a more thorough and systematic approach.

7.2.2. Scalability

There are multiple ways of testing the scalability of an application or a library. In the case of this research, it was tested how well the libraries scale with regards to the memory available to the application.

Most of the libraries had issues running on the smallest, 32 MB heap. The only library able to perform the whole benchmark on a 32 MB heap was RDFBeans. AliBaba and JOPA were able to execute OP1, but failed on the other operations. KOMMA managed to execute OP1 and OP2, but with large standard deviation and many outliers. Empire finished the OP6 benchmark, but its standard deviation was so large that the results are all but inconclusive. In addition, Empire ran out of memory also for OP4 on a 64 MB heap. The fact that it did not happen for OP3, which is similar to OP4, is likely due to the individual instances loaded by OP3 being discarded right after the verification phase, whereas for OP4 all the instances are loaded at once and then verified.

Contrary to the expectations, with increasing heap size, the benchmark runtime did not decrease. In fact, in some cases, it tended to increase slightly. This increase could be explained by the garbage collector having to manage a larger amount of memory. Overall, once the heap size passed the threshold after which the benchmark did not run out of memory, there was no definitive trend in terms of application performance with respect to the heap size. This can be seen in an example plot for OP1 in Fig. 6 (all plots regarding scalability w.r.t. heap size can be found in Appendix A.2).

OPEN IN VIEWER

Fig. 6.

Benchmark performance of OP1 w.r.t. heap size. Lower is better.

To summarize, the benchmark developed as part of the OTM comparison framework appears not suitable for measuring scalability of the OTM libraries. In hindsight, this seems logical. Concerning scalability w.r.t. heap size, once the benchmark data fit into the available memory, increasing the heap size brings no performance benefit. Especially since garbage collection is triggered after the setup of each round. A more useful scalability comparison could involve a variable number of concurrent clients. However, this is not supported by the presented benchmark. On the other hand, a concurrent access-based benchmark would have to be carefully designed in terms of data correctness and objectivity.

8.1. Feature comparison conclusions

The evaluated libraries may be split into several groups based on different points of view. One contains solutions which appear to be proof-of-concept prototypes with limited functionality. This group contains AutoRDF, JAOB, and the Semantic Framework – their usability is restricted to a rather small set of specific cases. Another group consists of libraries in which entities are merely proxies of the underlying RDF data. Such libraries could be problematic especially in larger (web) applications because they generally do not support detached objects. On the other hand, thanks to the fact that they use dynamically generated proxies, they are able to support a wide variety of shared behavior in entities. In case of ActiveRDF and SuRF, this is enhanced by the fact that they do not require a predefined object model. The last group may be denoted as general purpose persistence libraries and it contains Empire, JOPA and RDFBeans. These libraries attempt to provide functionality similar to JPA but in the world of ontologies.

In general, most of the libraries suffer from the fact that there is no formalization of the object-triple mapping. Such a gap leads to potentially inconsistent behavior and straightforward solutions which fail to handle less frequent cases. To give a final recommendation, for multiuser applications with non-trivial business logic, we believe that support for transactions (GC1), detached objects (GC2), query result mapping (GC3) and inheritance mapping (MC1) are a must have for an OTM library. Explicit treatment of inferred knowledge (OC1) would be important in scenarios where the mapped data are involved in inference.

8.2. Performance comparison conclusions

Several conclusions may be derived from the performance comparison results. AliBaba demonstrated very good performance and supports a rich set of features. However, while restriction to Sesame-based storage is not a big problem thanks to its wide support among triple store implementors, the unintuitive cascading strategy and lack of support for detached objects can be of concern. Empire exhibited worse than average time performance and a potential memory leak in query mapping. On the other hand, its API directly implements portions of the JPA standard [39] which can make it suitable for existing projects migrating from relational to semantic databases. JOPA provides rich features as well as excellent performance, memory utilization, and sound data access patterns for the general case. The performance of KOMMA critically depends on the use of optimization hints provided by its authors. Without them, KOMMA performed in most cases significantly worse than the other libraries. With them, its performance rivals the performance of AliBaba and JOPA. However, its API made it more difficult to use in the benchmark than the other libraries. RDFBeans is the most suitable library for environments with extremely limited memory. In comparison to the other four libraries evaluated in the performance benchmark, its time performance is, apart from retrieval operations, average.

OPEN IN VIEWER

Fig. 7.

Performance of the individual libraries on a 32 MB heap. The plots are grouped by the respective operations. Lower is better.

Overall, while the slowest player in an ontology-based application stack will always be the triple store (given the size and complexity of data in processes), the benchmark showed that OTM libraries significantly differ in terms of performance. These differences are given by the effectiveness with which OTM solutions access the underlying storage. Application developers thus may choose a library suitable for their case, e.g., RDFBeans for applications with limited resources, AliBaba for batch processing scenarios, JOPA for general cases. However, we argue that more important than performance differences are the differences in the semantics of the implementation, where, for example, Empire and RDFBeans can have unexpected side-effects in scenarios where the mapping does not cover the whole RDF data schema.

OPEN IN VIEWER

Fig. 8.

Performance of the individual libraries on a 64 MB heap. The plots are grouped by the respective operations. Lower is better.

OPEN IN VIEWER

Fig. 9.

Performance of the individual libraries on a 128 MB heap. The plots are grouped by the respective operations. Lower is better.

OPEN IN VIEWER

Fig. 10.

Performance of the individual libraries on a 256 MB heap. The plots are grouped by the respective operations. Lower is better.

OPEN IN VIEWER

Fig. 11.

Performance of the individual libraries on a 512 MB heap. The plots are grouped by the respective operations. Lower is better.

OPEN IN VIEWER

Fig. 12.

Performance of the individual libraries on a 1 GB heap. The plots are grouped by the respective operations. Lower is better.

OPEN IN VIEWER

Fig. 13.

Scalability w.r.t. heap size. Lower is better.

Regarding the libraries’ memory scalability and efficiency, the benchmark indicates that heap size is of little concern. Most real-world applications nowadays use vastly more than 32 MB of heap. None of the evaluated solutions showed any memory-related performance issues after crossing this threshold (with the exception of a possible memory leak in Empire). However, as was stated earlier, a more realistic concurrent access-based scenario running for a longer period of time could be more revealing.

The performance benchmark was designed so that adding new libraries into the comparison would be relatively straightforward. However, attention must be paid to providing equal conditions to all compared libraries. For this reason, the evaluation provided in the paper concentrated on a single triple store implementation. The performance benchmark is published online and so are the results of the comparison presented in this work. We believe that these results might help potential users of OTM libraries in deciding which one to choose. In addition, the benchmark can be considered an example application for each of the libraries, complete with an object model declaration and the persistence layer setup and usage. Given the scarce documentation of most of the evaluated libraries, this can be a welcome benefit.

In the future, the benchmark should be extended to allow evaluating scalability w.r.t. multiple concurrent users, providing an even more realistic test case for the OTM libraries.

OPEN IN VIEWER

Table 6

Results of the performance benchmark with all libraries running on a 32 MB heap. ( T ¯ ) denotes mean execution time, (σ) standard deviation, CI 95 95% confidence interval, and TPS represents average throughput per second (details in Section 7.2.1). O stands for the operation executed and L for the evaluated library. For each operation, the fastest library is underlined

OPEN IN VIEWER

Table 7

Results of the performance benchmark with all libraries running on a 64 MB heap. ( T ¯ ) denotes mean execution time, (σ) standard deviation, CI 95 95% confidence interval, and TPS represents average throughput per second (details in Section 7.2.1). O stands for the operation executed and L for the evaluated library. For each operation, the fastest library is underlined

OPEN IN VIEWER

Table 8

Results of the performance benchmark with all libraries running on a 128 MB heap. ( T ¯ ) denotes mean execution time, (σ) standard deviation, CI 95 95% confidence interval, and TPS represents average throughput per second (details in Section 7.2.1). O stands for the operation executed and L for the evaluated library. For each operation, the fastest library is underlined

OPEN IN VIEWER

Table 9

Results of the performance benchmark with all libraries running on a 256 MB heap. ( T ¯ ) denotes mean execution time, (σ) standard deviation, CI 95 95% confidence interval, and TPS represents average throughput per second (details in Section 7.2.1). O stands for the operation executed and L for the evaluated library. For each operation, the fastest library is underlined

OPEN IN VIEWER

Table 10

Results of the performance benchmark with all libraries running on a 512 MB heap. ( T ¯ ) denotes mean execution time, (σ) standard deviation, CI 95 95% confidence interval, and TPS represents average throughput per second (details in Section 7.2.1). O stands for the operation executed and L for the evaluated library. For each operation, the fastest library is underlined

OPEN IN VIEWER

Table 11

Results of the performance benchmark with all libraries running on a 1 GB heap. ( T ¯ ) denotes mean execution time, (σ) standard deviation, CI 95 95% confidence interval, and TPS represents average throughput per second (details in Section 7.2.1). O stands for the operation executed and L for the evaluated library. For each operation, the fastest library is underlined