Extending ontology pitfalls for better ontology evaluation

Abstract

This article presents a framework that detects potential ontology building errors to improve the ontology quality. These potential errors are called ontology pitfalls in the literature. This work extends the existing ontology pitfall set in the literature and suggests new solutions for ontology repair. We have also developed a Java implementation for detection of the proposed pitfalls. These pitfalls were evaluated with well-known ontologies using this implementation.

Keywords

Ontology evaluation ontology pitfalls ontology quality OOPS!

1. Introduction

Ontologies have become a popular tool for knowledge sharing and reuse. With the popularity of ontologies, measuring the quality of ontology has also become an important research area. Ontology evaluation is one of the key stages in ontology development. Ontology must be correct and present the correct results to the user. The accuracy of the results cannot be guaranteed unless the quality of the ontology is guaranteed. Ontology quality evaluation approaches can be classified into two as follows [1]:

Syntactic quality evaluation approaches: evaluates structural aspects of an ontology based on ontology development guidelines, common pitfalls, structural metrics and so on. Examples of this approach include [2 –10].

Semantic quality evaluation approaches: evaluates the semantic validity of concepts and relationships in an ontology based on the approval of field experts [11]. Examples of this approach include [12 –15].

In this article, ontology evaluation is carried out using the syntactic quality evaluation approach. The first of the studies using the syntactic quality approach is OntoClean [2]. OntoClean is a methodology that evaluates the accuracy and adequacy of taxonomic relationships in ontology. It uses concepts such as essence, identity and unity, which have their origins in philosophy, to evaluate the correctness of taxonomic relationships.

OntoQA (metric-based ontology quality analysis) [3] is another method using the syntactic quality evaluation approach. It evaluates the given ontology on the basis of schema metrics and instance metrics. OntoMetrics [10] is an on-line platform for ontology metric calculation. OntoMetrics uses the metrics proposed in OntoQA [3].

Gangemi et al. [4] propose structural, functional and usability–related measures for ontology evaluation and validation. Burton-Jones et al. [5] propose an ontology evaluation model based on semiotic theory. Semiotic-based Ontology Evaluation Tool (S-OntoEval) [6] aims at assessing the quality of the ontology by taking several metrics into consideration for assessing the syntactic, semantic and pragmatic aspects of ontology quality. AktiveRank [7] is a system based on a set of metrics that measures in which extent the concepts in the selected domain are represented by the ontology. Users are assumed to be using an ontology search engine like Swoogle [16]. With the search engine ActiveRank uses, concepts that match the user’s query are found. ActiveRank applies metrics that calculate how well the query results and their associated neighbouring concepts are represented in the ontology. DoORS (Domain Ontology Ranking System) [9] is another ontology evaluation tool that uses metrics drawn from semiotic theory.

OOPS! (OntOlogy Pitfall Scanner!) [8] is a live and online catalogue of common pitfalls in ontology development. It introduces 40 pitfalls, which are extracted from the literature and the manual analysis of ontologies. Currently, OOPS! has been used over 2000 times by users of over 50 countries [17]. This article extends the OOPS! ontology pitfall catalogue, which is one of the most common and accessible ontology evaluation tools. The new extension set contains seven new pitfalls.

Section 2 introduces the pitfalls in OOPS! catalogue. Section 3 introduces the new pitfall set. This chapter includes the pitfall definitions, examples and solutions. Section 4 describes the implementation of the testing framework. Section 5 presents the results of the test and evaluation of the test results. Finally, the concluding section 6 reviews the potential future work.

2. OOPS! Ontology pitfall catalogue

OOPS! [8] is a live and online catalogue of common pitfalls in ontology development. It introduces pitfalls, including a compendium of pitfalls extracted from the literature review and from the manual analysis of ontologies. For each pitfall, OOPS! also incorporates its value of importance level (critical, important and minor) assigned by ontology experts.

The current version of the catalogue consists of a list of 40 pitfalls as well as their descriptions (Table 1). OOPS! is also supported by a web-based tool, independent of any ontology development environment, for detecting potential pitfalls that could lead to modelling errors in OWL (Web Ontology Language) ontologies. OOPS! can identify semi-automatically the pitfalls whose codes are shown in black in Table 1. However, OOPS! is not able to detect the pitfalls whose codes are shown in grey yet.

Table 1.

OOPS! ontology pitfall catalogue.

Critical (1)
P01.	Creating polysemous elements
P03.	Creating the relationship is instead of using rdfs:subClassOf, rdf:type or owl:sameAs
P05.	Defining wrong inverse relationships
P06.	Including cycles in the hierarchy
P14.	Misusing owl:allValuesFrom
P15.	Misusing not some and some not
P16.	Misusing primitive and defined classes
P19.	Swapping intersection and union
P27.	Defining wrong equivalent relationships
P28.	Defining wrong symmetric relationships
P29.	Defining wrong transitive relationships
P31.	Defining wrong equivalent classes
P37.	Ontology not available
P39.	Ambiguous namespace
P40.	Namespace hijacking
Important (2)
P10.	Missing disjointness
P11.	Missing domain or range in properties
P12.	Missing equivalent properties
P17.	Specialising a hierarchy exceedingly
P18.	Specifying the domain or range exceedingly
P23.	Using incorrectly ontology elements
P24.	Using recursive definition
P25.	Defining a relationship inverse to itself
P26.	Defining inverse relationships for a symmetric one
P30.	Missing equivalent classes
P34.	Untyped class
P35.	Untyped property
P38.	No OWL ontology declaration
Minor (3)
P02.	Creating synonyms as classes
P04.	Creating unconnected ontology elements
P07.	Merging different concepts in the same class
P08.	Missing annotations
P09.	Missing basic information
P13.	Missing inverse relationships
P20.	Misusing ontology annotations
P21.	Using miscellaneous class
P22.	Using different naming criteria in the ontology
P32.	Several classes with the same label
P33.	Creating a property chain with just one property
P36.	URI contains file extension

OWL: Web Ontology Language; URI: Uniform Resource Identifier.

3. New pitfall set

This work introduces seven new pitfalls (Table 2) as an extension to OOPS! pitfall catalogue (Table 1). The PhD thesis of Villalón [18] is the original work that proposes the OOPS! Ontology pitfall catalogue. Villalón [18] proposes three importance levels and explains them as follows:

Critical (1): it is crucial to correct the pitfall. Otherwise, it could affect the ontology consistency, reasoning and applicability, among other characteristics.

Important (2): although not critical for ontology function, it is important to correct this type of pitfall.

Minor (3): it does not represent a problem. However, correcting it makes the ontology in better form and understandable.

Table 2.

Extended pitfall list.

Important (2)

P x_{2}

. Properties with similar names but missing structural relationships

P x_{3}

. Classes with similar names but missing structural relationships

P x_{4}

. Super-class with direct instances

P x_{5}

. Data property with more than one value

P x_{6}

. Empty class defined as the range of an object property

P x_{7}

. Insufficient external links with other ontologies

Minor (3)

P x_{1}

. Class definition with inadequate detail

These levels do not have clear boundaries in the sense that a particular pitfall in a level could be debatable depending on: (1) modelling decisions, (2) ontology requirements and (3) context of use by an ontology application. After determining this framework, ontology developers with different levels of expertise are asked to determine the importance level of the proposed ontology pitfalls. The importance level can be ‘critical’ (3 points), ‘important’ (2 points), ‘minor’ (1 point) or ‘not important’ (0). Then, the importance level of the ontology pitfall is determined according to the weighted average of the answers given. Weighting depends on the expertise level of the ontology developer (‘expert’ 3, ‘medium confidence’ 2 and ‘low confidence’ 1).

In this study, the same method was applied to 20 participants of varying levels (10/expert, 8/medium confidence, 2/low confidence). When the results are examined, the importance level of $P x_{1}$ is determined as ‘minor’ and the importance levels of the remaining pitfalls are determined as ‘important’.

The remaining of this section gives the definitions and examples of the newly proposed pitfalls:

1. Class definition with inadequate detail $(P x_{1})$ : if a class has no direct properties and no defined relationship with other classes other than ‘rdfs:subclassOf’ or ‘owl:equivalentClass’, then it means this class is not modelled in adequate detail.

Example: the definition of ‘RedWine’ class contains only the ‘rdfs:subClassOf’ relationship that binds this class to its super-classes:

<owl:Class rdf:ID = ‘RedWine’>

< rdfs:subClassOf rdf:resource = ‘#Wine’/>

</owl:Class>

Solution: extend the class definition with additional properties or relationships with other classes other than ‘rdfs:subclassOf’ or ‘owl:equivalentClass’. Alternatively, consider to remove the class definition.

2. Properties with similar names but missing structural relationships $(P x_{2})$ : if two properties have similar names but they are not related with ‘rdfs:subPropertyOf’, ‘owl:equivalentProperty’ or ‘owl:inverseOf’ constructs, in this case, it could be one of the following three issues:

same property has redundant definitions,

relationship between these two properties is missing,

there are unnecessary repetitions in the naming convention.

An example of each situation is presented as follows:

Example 1: property ‘front_axle’ of the ‘Car’ defined twice in the ontology: ‘front_axle_cm’ and ‘front_axle_inches’. Only one property definition remains in the ontology, and the unit specification should be removed from the property name:

<owl:DatatypeProperty rdf:ID = ‘front_axle_cm’>

<rdfs:domain rdf:resource = ‘#Car’/>

<rdfs:range rdf:resource = ‘&xsd; positiveInteger’/>

<owl:DatatypeProperty>

<owl:DatatypeProperty rdf:ID = ‘front_axle_inches’>

<rdfs:domain rdf:resource = ‘#Car’/>

<rdfs:range rdf:resource = ‘&xsd; positiveInteger’/>

<owl:DatatypeProperty>

Solution: remove one of the properties and delete the unit name from the remaining property name. If it is necessary, unit conversion should be handled in the application level rather than the data layer.

Example 2: there are two properties in the ontology ‘dimension’ and ‘dimension_width’, but they are not structurally related. Therefore, the missing structural relationship between them should be added to the ontology:

<owl:DatatypeProperty rdf:ID = ‘dimension’>

<rdfs:domain rdf:resource = ‘#Car’/>

<rdfs:range rdf:resource = ‘&xsd; positiveInteger’/>

<owl:DatatypeProperty>

<owl:DatatypeProperty rdf:ID = ‘dimension_width’>

<rdfs:domain rdf:resource = ‘#Car’/>

<rdfs:range rdf:resource = ‘&xsd; positiveInteger’/>

<owl:DatatypeProperty>

Solution:‘dimension_width’ property should be defined as ‘rdfs:subPropertyOf’ of ‘dimension’ property.

Example 3: there are a series of properties that are not structurally related to each other, but all of them have the expression ‘paper_’ in their names:

<owl:DatatypeProperty rdf:ID = ‘paper_weight’/>

<owl:DatatypeProperty rdf:ID = ‘paper_dimensions’/>

<owl:DatatypeProperty rdf:ID = ‘paper_color’/>

<owl:DatatypeProperty rdf:ID = ‘paper_quality’/>

Solution: create a new property having a name similar to ‘paper_properties’ and defined as the super property of all the properties above. Then remove the repeating ‘paper_’ expression in the names of these five properties.

3. Classes with similar names but missing structural relationships $(P x_{3})$ : if two classes have similar names but they are not related with ‘rdfs:subClassOf’, ‘owl:equivalentClass’ or ‘owl:disjointWith’ constructs, then these concepts are probably structurally related. Therefore, the missing structural relationship between them should be added to the ontology.

Example: the example ontology below has two classes: ‘Mammals’ and ‘MarineMammals’. The name of the ‘Mammals’ class is subsumed by the class ‘MarineMammals’, so probably ‘MarineMammals’ class is subclass of ‘Mammals’ class. However, these two classes are not hierarchically related:

<owl:Class rdf:ID = ‘Mammals’/>

<owl:Class rdf:ID = ‘MarineMammals’/>

Solution: define a ‘rdfs:subClassOf’ relation between two concepts. The class whose name is subsumed by the other class should be defined as the super-class:

<owl:Class rdf:ID = ‘Mammals’/>

<owl:Class rdf:ID = ‘MarineMammals’>

<rdfs:subClassOf rdf:resource = ‘#Mammals’/>

</owl:Class>

4. Super-class with direct instances $(P x_{4})$ : if a class has subclasses, but it has also its direct individuals (other than direct individuals of subclasses/subproperties indirectly linked to the class/property by inference), then these individuals probably belong to an undefined subclass of the class. Another possibility is that this individual belongs to an existing subclass, but it is incorrectly linked directly to the super-class.

Example: in the example below, we have two subclasses of Wine class: ‘WhiteWine’ and ‘RedWine’. We have an individual named ‘VasseFelixFiliusChardonnay’ that belongs to ‘WhiteWine’ class. We have another individual named ‘RadiusCabernet’ that belongs to ‘RedWine’ class. The last individual named ‘VieViteRose2018’ directly belongs to ‘Wine’ class:

<owl:Class rdf:ID = ‘Wine’/>

<owl:Class rdf:ID = ‘WhiteWine’>

<rdfs:subClassOf rdf:resource = ‘#Wine’/>

</owl:Class>

<owl:Class rdf:ID = ‘RedWine’>

<rdfs:subClassOf rdf:resource = ‘#Wine’/>

</owl:Class>

Solution: create a new subclass of the class (in the example, ‘Wine’ class) and move the class individuals to the new subclass. If this cannot be done, check if this instance belongs to an already existing subclass (in the example, ‘WhiteWine’ or ‘RedWine’).

5. Data property with more than one value $(P x_{5})$ : if a data property has more than one value, then these values probably correspond to the same value expressed in different units of measurement.

Example: in the example below, we have an instance of ‘Lion’ class named ‘Alex’. The weight of this lion has two different numeric values, but these values actually correspond to the same value expressed in different units of measure (first one is ‘kg’ and the second one is ‘lbs’):

</Lion>

Solution: remove one of the redundant values and express the value using one and only one unit of measurement.

6. Empty class defined as the range of an object property $(P x_{6})$ : if a class is defined as the range of an object property, but it has no individuals, then this class should be populated properly.

Example: in the example below, the range of the ‘hasColor’ property is defined as ‘Color’ class. However, ‘Color’ class has no individuals:

<owl:DatatypeProperty rdf:ID = ‘hasColor’>

<rdfs:domain rdf:resource = ‘#Car’/>

<rdfs:range rdf:resource = ‘#Color’/>

<owl:DatatypeProperty>

Solution: populate the ‘Color’ class properly.

7. Insufficient external links with other ontologies $(P x_{7})$ : in order to achieve semantic interoperability between different ontologies, it is important to establish semantic relationships and links between entities from different ontologies [19]. Therefore, a good ontology example should have sufficient external links to other ontologies. $P x_{7}$ tests whether the ratio of links from the given ontology to other ontologies is below a certain threshold value. Let the ontology classes that are not linked with a concept from an external ontology be ‘unlinked concepts’; then if the ratio of unlinked classes to all classes in the ontology is over 90%, $P x_{7}$ warning is received. 90% is the threshold value accepted in this study; this value may vary for different ontology applications in different domains.

Solution: create more external links to other domain ontologies.

4. Implementation of the testing framework

The proposed pitfalls are implemented in Java programming language using the Jena ontology library [20]. The developed application has been tested on 10 widely known ontologies listed as follows:

Pizza [21]: a well-known ontology in the semantic web community. It is developed for educational purposes by the University of Manchester.

MarineTLO [22]: is a top-level ontology about marine species, and it assists ongoing research on biodiversity.

GoodRelations [23]: is a lightweight ontology for exchanging e-commerce information, namely, data about products, offers, points of sale, prices, terms and conditions.

PROV (Provenance Ontology (or PROV-O)) [24]: provides a set of classes, properties and restrictions that can be used to represent and interchange provenance information generated in different systems and under different contexts.

Wine [25]: is an ontology for wines and the wine industry. It includes such concepts as vintages, wine regions, wineries, grape varieties and so on.

VIVO [26]: provides a set of types (classes) and relationships (properties) to represent researchers and the full context in which they work.

Symp (Symptom Ontology) [27]: is an ontology including disease symptoms encompassing perceived changes in function, sensations or appearance reported by a patient indicative of a disease.

DRPSNPTO (Dementia-Related Psychotic Symptoms Non-Pharmacological Treatment Ontology) [28]: represents the domain knowledge specific to non-pharmacological treatment of psychotic symptoms in dementia in the long-term care setting.

OM (Ontology of units of Measure) [29]: provides classes, instances and properties that represent the different concepts used for defining and using measures and units.

SOSA (Sensor, Observation, Sample and Actuator Ontology) [30]: a lightweight vocabulary to describe sensors, actuators, and samplers and the acts they can perform.

Statistics of the selected ontologies are presented in Table 3. The table shows the numbers of classes, object properties, data properties, class individuals and total axioms in the selected ontologies. Datatype properties relate individuals to literal data (e.g. strings, numbers, date times, etc.), whereas object properties relate individuals to other individuals. Axioms are the statements that are asserted to be true in the domain being described. In other words, the total axiom number indicates the number of all statements expressed in the ontology.

Table 3.

Statistics of the selected ontologies.

	Class	Object property	Data property	Individual	Axiom
Pizza	101	8	0	5	916
MarineTLO	106	93	2	3	1218
GoodRelations	38	53	49	46	1141
PROV	31	44	6	4	973
Wine	138	16	1	206	1046
VIVO	418	199	219	0	4862
Symp	944	1	0	0	6057
DRPSNPTO	610	60	0	0	3964
OM	809	23	11	2219	23,641
SOSA	16	21	2	25	357

PROV: provenance ontology; DRPSNPTO: dementia-related psychotic symptoms non-pharmacological treatment ontology; OM: ontology of units of measure; SOSA: sensor, observation, sample and actuator ontology.

The proposed pitfall set was implemented using the queries below. The queries are defined in Semantic Web Rule Language (SWRL) [31] and its built-ins (swrlb):

$P x_{1}$ : for each class $C$ in the ontology, count the number of $? p$ values returned by $Q_{1}$ , if the result is 0, then return $P x_{1}$

\begin{matrix} Q_{1} : \\ (C ? p ? c) \land [(? p swrlb : equal owl : disjointWith) \lor (? p swrlb : equal owl : complementOf) \lor \\ (? p swrlb : equal rdfs : first) \lor (? p swrlb : equal owl : members) \lor \\ (? p swrlb : equal rdfs : domain) \lor (? p swrlb : equal rdfs : range)] \end{matrix}

$P x_{2}$ : count the number of $? p$ values returned by $Q_{1}$ , if the result is 0 and the names of $P_{1}$ and $P_{2}$ are syntactically similar, then return $P x_{2}$

\begin{matrix} Q_{1} : \\ (P_{1} ? p P_{2}) \land [(? p swrlb : equal rdfs : subPropertyO f) \lor \\ (? p swrlb : equal owl : equivalentProperty) \lor (? p swrlb : equal owl : inverseO f)] \end{matrix}

$P x_{3}$ : count the number of $? p$ values returned by $Q_{1}$ , if the result is 0 and the names of $C_{1}$ and $C_{2}$ are syntactically similar, then return $P x_{3}$

\begin{matrix} Q_{1} : \\ (C_{1} ? p C_{2}) \land [(? p swrlb : equal rdfs : subClassO f) \lor \\ (? p swrlb : equal owl : equivalentClass) \lor (? p swrlb : equal owl : disjointWith)] \end{matrix}

$P x_{4}$ : for each class $C$ in the ontology, count the number of $? c$ values returned by $Q_{1}$ , and count the number of $? i$ values returned by $Q_{2}$ , if both results are greater than 0, then return $P x_{4}$

\begin{matrix} Q_{1} : \\ (? c rdfs : subClassO f C) \\ Q_{2} : \\ (? i rdf : type C) \end{matrix}

$P x_{5}$ : for each data property $P$ in the ontology, let $L$ be the list of distinct $? i$ values returned by $Q_{1}$ ; then for each $I$ in $L$ if the result of $Q_{2}$ is greater than 1 return $P x_{5}$

\begin{matrix} Q_{1} : \\ (? i P ? υ) \land (P rdf : type owl : DatatypeProperty) \\ Q_{2} : \\ (I P ? υ) \end{matrix}

$P x_{6}$ : count the number of $? i$ values returned by $Q_{1}$ , if the result is zero, then return $P x_{6}$

\begin{matrix} Q_{1} : \\ (? p rdfs : range C) \land (? i rdf : type C) \end{matrix}

$P x_{7}$ : if the namespaces of $? c_{1}$ and $? c_{2}$ value pairs returned by $Q_{1}$ are different, increment the linked class count by 1. If the linked class count is below 10% of the total class count in the ontology, then return $P x_{7}$

\begin{matrix} Q_{1} : \\ (? c_{1} rdf : type owl : Class) \land (? c_{2} rdf : type owl : Class) \land (? c_{1} ? p ? c_{2}) \end{matrix}

5. Evaluation

Table 4 shows the results of the test performed.

Table 4.

The incidence of pitfalls in selected ontologies.

	$P x_{1}$	$P x_{2}$	$P x_{3}$	$P x_{4}$	$P x_{5}$	$P x_{6}$	$P x_{7}$
Pizza	1	2	3	–	–	4	–
MarineTLO	1	1	1	–	–	59	+
V1	–	9	–	4	–	37	–
PROV	–	7	1	1	–	29	–
Wine	4	–	43	12	–	1	–
VIVO	78	42	55	–	–	125	–
Symp	944	–	–	–	–	–	+
DRPSNPTO	–	–	44	–	–	60	–
OM	–	7	1	1	–	29	–
SOSA	3	–	1	–	25	–	+

$P x_{1}$ was detected in most ontologies. The highest incidence of $P x_{1}$ is in symptom (symp) ontology (100%). This is because the symptom ontology is designed as a taxonomy that only models the hierarchy between classes. The fact that ‘no other relation between classes is defined’ is understood from the fact that the number of object properties in the ontology is 0. In Figure 1, knowledge organisation systems are listed according to their semantic expressivity. As semantic expressivity decreases, the incidence of $P x_{1}$ increases, because in this case, relations between classes are not defined in the knowledge organisation system, except for hierarchical relations.

Figure 1.

Knowledge organisation systems according to their semantic expressivity.

$P x_{2}$ was mostly found in the VIVO ontology (10%). One reason for this is that the rate of property definitions in this ontology is higher than other ontologies. A second reason is that there are probably unnecessary repetitions in the property names. For example, Figure 2 shows a series of properties that are not structurally related to each other from VIVO ontology. All of these properties have the expression agricultural area in their names. It would be an example of a more accurate naming approach to remove this expression from these property names and defining all properties as subproperties of a new property named with a name similar to agricultural area properties.

Figure 2.

Example VIVO properties that are not structurally related to each other.

$P x_{3}$ was mostly detected in Wine ontology (about 31%). Because Wine ontology imports Food ontology and many classes in Food ontology have similar names but are not directly related. For example, ‘DessertWine’ class does not have a structural relationship to the ‘Dessert’ class, although they have very similar names. In this case, the warnings about $P x_{3}$ pitfall should be ignored. However, there are many warnings about $P x_{3}$ in the dementia-related psychotic symptoms non-pharmacological treatment ontology (DRPSNPTO) ontology. This ontology has some classes with similar names but having missing structural relationships. Since the structural relations between these classes are not modelled in the ontology (e.g. ‘Brother’ and ‘Brother-in-law’), it would be appropriate to consider the $P x_{3}$ warnings.

$P x_{4}$ was mostly found in Wine ontology (9%). The reason why this pitfall is less common than the others is that this pitfall requires an instantiated ontology. Since the ontologies used and other ontology examples in the literature are generally domain ontologies, they model the ontology schema part and do not contain instance knowledge. Since there are a few class individuals in the Wine ontology, $P x_{4}$ can be found in this ontology. The fact that 9% pitfall detection has been made even in an ontology containing so a few individuals have led to the idea that $P x_{4}$ will be encountered frequently in crowded ontologies.

As we mentioned above, since the domain ontologies do not contain instance knowledge, we populated the Sensor, Observation, Sample and Actuator (SOSA) ontology manually to exemplify $P x_{5}$ . The reason for choosing SOSA ontology is that we detected an example case for $P x_{5}$ in the SOSA ontology schema. As can be seen from the class individual shown below, the value of the datatype attribute ‘hasSimpleResult’ in the SOSA ontology can be defined more than once using different units of measurements. However, the value must be defined only once using a single unit of measurement. Since we have defined 25 class individuals, $P x_{5}$ had been detected 25 times. As the ontology examples increase, the number of $P x_{5}$ will also increase:

<owl:NamedIndividual rdf:about = ‘http://www.w3.org/ns/sosa/observation1’/>

<hasFeatureOfInterestrdf:resource = ‘http://www.w3.org/ns/sosa/HurricaneMariaAPSample’/>

</owl:NamedIndividual>

$P x_{6}$ is the most common pitfall within the scope of the test. This is because ontologies are not populated with instance knowledge and they contain only schema information. The ranges of the properties were modelled in the schema, but instances of these range classes were not defined in the ontology. The rate of this pitfall will decrease as ontologies are populated with instance knowledge.

$P x_{7}$ has been found in only two ontologies in the test. This is because the test ontologies we chose are good examples of ontologies. Since the value of information is very much a function of what it links to and it is considered generally rather bad etiquette not to link to related external material, the selected good ontology examples have also created as many external links as possible with other ontologies. Only two ontology examples were found where the external links were below the threshold we determined. It is clear that if this test is performed on randomly selected ontologies instead of good ontology examples, the number of $P x_{7}$ will increase.

6. Conclusion and future work

This study proposes a new pitfall set to extend the OOPS! ontology evaluation catalogue, which is one of the most common and accessible ontology evaluation tools. A Java application using the Jena library has been developed to test the proposed pitfall set. The proposed pitfalls were tested on some well-known ontologies, and the results were evaluated. A possible future work would be to further extend the new pitfall set. Rather than fundamental or syntactical errors, further extensions in pitfalls will focus on the quality of external links of an ontology and how much they cover the domain they are modelling.

Another potential work is to perform a new ontology evaluation tool, which implements the synthesis of pitfall-based and metric-based approaches. Gathering pitfall-based and metric-based evaluation methods under one roof will enable ontology evaluation to be performed more efficiently with a single tool. Scalability will be considered in the design of the tool so that large-scale ontologies can be supported efficiently.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

ORCID iD

Tuğba Özacar

References

Iyer

Sanagavarapu

Raghu

Reddy Y

. A framework for syntactic and semantic quality evaluation of ontologies. In: International Conference on Secure Knowledge Management in the Artificial Intelligence Era, San Antonio, 8-9 October 2021, pp.73–93, Springer.

Guarino

Welty

. Evaluating ontological decisions with OntoClean. Communications of the ACM 2002; 45(2):61–65.

Tartir

Arpinar

Sheth

. Ontological evaluation and validation. In: Kameas

Healy

Poli

(eds) Theory and applications of ontology: computer applications. Springer, 2010, pp.115–130.

Gangemi

Catenacci

Ciaramita

et al. A theoretical framework for ontology evaluation and validation. In: Workshop on Semantic Web Applications and Perspectives, Trento, 14–16December 2005, http://www.ceur-ws.org/Vol-166/9.pdf

Burton-Jones

Storey

Sugumaran

et al. A semioticmetrics suite for assessing the quality of ontologies. Data and Knowledge Engineering 2005; 55(1):84–102.

Dividino

Romanelli

Sonntag

. Semiotic-based ontology evaluation tool(S-OntoEval).In: International conference on language resources and evaluation, Marrakech, 28–30May 2008, http://www.lrec-conf.org/proceedings/lrec2008/pdf/671_paper.pdf

Alani

Brewster

Shadbolt

. Ranking ontologies with AktiveRank. In: International Semantic Web conference, Athens, 5–9 November 2006, pp.1–15, Springer.

Poveda-Villalon

Gomez-Perez

Suarez-Figueroa

. Oops!(ontologypitfallscanner!): An on-line tool for ontology evaluation. International Journal on Semantic Web and Information Systems 2014; 10(2):7–34.

McDaniel

Storey

Sugumaran

. Assessing the quality of domain ontologies: Metrics and an automated ranking system. Data and Knowledge Engineering 2018; 115:32–47.

10.

Lantow

. Ontometrics:Putting metrics into use for ontology evaluation. In: International joint conference on knowledge discovery, knowledge engineering and knowledge management, Porto, 9-11 November 2016, pp.186–191, Springer.

11.

Amardeilh

Laublet

Minel

JL.

Document annotation and ontology population from linguistic extractions. In: International conference on Knowledge capture, Banff, 2–5 October 2005, pp.161–168, ACM.

12.

Wohlgenannt

Sabou

Hanika

. Crowd-based ontology engineering with the uComp Protege Plugin. Semantic Web Journal 2016; 7(4):379–398.

13.

Kiptoo

. Ontology enhancement using crowd sourcing: A conceptual architecture. International Journal of Crowd Science 2020; 4(3):231–243.

14.

Pittet

Barthelemy

. Exploiting users’ feedbacks: Towards a task-based evaluation of application ontologies throughout their lifecycle. In: International conference on knowledge engineering and ontology development, Lisbon, 12–14 November 2015, https://hal.archives-ouvertes.fr/hal-01459827/document

15.

Zhang

Saberi

Chang

. Semantic-based lightweight ontology learning framework: A Case study of intrusion detection ontology. In: International conference on web intelligence, Leipzig, 23–26 August 2017, pp.1171–1177, ACM.

16.

Ding

Finin

Joshi

et al. Swoogle: A search and metadata engine for the semantic web. In: International conference on information and knowledge management, Washington, 8–13 November 2004, pp.652–659, ACM.

17.

Poveda-Villalon

Suarez-Figueroa

Garcia-Delgado

et al. Oops!(ontology pitfall scanner!): Supporting ontology evaluation online. Semantic Web Journal 2009, https://www.semantic-web-journal.net/system/files/swj989.pdf

18.

Villalon

. Ontology evaluation: A pitfall-based approach to ontology diagnosis. PhDThesis, 2016, https://oa.upm.es/39448/1/MARIA_POVEDA_VILLALON.pdf

19.

Chakraborty

. OntoConnect: Domain-agnostic ontology alignment using neural networks. PhDThesis, 2021, Arizona State University, Tempe, AZ.

20.

Carroll

Dickinson

Dollin

et al. Jena: Implementing the semantic web recommendations. In: International World Wide Web conference on alternate track papers and posters, NewYork, 19–21 May 2004, pp.74–83, ACM.

21.

Drummond

Horridge

Stevens

et al. Pizza ontology, 2007, https://protege.stanford.edu/ontologies/pizza/pizza.owl

22.

Tzitzikas

Allocca

Bekiari

et al. Integrating heterogeneous and distributed information about marine species through a top level ontology. In: Metadata and Semantics Research Conference, Thessaloniki, 19-22 November 2013, pp.289–301, Springer.

23.

Hepp

. Goodrelations: An ontology for describing products and services offers on the web. In: International conference on knowledge engineering and knowledge management, Las Vegas, 14–17 July 2008, pp.329–346, Springer.

24.

Lebo

Sahoo

McGuinness

et al. PROV-O: The PROV ontology (W3C Recommendation), 2013, https://www.w3.org/TR/prov-o/

25.

W3C. Wineontology, 2003, http://www.w3.org/TR/owl-guide/wine.rdf

26.

Borner

Conlon

Corson-Rikert

et al.Vivo: A semantic approach to scholarly networking and discovery. Synthesis Lectures on the Semantic Web Theory and Technology 2012; 7(1):1–178.

27.

The OBO Foundry. The symptom ontology (symp), 2017, https://obofoundry.org/ontology/symp.html

28.

Zhang

. Dementia-related psychotic symptoms non-pharmacological treatment ontology, 2021, ncbo-prd-ui-lb.stanford.edu/ontologies/DRPSNPTO

29.

Rijgersberg

Van Assem

Top

. Ontology of units of measure and related concepts. Semantic Web Journal 2013; 4(1):3–13.

30.

Janowicz

Haller

Cox

et al. SOSA: A lightweight ontology for sensors, observations, samples and actuators. Journal of Web Semantics 2019; 56:1–10.

31.

Horrocks

Patel-Schneider

Boley

et al. SWRL: A semantic web rule language combining OWL and RuleML (W3C Member Submission), 2004, https://www.w3.org/Submission/SWRL/