Main Page

From oquarewiki
Jump to: navigation, search


[edit] Welcome to Oquare Wiki

This Wiki is for discussing the characteristics and subcharacteristics of the ontology evaluation method called OQUARE [1]. OQUARE is a framework for evaluating the quality of ontologies based on the standard ISO/IEC 25000:2005 for Software product Quality Requirements and Evaluation known as SQuaRE [2], recent efforts have been put on investigating how such software quality standards can be adapted to measure ontology quality. In Fernandez-Breis, Egaña, and Stevens (2009)[3], we presented an approach based on the ISO 9126.

[edit] The Ontology Quality Evaluation Framework (OQuaRE)

SQuaRE covers two main processes: software quality requirements specifications and software quality evaluation. Besides, the usage of SQuaRE requires the definition of the following components: quality model, quality metrics, quality requirements and quality evaluation. SQuaRE permits the definition of the quality model in terms of quality characteristics. In this way, this standard suggests a series of quality characteristics that should be used for measuring quality, in addition to this, each quality characteristic has a set of quality subcharacteristics associated, which are measured through quality metrics.

Our adaptation to the evaluation of ontologies is OQuaRE which aims to define all the elements required for ontology evaluation: evaluation support, evaluation process and metrics. The current version of our framework includes, so far, the quality model and the quality metrics, because they are the basic modules for the technical evaluation of the quality of the ontology, and which are explained next.

[edit] The OQuaRE Quality Model

This model reuses and adapts the following SQuaRE characteristics to ontologies: functional adequacy, reliability, operability, maintainability, compatibility, transferability, perfformance efficiency and quality in use. Moreover, according to requirements, principles and characteristics of ontologies and to the state of the art of ontology evaluation [4], structural features of ontologies are important to evaluate their quality, but they are not considered in the standard. Consequently, we added such characteristic to our framework. In order to determine the quality subcharacteristics, we combined the ones suggested in SQuaRE with the ones suggested by state-of-the-art methods from the ontology evaluation community. thefunctional adequacy subcharacteristics are the intended uses for ontologies identified in [5]

[edit] The quality metrics of OQuaRE

This model includes the quality metrics attached to the elements of the quality model. A series of metrics that will be automatically calculated for some of the characteristics and subcharacteristics of OQuaRE.

[edit] Structural

This category is the only one in this framework that is not specified as such in SQuaRE, but it is important when evaluating ontologies, since it accounts for ontology quality factors such as consistency, formalisation, redundancy or tangledness

Subcharacteristic Definition Description Metrics
Formalisation Capability of the ontology to support reasoning An ontology should be expressed in a common formal language E.j. for biontologies in agree with the ontology principle FP 002 format [6]: OBO Format, OWL or OWL2 concrete syntax, RDF/XML, OWL2-XML, OWL2-Manchester Syntax, Common Logic concrete syntax, CLIF, Conceptual Graphs, etc
Formal relations support Capability of the ontology to represent relations supported for formal theories different to the formal support for taxonomy It accounts for the types of formal relations supported by the ontology, different to part-of relations RROnto
Redundancy Capability of the ontology to be informative Some redundancy types are: Inferred information more than once from the relations, classes and instances found in ontology.

The same/different formal definition of classes, properties or instances referring to different/same classes, properties or instances [7]

Structural accuracy Degree of the correctness of the terms used in the ontology correctness of taxonomic links, use of upper level with disjoint categories, consideration of "all some rule" in case of existential restrictions, value restrictions only if disjoint partitions available, domain and range restrictions of object properties, sufficient metadata and annotation properties, free text definitions where necessary
Consistency Degree of the consistency of the ontology consistent naming conventions, Logical consistency, structural consistency, consistent distinction class - instance
Tangledness This measures the distribution of multiple parent categories, so that it is related to the existence of multiple inheritance. TMOnto
Cycles The existence of cycles through a particular semantic relation is usually a sign of bad design, since they may lead to inconsistencies.
Cohesion An ontology has a high cohesion if the classes are strongly related. LCOMOnto
Domain coverage The degree to which The ontology cover the specified domain It is evaluated by experts.

[edit] Functional adequacy

The capability of the ontologies to provide concrete functions.

Subcharacteristic Definition Description Metrics
Reference ontology Degree in which the ontology can be used as a reference resource for the particular domain the ontology is built for. Machines can exploit better the reference knowledge offered by an ontology which have a more explicit structure
Controlled vocabulary Capability of the ontology to avoid heterogeneity of the terms. The labels of the ontological entities are used to avoid heterogeneity, which would complicate data processing and analysis. Ontologies provide terminology management, unifying the terms used for referring to the included knowledge entities which should have an unambiguous and non-redundant definition. ontological concepts are described with different terminologies, different meanings are assigned to the same word in different contexts and different taxonomies are examples of synonymy, polysemy and structural heterogeneity [8]. ANOnto
Schema and value reconciliation. Degree in which ontology provide a common data model that can be applied to reconciliation and integration. Integration is building a new ontology reusing other available ontologies.

Different ontologies with subjective features and particular perspective on the world, cannot be compared without reconciliation and integration, which are necessary to interchange, migration and standardization of information and knowledge of such ontologies

RROnto, AROnto, Formal degree, consistency degree
Consistent search and query The degree which the formal model and structure of the ontology provide a semantic context to evaluate which are the data wanted by the users, allowing better querying and searching methods ANOnto,RROnto, AROnto, INROnto, Formal degree
Knowledge acquisition – representation. Capability of the Ontology to represent the knowledge acquired. Knowledge acquisition is the gathering, storage, and encoding of existing information. ANOnto, RROnto, NOMOnto
Clustering Degree in which the annotations of data with respect to ontology terms can be used for clustering such data against the aspects of the ontology. Clustering can be defined as the process of organizing objects into groups whose members are more similar to each other than to individuals in other groups. ANOnto
Similarity Capability of the component of the ontology to be compared for similarity There exist different similarity measures: Taxonomy similarity, Relation similarity, Attribute similarity, semantic similarity. [9] RROnto, AROnto
Indexing and linking Degree in which the classes defined in the ontology can act as indexes for quick information retrieval RROnto, AROnto, INROnto
Results representation Capability of the ontology to analize complex results such as microarrays experiments AROnto, CROnto
Classifying instances Degree in which ontology Instances can be recognized as member of a certain class
Text analysis Capability of the structure of the ontology to helps detecting associations between words or concepts and classifying word types. Formal degree
Guidance Capability of the ontology to guide the specification of domain theories. Ontology by capturing knowledge about a domain and encapsulating constraints about class membership provides guidance in the specification of domain theories and support decision making processes. INROnto, AROnto
Decision trees Capability of the ontology to be used building Decision trees. Decision trees are used to represent the logical structures of classification rules for domain specific empirical data. INROnto, AROnto, TMOnto
Knowledge reuse The degree to which The ontology knowledge can be used to build other ontologies. ANOnto,AROnto, INROnto, Formal degree, NOMOnto,LCOMOnto, Consistency degree
Inference The degree to which The formal model of the ontology can be used by reasoners to make implicit knowledge explicit. Inference expands the knowledge base with additional information using the existing data, metadata, and rules. formal degree, RROnto, CROnto
Precision The degree to which The ontology provides the right or specified results with the needed degree of accuracy

[edit] Compatibility

The ability of two or more software components to exchange information and/or to perform their required functions while sharing the same hardware or software environment

Subcharacteristic Definition Description Metrics
Replaceability The degree to which The ontology can be used in place of another specified Ontology for the same purpose in the same environment. WMCOnto, DITOnto, NOCOnto, NOMOnto
Interoperability The degree to which the ontology can be cooperatively operable combining its knowledge with one or more other ontologies. Ontology matching consists of matching a concept from one ontology to another.

[edit] Transferability

The degree to which the software product can be transferred from one environment to another

Subcharacteristic Definition Description Metrics
Portability The degree in which an Ontology or one part of the ontology can be transferred from one hardware or software environment to another The degree to which The ontology can be translated between different formal languages, or how easily the code can be moved to another language
Adaptability The degree to which The ontology can be adapted for different specified environments (languages, expresivity levels) without applying actions or means other than those provided for this purpose for the Ontology considered. WMCOnto, DITOnto, RFCOnto, CBOnto

[edit] Operability

Effort needed for use, and on the individual assessment of such use, by a stated or implied set of users.

Subcharacteristic Definition Description Metrics
Appropriateness recognisability The degree to which the Ontology enables users to recognise whether it is appropriate for their needs. The ability to recognise the appropriateness of the functions from initial impressions of the ontology and/or any associated documentation such as Manuals, guides, comments.
Learnability The degree to which the ontology enables users to learn its application. Effectiveness of the user documentation and/or help system. WMCOnto, LCOMOnto, RFCOnto, NOMOnto,CBOnto, NOCOnto
Ease of use The degree to which the ontology makes it easy for users to operate and control it.
Helpfulness The degree to which the Ontology provides help when users need assistance. The ontology provide clear error messages, manuals and guides for help the users, including help comprehensive, effective and easy to find.

[edit] Maintainability

The capability of ontologies to be modified for changes in environments, in requirements or in functional specifications.

Subcharacteristic Definition Description Metrics
Modularity The degree to which the ontology is composed of discrete components such that a change to one component has minimal impact on other components. WMCOnto, CBOnto
Reusability The degree to which an asset (part of) the ontology can be used in more than one ontology, or in building other assets. WMCOnto, CBOnto, DITOnto, NOCOnto, RFCOnto, NOMOnto
Analysability The degree to which The ontology can be diagnosed for deficiencies or causes of failures (inconsistences), or for the parts to be modified to be identified. WMCOnto, LCOMOnto, RFCOnto, NOMOnto,CBOnto, DITOnto
Changeability The degree to which The Ontology enables a specified modification to be implemented. The ease with which a The ontology can be modified. WMCOnto, LCOMOnto, RFCOnto, NOMOnto,CBOnto, NOCOnto, DITOnto
Modification stability: The degree to which The ontology can avoid unexpected effects from modifications of the software or knowledge. WMCOnto, RFCOnto, NOMOnto,CBOnto, NOCOnto
Testability The degree to which the ontology modified can be validated. WMCOnto, RFCOnto, NOMOnto,CBOnto,DITOnto

[edit] Quality in use

Quality in a particular context of use. Quality in use is the degree to which a product used by specific users meets their needs to achieve specific goals

Subcharacteristic Definition Description Metrics
Usability in use Effectiveness in use: the degree to which specified users can achieve specified goals with accuracy and completeness in a specified context of use.

Efficiency in use: The degree to which specified users expend appropriate amounts of resources in relation to the effectiveness achieved in a specified context of use. Satisfaction in use: The degree to which users are satisfied in a especified context of use. Satisfaction is further subdivided into sub-subcharacteristics: Likability (cognitive satisfaction), Pleasure (emotional satisfaction), Comfort (physical satisfaction), Trust.

Flexibility in use Context conformity in use: The degree to which usability in use meets requirements in all the intended contexts of use. Context extendibility in use: The degree of usability in use in contexts beyond those initially intended


  1. A. Duque-Ramos, J.T. Fern�andez-Breis, R. Stevens, N. Aussenac-Gilles, Oquare: a square-based approach for evaluating the quality of ontologies Journal of Research and Practice in Information Technology 43(2011) 159-73.
  2. ISO25000 2005. ISO/IEC 25000 2005, Software engineering - Software product quality requirements and evaluation (SQuaRE) - guide to square (ISO/IEC 25000). Geneva, Switzerland: International Organization for Standardization.
  3. Fernández-Breis, J., Egaña, M. and Stevens, R. (2009): A quality evaluation framework for bio-ontologies. In: PRECEDINGS, N., ed. ICBO International Conference on Biomedical Ontology, University at Buffalo, NY. 136–139
  4. GANGEMI, A., CATENACCI, C., CIARAMITA, M. and LEHMANN, J. (2006): Modelling ontology evaluation and validation. Semantic Web: Research and Applications, Proceedings, 4011: 140 –154.
  5. STEVENS, R. and LORD, P. (2009): Application of ontologies in bioinformatics. In: BERNUS, P., BŁAŻEWICS, J., SCHMIDT, G., SHAW, M., STAAB, S. and STUDER, R. (eds.) Handbook on Ontologies. Springer Berlin Heidelberg.
  7. FAHAD M., QADIR, M. (2008). A Framework for Ontology Evaluation. In Proceedings International Conference on Conceptual Structures (ICCS'08), Toulouse, France, July, page 711.
  8. KIU, C.-C., & LEE, C.-S. (2006). Ontology Mapping and Merging through OntoDNA for Learning Object Reusability. Educational Technology & Society, 9 (3), 27-42
  9. MAEDCHE A., ZACHARIAS V. (2002). Clustering Ontology-based Metadata in the Semantic Web. In Proceedings 6th European Conference on Principles of Data Mining and Knowledge Discovery, Springer-Verlag, London, Uk, 348-360
Personal tools

OQuaRE Framework
The Case Studies
Wiki Options