An Evaluation Methodology and Framework for Semantic Web Services Technology

Title: An Evaluation Methodology and Framework for Semantic Web Services Technology
Authors: Ph.D. Thesis Ulrich Küster
Source: Friedrich-Schiller-University Jena
Place: Jena, Germany (also available as a book under ISBN 978-3-8325-2538-5)
Date: 2010-06-01

To foster reuse state of the art software engineering has been driven over decades by the trend towards more and more component based software development. In recent years another trend towards more and more distributed and more loosely coupled systems could be observed. Service oriented architectures (SOAs) are the latest product of this long-reaching development. Web services in particular have become increasingly popular as the probably most prominent implementation of a SOA.

The grand vision of the web service paradigm is to have a rich library of millions of web services available online that provide access to information, functionality or resources of any kind and that can be easily integrated into existing applications or composed in a workflow-like fashion to form new applications.

Even though this promising technology has already proven to be an effective way of creating widely distributed and loosely coupled systems, the manual tasks of integrating the services is still labor intensive and thus expensive work. Thus – following the vision of the Semantic Web – the idea of Semantic Web Services (SWS) was introduced, applying the principles of the semantic web to the web service paradigm. Numerous efforts providing formal semantic descriptions for component services have been put forward. Based on such descriptions, frameworks are designed to support automated or semi-automated dynamic service discovery, composition, binding and invocation, enabling the creation of new kinds of flexible and adaptable applications and reducing long-term development cost.

SWS related research has flourished in recent years and the presented approaches become increasingly more sophisticated and mature. Yet, very little effort is put into the evaluation of the various approaches. Until very recently there were no comparative evaluations and it was impossible to find two systems which had been evaluated on the same use cases. Existing evaluations were mostly concentrated either on artificially synthesized datasets under questionable assumptions or based on one or two use cases or case studies which are often reverse engineered from the solution. This shortcoming hinders future scientific progress and the transfer of research results into industry.

The presented thesis argues that established evaluation methodologies and standard benchmarks that allow efficient comparative evaluations of the competing approaches are needed for the further advancement of the field. It further argues that community based benchmarking initiatives are the most suitable vehicle to define such standards. Common initiatives not only promote the relevance, quality and acceptance of evaluations, their existence often also results in greater communication and collaboration among different researchers leading to a stronger consensus on the community’s research goals.

To lay the foundation for thorough SWS evaluations, the important questions what to evaluate, which criteria to use, how to measure those criteria and how to achieve reliability, validity and impartiality need to be answered. Thus, as the first major contribution of this thesis, a comprehensive and well-founded conceptual model for SWS technology evaluation that identifies the criteria of evaluation and requirement standards to ensure and promote evaluation quality is presented. Based upon this model, the state of the art is reviewed in detail. Three further contributions towards improved SWS technology evaluation are motivated and provided.

First, issues around test data for SWS evaluation are investigated. Based upon a requirements analysis, deficiencies of the state of the art are discussed. It is argued that future data should be developed collaboratively. To support this, a portal is designed and implemented, which allows sharing and reusing test data more effectively and creating better datasets in a distributed way. The portal’s utility is illustrated by using it for the development of an exemplary new test collection that improves the state of the art in important aspects.

Second, a methodology and benchmark for evaluating the functional scope of SWS frameworks is developed. It allows certifying capabilities of different approaches in an objective way and additionally establishes an understanding of the fundamental challenges in the covered area. The methodology has been implemented and executed under the umbrella of the SWS Challenge community evaluation initiative.

Third, a methodology and benchmark for evaluating SWS matchmakers is developed. It assesses the correctness of SWS discovery across formalisms based upon a set of improved measures. Additionally, it provides means for investigating other important aspects like the necessary coupling between service requesters and providers or the effects of more or less comprehensive service descriptions. It has been implemented and executed under the umbrella of the S3 Contest, the second community initiative in the area.

The thesis contributions are validated by discussing them methodologically and assessing them with respect to the requirements catalogue provided as part of this thesis. Additional validation results from their successful dissemination to the community via the before mentioned initiatives, which is also documented within this thesis. The thesis concludes with a summary and an outlook on future research in the area.

File: Diss.pdf
  author = {Ulrich K\"uster},
  title = {An Evaluation Methodology and Framework for Semantic Web Services Technologies},
  school = {Friedrich-Schiller-University Jena},
  year = {2010},
  address = {Jena, Germany},
  month = {June}