Navigation

Towards Standard Test Collections for the Empirical Evaluation of Semantic Web Service Approaches

Title: Towards Standard Test Collections for the Empirical Evaluation of Semantic Web Service Approaches
Authors: Ulrich Küster, Birgitta König-Ries
Source: International Journal of Semantic Computing 2(3)
Date: 2008-09-01
Abstract:

Semantic web services have received a significant amount of attention in the last years and many frameworks, algorithms and tools leveraging them have been proposed. Nevertheless surprisingly little effort has been put into the evaluation of the approaches so far. The main blocker of thorough evaluations is the lack of large and diverse test collections of semantic web services. In this paper we analyze requirements on such collections and shortcomings of the state of the art in this respect. Our contribution to overcoming those shortcomings is OPOSSum, a portal to support the community to build the necessary standard semantic web service test collections in a collaborative way. We discuss, how existing test collections have been integrated with OPOSSum, showcase the benefits of OPOSSum by an illustrative use case and outline next steps towards better standard test collections of semantic web services.

File: IJSC08-Draft.pdf
URL: http://www.worldscinet.com/cgi-bin/details.cgi?id=voliss:ijsc_0203&type=toc
BibTex:
@ARTICLE{KK08c,
  author = {Ulrich K\"uster and Birgitta K\"onig-Ries},
  title = {Towards Standard Test Collections for the Empirical Evaluation of
	Semantic Web Service Approaches},
  journal = {International Journal of Semantic Computing},
  year = {2008},
  volume = {2},
  pages = {381--402},
  number = {3},
  month = {September},
  abstract = {Semantic web services have received a significant amount of attention
	in the last years and many frameworks, algorithms and tools leveraging
	them have been proposed. Nevertheless surprisingly little effort
	has been put into the evaluation of the approaches so far. The main
	blocker of thorough evaluations is the lack of large and diverse
	test collections of semantic web services. In this paper we analyze
	requirements on such collections and shortcomings of the state of
	the art in this respect. Our contribution to overcoming those shortcomings
	is OPOSSum, a portal to support the community to build the necessary
	standard semantic web service test collections in a collaborative
	way. We discuss, how existing test collections have been integrated
	with OPOSSum, showcase the benefits of OPOSSum by an illustrative
	use case and outline next steps towards better standard test collections
	of semantic web services.}
}