On the Evaluation of Semantic Web Service Frameworks

In recent years a huge amount of research effort and funding has been devoted to the area of semantic web services (SWS). This has resulted in the proposal of numerous competing approaches to facilitate the automation of mediation, choreography and discovery for web services using semantic annotations. However, despite of a wealth of theoretical work, too little effort has been spent towards the comparative experimental evaluation of the competing approaches so far. Progress in scientific development and industrial adoption is thereby hindered. An established evaluation methodology and standard benchmarks that allow the coparative evaluation of different frameworks are thus needed for the further advancement of the field. To this end, a criteria model for SWS evaluation is presented and the existing approaches towards SWS evaluation are comprehensively analyzed. Their shortcomings are discussed in order to identify the fundamental issues of SWS evaluation. Based on this discussion, a research agenda towards agreed upon evaluation methodologies is proposed.

@ARTICLE{KKPK08,
  author = {Ulrich K\"uster and Birgitta K\"onig-Ries and Charles Petrie and
	Matthias Klusch},
  title = {On the Evaluation of Semantic Web Service Frameworks},
  journal = {International Journal On Semantic Web and Information Systems},
  year = {2008},
  volume = {4},
  number = {4},
  abstract = {In recent years a huge amount of research effort and funding has been
	devoted to the area of semantic web services (SWS). This has resulted
	in the proposal of numerous competing approaches to facilitate the
	automation of mediation, choreography and discovery for web services
	using semantic annotations. However, despite of a wealth of theoretical
	work, too little effort has been spent towards the comparative experimental
	evaluation of the competing approaches so far. Progress in scientific
	development and industrial adoption is thereby hindered. An established
	evaluation methodology and standard benchmarks that allow the coparative
	evaluation of different frameworks are thus needed for the further
	advancement of the field. To this end, a criteria model for SWS evaluation
	is presented and the existing approaches towards SWS evaluation are
	comprehensively analyzed. Their shortcomings are discussed in order
	to identify the fundamental issues of SWS evaluation. Based on this
	discussion, a research agenda towards agreed upon evaluation methodologies
	is proposed.},
  keywords = {evaluation, survey, overview, semantic web services}
}