Relevance Judgments for Web Services Retrieval – A Methodology and Test Collection for SWS Discovery Evaluation

Semantic web services (SWS) promise to take service oriented computing to a new level by allowing to automatically locate and use functionality exposed as web services. At the core of SWS are solutions to the problem of SWS discovery, i.e., the problem of comparing semantic goal descriptions with semantic offer descriptions to determine services relevant to a given request.
A plethora of different approaches to this problem have been proposed, but their comparative evaluation is challenging.
While the evaluation setups from the information retrieval (IR) community provided a natural starting point to SWS discovery evaluation, their shortcomings and the applicability of their assumptions have not been sufficiently discussed so far.

This paper discusses the differences between traditional IR and SWS discovery and the resulting fundamental problems. It proposes a methodology and experimental setup for SWS discovery evaluation that addresses these problems. An initial realistic service test collection is presented and issues related to the central notion of relevance in the context of service discovery are examined. In particular the consistency of relevance judgments using three different relevance scales is experimentally investigated and the consequences on the evaluation methodology are discussed.

@INPROCEEDINGS{KK09,
  author = {Ulrich K\"uster and Birgitta K\"onig-Ries},
  title = {Relevance Judgments for Web Services Retrieval - A Methodology and
	Test Collection for SWS Discovery Evaluation},
  booktitle = {Proceedings of the 7th IEEE European Conference on Web Services (ECOWS09)},
  year = {2009},
  month = {November},
  address = {Einhoven, The Netherlands},
  abstract = {Semantic web services (SWS) promise to take service oriented computing
	to a new level by allowing to automatically locate and use functionality
	exposed as web services. At the core of SWS are solutions to the
	problem of SWS discovery, i.e., the problem of comparing semantic
	goal descriptions with semantic offer descriptions to determine services
	relevant to a given request.
	
	A plethora of different approaches to this problem have been proposed,
	but their comparative evaluation is challenging.
	
	While the evaluation setups from the information retrieval (IR) community
	provided a natural starting point to SWS discovery evaluation, their
	shortcomings and the applicability of their assumptions have not
	been sufficiently discussed so far.
	
	
	This paper discusses the differences between traditional IR and SWS
	discovery and the resulting fundamental problems. It proposes a methodology
	and experimental setup for SWS discovery evaluation that addresses
	these problems. An initial realistic service test collection is presented
	and issues related to the central notion of relevance in the context
	of service discovery are examined. In particular the consistency
	of relevance judgments using three different relevance scales is
	experimentally investigated and the consequences on the evaluation
	methodology are discussed.}
}