Evaluating Semantic Web Service Matchmaking Effectiveness Based on Graded Relevance

Semantic web services (SWS) promise to take service oriented computing to a new level by allowing to semi-automate time-consuming programming tasks. At the core of SWS are solutions to the problem of SWS matchmaking, i.e. the problem of comparing semantic goal descriptions with semantic offer descriptions to determine services able to fulfill a given request.
Approaches to this problem have so far been evaluated based on binary relevance despite the fact that virtually all SWS matchmakers support more fine-grained levels of match. In this paper, a solution to this discrepancy is presented. A graded relevance scale for SWS matchmaking is proposed as are measures to evaluate SWS matchmakers based on such graded relevance scales. The feasibility of the approach is shown by means of a preliminary evaluation of two hybrid OWL-S matchmakers based on the proposed measures.

@INPROCEEDINGS{KK08a,
  author = {Ulrich K\"uster and Birgitta K\"onig-Ries},
  title = {Evaluating Semantic Web Service Matchmaking Effectiveness Based on
	Graded Relevance},
  booktitle = {Proceedings of the 2nd International Workshop SMR$^2$ on Service
	Matchmaking and Resource Retrieval in the Semantic Web at the 7th
	International Semantic Web Conference (ISWC08)},
  year = {2008},
  month = {October},
  address = {Karlsruhe, Germany},
  abstract = {Semantic web services (SWS) promise to take service oriented computing
	to a new level by allowing to semi-automate time-consuming programming
	tasks. At the core of SWS are solutions to the problem of SWS matchmaking,
	i.e., the problem of comparing semantic goal descriptions with semantic
	offer descriptions to determine services able to fulfill a given
	request. 
	
	Approaches to this problem have so far been evaluated based on binary
	relevance despite the fact that virtually all SWS matchmakers support
	more fine-grained levels of match. In this paper, a solution to this
	discrepancy is presented. A graded relevance scale for SWS matchmaking
	is proposed as are measures to evaluate SWS matchmakers based on
	such graded relevance scales. The feasibility of the approach is
	shown by means of a preliminary evaluation of two hybrid OWL-S matchmakers
	based on the proposed measures.}
}