Data and Results for the Experiment reported in the Paper "Evaluating Semantic Web Service Matchmaking Effectiveness Based on Graded Relevance"

This page lists data and results for the experiment reported in the paper "Evaluating Semantic Web Service Matchmaking Effectiveness Based on Graded Relevance" by Ulrich Küster and Birgitta König-Ries, submitted to the Second International Workshop SMR2 2008 on Service Matchmaking and Resource Retrieval in the Semantic Web at the 7th International Semantic Web Conference (ISWC2008).

Data

Service Rankings

The following table makes the test data available. For each query the ranked list of services is available as query names as used by OWLS-TC or alternatively as IDs as used by OPOSSum. The ranked lists were provided by Patrick Kapahnke and Matthias Klusch from DFKI Saarbrücken from data obtained within the S3 Contest on Semantic Service Selection. The data was created using OWLS-TC 2.2 Revision 2.

The files from Patrick Kapahnke and Matthias Klusch contain a ranked list of service names as used in OWLS-TC. These had to be mapped to OPOSSum service ids. The table below contains the original rankings as well as the rankings after they have been converted to OPOSSum IDs. This file contains the mapping from OWLS-TC service names to OPOSSum IDs that was computed to convert the ranking files. It also shows a flag that indicates whether the corresponding service belongs to the OWLS-TC Education subset (thus part of the evaluation data) or not.

Query OPOSSum ID Results M3 Results iMatcher
country_skilledoccupation_service.owls 5654 Query names Query IDs Query names Query IDs
governmentdegree_scholarship_service.owls 5659 Query names Query IDs Query names Query IDs
novel_author_service.owls 5664 Query names Query IDs Query names Query IDs
publication-number_publication_service.owls (not evaluated since no iMatcher results) 5666 Query names Query IDs Query names Query IDs
researcher-in-academia_address_service.owls 5668 Query names Query IDs Query names Query IDs
university_lecturer-in-academia_service.owls 5675 Query names Query IDs Query names Query IDs

Relevance Judgements

This file list the expert relevance judgments used in our experiment. The first column shows the OPOSSum request ID, the second column shows the OPOSSum service ID, the third column shows the judged relevance. The file contains both graded judgements and the binary judgements from OWLS-TC.

Results

The results of evaluating the ranked lists with the various measures (see paper) are available here. The file lists the following data:

Links