OAEI-2006 - Consensus workshop
!NEW! OAEI-2007 - seeking contributions to the 'conference' track collection !NEW!
Content of web:
Motivation and Task
Data sets
Organisation and Presentation
Evaluation
Results of participants
Motivation and Task
As a complement to current type of ontology alignment evaluation scenario, this year will also be held a consensus workshop. It will
serve for presentation and discussion of results of mining for interesting ontology alignments. There will
be no prior reference alignment, and consensus on alignment will be reached via discussion.
Compared to the contest scenario, it potentially enables to discuss:
- alignment types unforeseen by organizers,
- alignment types too sophisticated to be evaluated using standard metrics,
- negative results as useful feedback.
Data sets
The collection of ontologies is dealing with conference organisation. They have been developed within OntoFarm project. 10 ontologies are included. Each of them is shortly described in the
following table. Ontologies have been based upon three types of underlying resources:
- actual conference (series) and its web pages (indicated in column Type with mark Web),
- actual software tool for conference organisation support (indicated in column Type with mark Tool),
- experience of people with personal participation in conference organisation (indicated in column Type with mark Insider).
Name is derived from name of conference or conference organisation tool. For downloading this ontology, just click on its name.
DL expressivity was obtained from SWOOP tool which is Hypermedia-based OWL Ontology Browser and Editor.
We deeply acknowledge the effort of contributors to the collection:
Július Štuller (and his team), Dušan Rak, Petr Tomášek, Luděk Mácha, Lenka
Navrátilová, Pavel Kříž, and David Merka.
Organisation and Presentation
Send any questions, comments, or suggestions to:
Ondřej Šváb ( svabo at vse dot cz ) or Vojtěch Svátek ( svatek at vse dot cz ).
Final results have been presented at Ontology Matching workshop 2006 during so-called 'Consensus building workshop'.
During presentation, 'controversial' mappings have been argued. Argumentation process is now under processing.
Presentation from 'Consensus building workshop' is here.
Evaluation
According to the nature of this track, we are mainly interested in some
"interesting" mappings ("nuggets"). Although traditional evaulation was not
our intention, we made some sort of evaluation as a side-effect of processing
results from our six participants.
All the statistics as well as precision and recall have been provisionally made by track organisers, who can
often be subjective; the focus of the track is on interesting individual
alignments and repeated patterns rather than on precision/recall figures.
The abovementioned table encompasses several numerical statistics related to the results of six participants, called according to
name of their systems (Automs, Coma++, OWL-CtxMatch, Falcon, HMatch and RiMOM). Finally, there is row with overall statistics.
In the following, columns are explained:
- measure shows whether mapping is strictly true/false or is scaled between 0 and 1,
- # alignments shows number of alignments per one system,
- # mappings shows number of all mappings which have been included in "assessment,"
- # correct shows number of correct mappings, in other words, it is number of true positive mappings,
- # uniq correct shows number of unique correct mappings,
ie. o1:entity1 o2:entity2 was counted once if at least one system include it in its result.
Also o1:entity1 o2:entity2 and o2:entity2 o1:entity1 was taken as the same one,
- # interesting shows number of "interesting" correct mappings, these mappings are correspondences which is not so easy to find at first sight (due to eg. string-based approach is not enough),
- # incorrect shows number of incorrect mappings, ie. false positive mappings,
- # subsumptions shows number of mappings which have been wrong classified as equivalence, but it should be in
correspondence with relation of subsumption, in arbitrary direction,
- # siblings shows number of mappings which have been wrong classified as equivalence, but mapped elements are rather siblings,
- # inversion shows number of mappings which have been wrong classified as equivalence, but mapped relations are rather inverse than equivalent,
- # relClass shows number of mappings which is mapping between class and relation or vice versa,
- # int FP shows number of "interesting" incorrect mappings, ie. "interesting" false positive mappings.
These mappings are incorrect correspondences which do not belong to any abovementioned groups (eg. subsumptions, siblings, etc.),
- # unclear shows number of unclear mappings where evaluator has not been able to decide whether mapping is correct or not.
The following columns are dealing with measure of precision and recall:
- precision (P) is computed as ratio of the number of all correct mappings to the number of all mappings,
- rrecall (rR) is computed as ratio of the number of all unique correct mappings (sum of all unique correct mappings per one system) to the number of all unique correct mappings found by any of systems (per all systems).
This is our "relative" recall.
Additional comments:
- in the case of HMatch system, 4529 relations with measure of similarity 0.64 were not included in assessment to make results manually processable,
- in the case of RiMOM system, mapping with measure higher than 0.199 were included in assessment. Number of mappings with measure of similarity lower than 0.199 or equal is 2477,
- in the right-hand corner is stated the number of all unique correct mappings which is useful for our realtive (approximative) assessment of recall.
Results of participants
Results of all participants and manually labeled data in this track can be downloaded from results page.