IAA: calculation methods

As of now, the GUI shows a default method to calculate the Inter-Annotator Agreement (IAA) for all annotation types, namely: document labels, entity types, entity normalizations, entity labels, and entity relations. The default method is called in the descriptions below: exact_v1. The API, on top, allows you retrieve the IAA calculated with three different methods, namely: {exact_v1, overlapping_v1, documentlevel_v1}. Not all calculation methods are available for all annotation types. Below we describe the possible IAA calculation methods, grouped by annotation type. First some definitions:

Definitions


Document labels

Given two different document labels (from two different annotation versions, e.g. different members), they are equal if:

exact_v1 (default)


Entity Types

Given two entity types (from two different annotation versions, e.g. different members), they are equal if:

exact_v1 (default)

overlapping_v1


Entity Normalizations

Given two entity normalizations (from two different annotation versions, e.g. different members), they are equal if:

exact_v1 (default)

overlapping_v1

documentlevel_v1

In this case, the entity offsets are disregarded. In practice, the entity normalizations’ values (of a document) are compared as if they were sets. That is, for instance, if annotation version A had the normalization x annotated 5 times, and the annotation version B had the same normalization x annotated only once, the repetitions are disregarded and both annotations, for this particular normalization, are considered equal.


Entity Labels

Given two entity labels (from two different annotation versions, e.g. different members), they are equal if:

exact_v1 (default)

overlapping_v1


Entity Relations

Given two entity relations (from two different annotation versions, e.g. different members), relating two entities (e.g. a & b and a’ & b’, resp.), they are equal if:

exact_v1 (default)

overlapping_v1