IAA: calculation methods

As of now, the GUI shows a default method to calculate the Inter-Annotator Agreement (IAA) for all annotation types, namely: document labels, entity types, entity normalizations, entity labels, and entity relations. The default method is called in the descriptions below: exact_v1. The API, on top, allows you retrieve the IAA calculated with three different methods, namely: {exact_v1, overlapping_v1, documentlevel_v1}. Not all calculation methods are available for all annotation types. Below we describe the possible IAA calculation methods, grouped by annotation type. First some definitions:

Definitions


Document labels

Given two different document labels (from two different annotation versions, e.g. different members), they are equal iff:

exact_v1 (default)


Entity Types

Given two entity types (from two different annotation versions, e.g. different members), they are equal iff:

exact_v1 (default)

overlapping_v1


Entity Normalizations

Given two entity normalizations (from two different annotation versions, e.g. different members), they are equal iff:

exact_v1 (default)

overlapping_v1

documentlevel_v1


Entity Labels

Given two entity labels (from two different annotation versions, e.g. different members), they are equal iff:

exact_v1 (default)

overlapping_v1


Entity Relations

Given two entity relations (from two different annotation versions, e.g. different members), relating two entities (e.g. a & b and a’ & b’, resp.), they are equal iff:

exact_v1 (default)

overlapping_v1