Multi-user annotation

tagtog is a multi-user tool. Collaborate with other users to annotate faster and improve the quality of your annotations.

It supports different roles as annotator. Each user can annotate their own copy of the text, facilitating the review process and measurement of inter-annotator agreement (IAA).

Roles

Role Description
admin Can read all user’s annotations, but cannot edit them. They can edit master’s and their own annotations. Moreover, they can edit the project’s settings.
supercurator They can edit master's and their own annotations.
curator Can edit their own annotations. They cannot edit master's annotations, but can export master into their annotations. Coming soon.
reader Users invited as collaborators to a project. They cannot edit any annotations. They can only read master's annotations. Coming soon.
Admin role
Component Privileges
Master annotations Replace master annotations using the annotations from other project member.
Guidelines Edit project guidelines.
Entities Edit or create entity types.
Document labels Edit or create document labels
Entity labels Edit or create entity labels
Dictionaries Edit or create dictionaries
Annotatables Edit options under the Annotatables settings
Annotations Edit options under the Annotations settings
Members Add or remove members to the project
Project Delete project

Annotation versions

Each user has an independent version of the annotations for each single document. For instance, UserA could have 20 entities; UserB could have 5 different entities on the same exact document. In addition, each document has a master version which is usually treated as the final/official version.

Annotation flows

There are different ways you can organize your annotation tasks. These are the most common:


Annotators annotate directly on the master version. No review.

This is the simplest flow and there is no review step. Make this choice if you are working alone, or if you trust your annotators' annotations or if time is a constraint. This is the project's default.

1Add users to your project. As a project's admin, go to Settings → Members to add members to your project.

2Create clear guidelines. Here the admin writes what is to be annotated and which type of annotations to use. Clear and complete guidelines are key to align all project members.

3Import text. Admins and supercurators can import the documents to be annotated by the group. Any project member can see these documents.

4Distribute documents among annotators. Either let users annotate non-yet-confirmed documents, or otherwise, for example, manually assign document ids to each user.

5The group starts annotating. Each user annotates only the master version of the assigned documents. Once a document is annotated, the user marks the annotations as completed by clicking the Confirm button. admin's can check the progress in the document list view.


Documents are automatically distributed; one annotator per document

Make this choice if the annotation task is simple or if time is a constraint. If you assign each document to only one annotator, the quality of the annotations depends on the assigned user.

1Add users to your project. As a project's admin, go to Settings → Members to add members to your project.

2Create clear guidelines. Here the admin writes what is to be annotated and which type of annotations to use. Clear and complete guidelines are key to align all project members.

3Import text. Admins and supercurators can import the documents to be annotated by the group. Any project member can see these documents.

4Distribute documents among annotators. As a project's admin, go to Settings → Members and select 1 annotator per document. Additionally, choose whether the project's owner should be assigned documents to annotate or not.

5The group starts annotating. Users annotate their version of the annotations for the documents assigned. Once completed, the user mark her/his version as completed by clicking on the Confirm button.

6Review. Admins review the annotated documents and if ready, import the user's annotations to the master version (final version). If the annotations were not Confirmed yet, admins should click on the Confirm button in the master version to indicate that the review is completed.


Documents are automatically distributed; multiple annotators per document

This flow is ideal for those projects requiring high-quality annotations and complex annotation tasks (specific skills required, divergent interpretations, etc.).

1Add users to your project. As a project's admin, go to Settings → Members to add members to your project.

2Create clear guidelines. Here the admin writes what is to be annotated and which type of annotations to use. Clear and complete guidelines are key to align all project members.

3Import text. Admins and supercurators can import the documents to be annotated by the group. Any project member can see these documents.

4Distribute documents among annotators. As a project's admin, go to Settings → Members and select 2 annotators or more per document. Additionally, choose whether the project's owner should be assigned documents to annotate or not.

5The group starts annotating. Users annotate their version of the annotations for the documents assigned. Once completed, the user mark her/his version as completed by clicking on the Confirm button.

6Review. Admins review the annotated documents and if ready, import the user' annotations to the master version (final version). If the annotations were not Confirmed yet, admins should click on the Confirm button in the master version to indicate that the review is completed.