Multi-user annotation

tagtog is a multi-user tool. Collaborate with other users to annotate faster and improve the quality of your annotations.

It supports different roles as annotator or admin. Each user can annotate their own copy of the text, facilitating the review process and measurement of inter-annotator agreement.

Roles

Role Description
admin The user who created the project. This user can read other user's annotations, but cannot edit them. They can import into master any user's annotations.
supercurator Users invited as collaborators to a project who only can see their annotations and master's. They can edit master's annotations and export these into their annotations.
curator Users invited as collaborators to a project who only can edit their own annotations. They cannot edit master's annotations, but can export master into their annotations. Coming soon.
reader Users invited as collaborators to a project. They cannot edit any annotations. They can only read master's annotations. Coming soon.
Admin role
Component Privileges
Master annotations Replace master annotations using the annotations from other project member.
Guidelines Edit project guidelines.
Entities Edit or create entity types.
Document labels Edit or create document labels
Entity labels Edit or create entity labels
Dictionaries Edit or create dictionaries
Annotatables Edit options under the Annotatables settings
Annotations Edit options under the Annotations settings
Project Delete project

Annotation versions

Each user has a version of the annotations for each single document. E.g. UserA can have 20 annotations, UserB can have 5 different annotations on the same exact document. In addition to these versions, each document has a master version which is usually treated as the final/official version.

Annotation flows

There are different ways you can organize your annotation tasks. These are the most common:


Each annotator annotates each document

This flow is ideal for those projects requiring high-quality annotations and complex annotation tasks (specific skills required, divergent interpretations, etc.).

1Add users to your project. Once you create a project, you become its admin. Go to Settings > Members to add members to your project. Currently, each project member is added as supercurator.

2Create clear guidelines. Here the admin writes what is to be annotated and which type of annotations to use. Clear and complete guidelines are key to align all project members.

3Import text. Any project member can import the documents to be annotated by the group. Any project member can see these documents.

4The group starts annotating. Users annotate their version of the annotations. When the annotations' version of a document is complete, the user mark her/his version as completed by clicking the Confirm button. Each user will do the same when their version is completed.

5Review. The admin can review every annotation version for each document and check whether they are completed or not. The admin can import any of these versions to master (final version) and mark this version as completed by clicking the Confirm button to indicate the annotations for this document are reviewed and ready.


Documents to annotate are distributed among annotators

Make this choice if the annotation task is simple. If you assign each document to only one annotator, the quality of the annotations depends on the user assigned.

1Add users to your project. Once you create a project, you become its admin. Go to Settings > Members to add members to your project. Currently, each project member is added as supercurator role.

2Create clear guidelines. Here the admin writes what is to be annotated and which type of annotations to use. Clear and complete guidelines are key to align all project members.

3Import text. Any project member can import the documents to be annotated by the group. Any project member can see these documents.


4Distribute documents among annotators. You can, for example, assign document ids to each user.

5The group starts annotating. Users annotate their version of the annotations for the documents assigned. Once completed, the user mark her/his version as completed by clicking the Confirm button.

6Review. Admin reviews the annotated documents and if ready, import the user annotations to the master version (final version). To indicate the annotations for this document are reviewed, admin clicks the Confirm button in the master version.


Documents to annotate are distributed among annotators. No review.

Make this choice if time is a constraint and you trust the annotations to be produced. This is the simplest flow and there is no review step.

1Add users to your project. Once you create a project, you become its admin. Go to Settings > Members to add members to your project. Currently, each project member is added as supercurator role.

2Create clear guidelines. Here the admin writes what is to be annotated and which type of annotations to use. Clear and complete guidelines are key to align all project members.

3Import text. Any project member can import the documents to be annotated by the group. Any project member can see these documents.


4Assign each document to one annotator. To identify a document you use the document id.

5The group starts annotating. Each user annotates only the master version of the assigned documents. Once a document is annotated, user marks the annotations as completed by clicking the Confirm button. admin can check the progress in the document pool.