Walkthrough for reviewers
This page walks through the different ways that users with the Reviewer role can review annotations from all annotators.
As a reviewer, you will likely be tasked to manually annotate documents, in addition to reviewing annotations from others. Follow the Walkthrough for annotators to get step-by-step instructions annotating documents for the different task types.
You can review annotations in two places:
- Within a batch: View all annotations made on a single document side-by-side.
- On the Review page: View a full list of all annotations and their corresponding documents.
In addition, you have some additional permissions that allow you to manage batches.
Review documents within a batch
While inside a batch that you are assigned, being a reviewer allows you to review other users' annotations. With multi-schema annotation, reviewing becomes a superset of annotating. In this view, you can see all annotations by toggling the Reviewer mode on and off.
How you review the annotations differs based on the task type. For all task types, you can click the comment 
icon to add an explanation for why you agree or disagree with other annotations. In addition, you can click the slice 
icon to add a slice to the document. For example, you may want to create a slice called contains-disagreements to slice documents that have significant disagreement among the annotators.
Review classification and extraction annotations
For classification and extraction applications, you can see all annotations made by other annotators. You can also see the annotator's name and the annotation label that they selected.
Review sequence tagging annotations
In sequence tagging applications, you can see annotations in the label pop-up when you select an annotated span.
Review all annotations on the review page
The Review page provides a full list of all annotations and their corresponding documents to easily review all annotations in one place. To access the Review page:
- Click Datasets in the left-side menu.
- Click the name of the dataset that you want to review.
- Click the Review tab.
Select an annotation to pull up the full document text. Here, you have the option to leave comments and add slices to the document. In addition, you can select the 
Filters button on the top-right corner of your screen to filter the table by a particular annotator.
Manage batches
As a reviewer, you have additional permissions that allow you to manage batches:
- Rename and delete batches.
- Aggregate annotations in a batch.
- Set an annotator as an expert for that batch to view the agreement rate for each annotator relative to the expert.