Walkthrough for reviewers
This page walks through the different ways that users with the Reviewer role can review annotations from all annotators.
As a reviewer, you will likely be tasked to manually annotate documents, in addition to reviewing annotations from others. Follow the Walkthrough for annotators to get step-by-step instructions annotating documents for the different task types.
You can review annotations in two places:
- Within a batch: View all annotations made on a single document side-by-side.
- On the Review page: View a full list of all annotations and their corresponding documents.
In addition, you have some additional permissions that allow you to manage batches.
Review documents within a batch
While inside a batch that you are assigned, being a reviewer allows you to review other users' annotations and mark them with agreement or disagreement. In this view, you can see all annotations made on a single document side-by-side.
How you review the annotations differs based on the task type. For all task types, you can click the comment 
icon to add an explanation for why you agree or disagree with other annotations. In addition, you can click the tag 
icon to add a tag to the document. For example, you may want to create a tag called contains-disagreements to tag documents that have significant disagreement among the annotators.
Review classification and extraction annotations
For classification and extraction applications, click Other Annotations at the bottom of the right pane to view all annotations for the document. If you agree with an annotation, you can click the Agree button to update your annotation to match.
Review sequence tagging annotations
In sequence tagging applications, you can toggle between two modes: Annotating and Reviewing.
- In Annotating mode, you can label spans as usual and see your own annotations.
- In Reviewing mode, you can see the highlighted spans from other annotators.
You can visualize annotator conflict by inspecting where multiple overlapping labels exist. Click the label highlight to get more information about the span and annotator. You can use the up and down arrows on your keyboard to quickly look through the data associated with each highlight.
Review all annotations on the review page
The Review page provides a full list of all annotations and their corresponding documents to easily review all annotations in one place. To access the Review page:
- Click Annotate in the left-side menu.
- Click Review in the top-left corner of your screen.
Select an annotation to pull up the full document text. Here, you have the option to leave comments and add tags to the document. In addition, You can select the 
Filters button on the top-right corner of your screen to filter the table by a particular annotator.
Manage batches
As a reviewer, you have additional permissions that allow you to manage batches:
- Rename and delete batches.
- Aggregate annotations in a batch.
- Set an annotator as an expert for that batch to view the agreement rate for each annotator relative to the expert.