Walkthrough for reviewers

This guide walks you through the reviewer workflow within annotation tasks. As a reviewer, you can evaluate annotations from multiple annotators and commit final decisions to ground truth.

info

As a reviewer, you may also need to annotate documents yourself. See the Annotator workflow for step-by-step annotation instructions.

Getting started with reviewer mode

Accessing reviewer mode

Reviewer mode is available in the annotation data viewer. There are two ways to navigate to an annotation task:

1. From the Annotate tab: Click Annotate in the left-side menu, then select your task from the list.
2. From the Datasets page: Click Datasets in the left-side menu, select your dataset, go to the Linked annotation tasks tab, then click the task you want to work on.

Once you're in the task, you can access the annotation data viewer from the Overview tab. From the Overview tab, select the Review button. This takes you to the annotation data viewer with the reviewer mode toggled on for you. There are two modes within this dataviewer:

• Annotation view: Your personal annotation interface
• Reviewer view: Interface showing all data points with annotator submissions for review

You can toggle between these modes using the reviewer toggle.

Understanding data point status

Each data point progresses through these status stages:

Status	Description
Needs assignees	Minimum number of required annotators annotators have not been assigned to this data point
In annotation	Annotators are currently working on this data point
Ready for review	Minimum number of required annotators have completed their work and the data point awaits your review
Completed	You have reviewed and committed the annotations to ground truth

Filtering and finding data points

Use the filtering options to focus your review efforts on specific data points based on your workflow needs. Filter by Ready for review status to see only data points that have completed annotations and are waiting for your review. This helps you efficiently manage large annotation tasks by viewing targeted subsets of data.

Reviewing annotations

Review tools and interactions

While reviewing annotations, you have access to several tools:

• Comments: Add explanations for why you agree or disagree with specific annotations. Access this tool via the comment icon in the top right corner of the document.
• Slices: Create document slices (e.g., "contains-disagreements") to categorize data points. Add or manage slices by clicking the slice icon in the top right corner of the document.

Reviewing different question types

The review interface adapts based on your task's label schema configuration. Learn how to review each question type:

• Reviewing single-label questions
• Reviewing multi-label questions
• Reviewing text label questions
• Reviewing sequence tagging questions

Finalizing ground truth

When you're ready to commit

After reviewing all annotator submissions and making your decisions, you'll finalize the ground truth for each data point. It is recommended to commit annotations to ground truth by going through each data point individually, but there is also a bulk commit option.

Committing annotations per data point

Commit requirements:

• Complete all questions for the data point by making your selections
• The commit button will turn blue when the data point is ready to commit

Committing ground truth

When you press the Commit button:

1. Initial commit:

   • Your selections are saved as ground truth with pink badges marking each committed label
   • A timestamp appears showing exactly when you committed
   • The data point status changes to Completed
   

2. Making changes after commit:

   • Modify your selections as needed
   • Press Commit again to update
   • Timestamp updates to show the latest commit
   • Pink ground truth badges will update to reflect your new selections
   • Status stays Completed

Committing annotations in bulk

You can only bulk commit annotations on data points that are Ready for review. If you select data points that are not ready for review, they will automatically be skipped. To commit annotations in bulk, complete the following steps:

1. Navigate to table view.
2. From table view, select the data points you'd like to commit.
3. Press the blue Commit annotations button in the top right hand corner of the dataviewer.
4. Confirm you'd like to commit annotations.

Bulk committing annotations will also move data points from Ready for review to Complete.

The bulk commit algorithm automatically aggregates the annotations depending on the type of label schema.

• Single label schema: majority vote between all annotations and ties are broken randomly
• Text label schema: all annotations are concatenated together to create one text label
• Multi label schema: simple union where the label (present or absent) with the most votes wins. Ties are broken randomly
• Sequence tagging: a superset of all annotations is created, including overlapping sequences where applicable

Ground truth is attributed to a new aggregation source and should be visible in the UI after the bulk commit process completes.

You can also bulk commit ready-for-review datapoints through SDK. See AnnotationTask.commit_annotations in SDK documentation.

Working with existing ground truth

Some data points may already have ground truth from:

• Data imports
• Other tasks on same data points

What you'll see:

• Pink ground truth badge indicating existing ground truth
• Data point will not show as Completed for this specific task

What you need to do:

• Review the existing ground truth and correct any errors
• Press Commit to mark the data point as completed in this task context
• This confirms this data point is completed for your current task
• For tasks with many data points that have existing ground truth, you can use the SDK to complete multiple data points at once rather than completing each one individually. See AnnotationTask.mark_datapoints_complete in SDK documentation.