Walkthrough for annotators

This page walks through the process of manually annotating documents. This walkthrough is designed for users with the Annotator role, who need to access the Snorkel AI Data Development Platform to annotate the documents that are assigned to them.

Access your tasks

After you log in to the Snorkel AI Data Development Platform, you need to access your tasks to start annotating. A task is a collection of documents that are assigned to you to manually annotate. A task typically contains a subset of the total documents. Learn more about tasks here. For many use cases, using the Snorkel AI Data Development Platform reduces the number of documents that need to be manually annotated.

To access your tasks, click Annotate on the left-side menu.

The My tasks page shows the tasks for you to annotate:

The following information about each task is available:

• Task: The name of the task.
• Description: The description of the task.
• Date assigned: The date the task was assigned to you.
• Data points remaining: The number of datapoints that you still need to annotate.

Now, you can start annotating your documents!

Annotate your documents

Once you click Annotate, your screen will look similar to the image below:

The main canvas shows the document text. The right side-pane shows all the labeling questions you need to answer about the document.

Here is some information about the various buttons that you can see on your screen:

• Click Record view (or Table view) to change the appearance of the documents in the main canvas.
• Click the filter icon to filter the documents that are shown to you.
• Click the show/hide icon to configure which columns you can see in the main canvas.
• Click the arrow icons to page through the documents.
• Click the slice icon to add slices to the document.
• Click the comment icon to add comments to the document. For example, you can write a comment to explain your reasoning behind a selecting a particular label.
• Click the arrow up/down icons to expand and collapse individual columns.

How to Annotate

The following sections walk through how to annotate documents for the four different label types:

• single-label
• multi-label
• text
• sequence tagging

Single-label questions

For single-label questions, your goal is to assign a single class to each document. For example, assigning banking contract documents to one of the following classes: "employment" "loan," "services," or "stock."

On the right-side pane, you'll see all possible classes that you can label your document. To label a document, click the class in the right-side menu. Then click the arrow button to move on to the next document.

Multi-label questions

For multi-label questions, individual documents can have multiple label values. For example, let's say you are looking at movie review documents. You can label the movie as "Short Film," "Black and White," "Japanese Movies," or "World Cinema." Given these labels, you can see that a single movie can fall into multiple categories. In this case, for a given document, you can label each possible class as present, absent, or abstain from voting.

On the right-side pane, you'll see all possible classes. For each class, you can click:

• to label the class as Abstain.
• to label the class as Present.
• to label the class as Absent.

By default, all classes are initially labeled as Abstain. If enabled for your application, you can set the default label for each class to either Present, Absent, or Abstain.

Text questions

For text questions, your goal is to write free form text according to the label question name and description. For example, the documents could contain responses to questions and the text label is the rationale for why that is a good or bad response to the question.

Sequence tagging questions

For sequence tagging questions, your goal is to highlight and label spans throughout a document. Spans are key pieces of information that you want to extract from a document. For example, let's say we want to identify all mentions of company names in a document. In this case, a company name is considered a span. Your goal is then to highlight and label all company names that you find while reading through the document.

To label spans:

1. Highlight a span.
2. Select the label that you want for that span in the Annotation modal.

Check progress

While you are annotating documents, you can filter by Submission status to see only the data points that you still need to annotate. Once you have submitted annotations for a data point, it will transition from to do to submitted. Your annotations are saved immediately when you make them, but the status is not updated until you click the submit button.

You can see progress across all your tasks in the My Tasks page under Annotate in the sidebar. Each task includes the number of datapoints remaining for you to annotate.

Review comments and slices

Occasionally a reviewer will leave comments on documents. For example, if you left a comment asking a clarifying question, a reviewer may respond to your comment. To see all documents with comments:

1. Click the filter icon.
2. Click Comments.
3. Under User, select the person that you want to see comments from. Alternatively, you can select Any user.
4. Click the checkmark to save the filter.

Now, just the documents with comments are shown, making it easier for you to review any comments.

You can follow similar steps to filter documents based on slices:

1. Click the filter icon.
2. Click Slice.
3. Under Slice, select a slice option.
4. Under Operator, select is if you want to see all documents with that slice or is not if you want all documents with that slice removed from view.
5. Click the checkmark to save the filter.