Label form questions

The following sections walk through how to annotate documents for the four different question types:

• single-label
• multi-label
• text label
• sequence tagging

Single-label

In single-label questions, an annotator will assign a single label to each document. For example, assigning banking contract documents to one of the following classes: "employment," "loan," "services," or "stock."

To set up a single-label question, replace the placeholder name with a clear, descriptive question (e.g., “What sentiment does this text express?”). Then, click on a label name (e.g. "Option 1") to modify it (except for the default UNKNOWN label) or click "Add option" to add additional label options.

Multi-label

In multi-label questions, individual documents can have multiple label values. For example, let's say you are looking at movie review documents. You can label the movie as "Short Film," "Black and White," "Japanese Movies," or "World Cinema." Given these labels, you can see that a single movie can fall into multiple categories. In this case, for a given document, you can label each possible class as present, absent, or abstain from voting. By default, all classes are initially labeled as Abstain (If enabled for your application, you can configure the default label).

To set up a multi-label question, replace the placeholder name with a clear, descriptive question (e.g., “What sentiment does this text express?”). Then, click on a label name (e.g. "Option 1") to modify it. Click "Add option" to add additional labels.

Text label

Text label questions, allow for annotators to enter freeform text.

To set up a text label question, replace the placeholder name with a clear, descriptive question (e.g., “What sentiment does this text express?”). (Optional) Then, add a description for the question.

Sequence tagging

Snorkel supports sequence tagging for extraction tasks. Annotators can label spans of text to extract key information from a document. To label spans in the document, you can highlight a section of text and select the span label from the pop-up menu.

To set up a sequence tagging question:

1. Replace the placeholder name with a clear, descriptive question (e.g., “Company name”).
2. Select the primary text field of your dataset for which the sequence tags should be applied. This is the only column that you will be able to use this label schema on.
3. Set the IOU Agreement score. The IOU Agreement score is the percentage (Intersection over Union) of overlap required between two annotations with the same label, to consider them as an agreement. This only applies to the IAA matrix.
4. Select whether or not text spans are allowed to overlap.
5. Click on a label name (e.g. "Option 1") to modify it.
6. (Optional) Click "Add option" to add additional labels.