Managing tasks
This page walks through how to manage annotation tasks. Here you can learn how to add data points, how to create questions for your task, and how to assign annotators to data points.
You must have first created a task in order to manage it.
To manage tasks you must have the Reviewer role. For more information about access and permissions, see Roles and user permissions.
The linked tasks page
After navigating to a given dataset, under the Linked annotation tasks tab, you will find the list of all tasks associated with the given dataset. Clicking an entry in the list will navigate you to the task management page. On the right hand side of each item in the list is a menu with options to delete a given task, or begin annotating it.
The task management page
To begin managing a task, select it from the linked tasks page, doing so will navigate you to the task management page, on which you will see and be able to configure the following:
- Task title and description
- A link to the list of data points assigned to the task
- A link to the list of questions asked for each data point the task
- A link to the list of annotators assigned to the data points
- The minimum number of annotations per data point required
Adding/Removing data points
When you click Add (in the case of a task with no datapoints) or Edit (in the case of a task with some datapoints) you will be brought to a collection of data points in the dataset associated with the task. Here you can see the data for each datapoint as well as whether or not it has been added to the task.
At the top right of the table you will see a count of how many data points have already been added to the task as well as buttons to Remove from task and Add to task which become enabled once selections are made in the table.
To add or remove data points, you must first select them with the checkboxes on the left hand side of the table. You can also select the checkbox in the table header, which will select all data points in the current page, and will provide an option to select all data points in the dataset.
Once you have selected some data points, click on either Add to task or Remove from task to manage the data points that are included in this task.
Clicking Next will take you back to the task configuration page.
Creating and managing questions
The second step to configure an Annotation Task is to set up the Label Form, which is the set of questions that annotators are being asked to answer for each assigned data point for the task.
Choose from any of the previously created questions for this dataset in the dropdown.
Or, create new ones by clicking the + Add Question button. When creating new questions, use the dropdown on the top-right to change the new question's type. Refer to Label Form Questons for more information on each type.
Click Next to save your label schema and will take you back to the task configuration page.
Assigning annotators
When you click Assign (in the case of a task with no annotators) or Edit (in the case of a task with some annotators assigned) you will be brought to the selected data points for this task. Here you can see the data for each datapoint as well as a column listing the assignees (if there is one assignee it will be displayed directly, if there are multiple, it will display the number of assignees and hovering over it will reveal the names of the asignees in a tooltip), and a column listing the status of the datapoint. For more information on statues see ++LINK TO STATUSES DOC++
At the top right of the table you will see a button to Assign annotators. Until you select one or more data points, this buttons will be disabled.
To assign or unnassign annotators to data points, you must first select them with the checkboxes on the left hand side of the table. You can also select the checkbox in the table header, which will select all data points in the current list, and which will present you with the option to select all data points in the dataset.
Once you have selected some data points, you will see in the top right of the table a count of how many you have selected, and the button to assign annotators will be enabled. Clicking it will open up a menu to either add or remove annotators.
If you select Add annotators, you will be presented with a list of users available in your current workspace to assign to data points. When you click the Assign button, all users that you have selected will be assigned to the data points you had selected when when you clicked the Add annotators button.
This button is initially disabled until you select at least one user.
You are also presented with two methods of assignment:
- Assign all selected annotators to data points
If you select this option, every annotator you have selected will be assigned to every data point you had selected when you clicked Add annotators.
- Randomly distribute annotators among data points
If you select this option, an input will appear where you can enter how many annotators you would like assigned per data point. This number must be no fewer than one and no greater than the number of users you have selected. Once you have entered a valid number, the Assign button will be enabled, and pressing it will ensure that each datapoint is randomly assigned the number of annotators that you entered.
If you select Remove annotators, you will be presented with a list of annotators each currently assigned to at least one of the data points you have selected. Choose which annotators you would like to remove from the selected data points, then click Remove.
Once you have clicked Assign in the assignment form, or Remove in the removal form, you will be brought back to the table view of data points, where you can see your changes reflected.
A given data point will have the Needs assignees status until you have assigned the minimum number of required annotators per data point for the task. Once you have done this, it will automatically move to In annotation, and if you remove annotators from a datapoint such that it dips below the minimum, it will automatically move back to Needs assignees.
Clicking Next will take you back to the task configuration page.