You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/langsmith/annotation-queues.mdx
+44-38Lines changed: 44 additions & 38 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,86 +3,92 @@ title: Use annotation queues
3
3
sidebarTitle: Use annotation queues
4
4
---
5
5
6
-
Annotation queues are a powerful LangSmith feature that provide a streamlined, directed view for human annotators to attach feedback to specific runs. While you can always annotate traces inline, annotation queues provide another option to group runs together, then have annotators review and provide feedback on them.
6
+
_Annotation queues_provide a streamlined, directed view for human annotators to attach feedback to specific [runs](/langsmith/observability-concepts#runs). While you can always annotate [traces](/langsmith/observability-concepts#traces) inline, annotation queues provide another option to group runs together, then have annotators review and provide [feedback](/langsmith/observability-concepts#feedback) on them.
To create an annotation queue, navigate to the **Annotation queues** section through the homepage or left-hand navigation bar. Then click **+ New annotation queue** in the top right corner.
12
+
1. Navigate to the **Annotation queues** section on the left-hand navigation panel of the [LangSmith UI](https://smith.langchain.com).
13
+
1. Click **+ New annotation queue** in the top right corner.
15
16
16
17
### Basic Details
17
18
18
-
Fill in the form with the **name** and **description** of the queue. You can also assign a **default dataset** to queue, which will streamline the process of sending the inputs and outputs of certain runs to datasets in your LangSmith workspace.
19
+
1.Fill in the form with the **Name** and **Description** of the queue. You can also assign a **default dataset** to queue, which will streamline the process of sending the inputs and outputs of certain runs to datasets in your LangSmith [workspace](/langsmith/administration-overview#workspaces).
19
20
20
21
### Annotation Rubric
21
22
22
-
Begin by drafting some high-level instructions for your annotators, which will be shown in the sidebar on every run.
23
+
1. Draft some high-level instructions for your annotators, which will be shown in the sidebar on every run.
24
+
1. Click **+ Desired Feedback** to add feedback keys to your annotation queue. Annotators will be presented with these feedback keys on each run.
25
+
1. Add a description for each, as well as a short description of each category, if the feedback is categorical.
23
26
24
-
Next, click "+ Desired Feedback" to add feedback keys to your annotation queue. Annotators will be presented with these feedback keys on each run. Add a description for each, as well as a short description of each category if the feedback is categorical.
27
+
For example, with the descriptions in the previous screenshot, reviewers will see the **Annotation Rubric** details in the right-hand pane of the UI.
27
30
28
-
Reviewers will see this:
29
-
30
-
31
+
31
32
32
33
### Collaborator Settings
33
34
34
-
There are a few settings related to multiple annotators:
35
-
36
-
***Number of reviewers per run**: This determines the number of reviewers that must mark a run as "Done" for it to be removed from the queue. If you check "All workspace members review each run," then a run will remain in the queue until all workspace members have marked it "Done".
35
+
When there are multiple annotators for a run:
37
36
38
-
* Reviewers cannot view the feedback left by other reviewers.
39
-
* Comments on runs are visible to all reviewers.
37
+
-**Number of reviewers per run**: This determines the number of reviewers that must mark a run as **Done** for it to be removed from the queue. If you check **All workspace members review each run**, then a run will remain in the queue until all [workspace](/langsmith/administration-overview#workspaces) members have marked their review as **Done**.
40
38
41
-
***Enable reservations on runs**: We recommend enabling reservations. This will prevent multiple annotators from reviewing the same run at the same time.
39
+
- Reviewers cannot view the feedback left by other reviewers.
40
+
- Comments on runs are visible to all reviewers.
42
41
43
-
1.**How do reservations work?**
42
+
-**Enable reservations on runs**: When a reviewer views a run, the run is reserved for that reviewer for the specified **Reservation length**. If there are multiple reviewers per run as specified above, the run can be reserved by multiple reviewers (up to the number of reviewers per run) at the same time.
44
43
45
-
When a reviewer views a run, the run is reserved for that reviewer for the specified "reservation length". If there are multiple reviewers per run as specified above, the run can be reserved by multiple reviewers (up to the number of reviewers per run) at the same time.
44
+
<Tip>
45
+
We recommend enabling reservations. This will prevent multiple annotators from reviewing the same run at the same time.
46
+
</Tip>
46
47
47
-
2.**What happens if time runs out?**
48
+
If a reviewer has viewed a run and then leaves the run without marking it **Done**, the reservation will expire after the specified **Reservation length**. The run is then released back into the queue and can be reserved by another reviewer.
48
49
49
-
If a reviewer has viewed a run and then leaves the run without marking it "Done", the reservation will expire after the specified "reservation length". The run is then released back into the queue and can be reserved by another reviewer.
50
+
<Note>
51
+
Clicking **Requeue** for a run's annotation will only move the current run to the end of the current user's queue; it won't affect the queue order of any other user. It will also release the reservation that the current user has on that run.
52
+
</Note>
50
53
51
-
<Note>
52
-
Clicking "Requeue at end" will only move the current run to the end of the current user's queue; it won't affect the queue order of any other user. It will also release the reservation that the current user has on that run.
53
-
</Note>
54
+
As a result of the **Collaborator settings**, it's possible (and likely) that the number of runs visible to an individual in an annotation queue differs from the total number of runs in the queue compared to another user's queue size.
54
55
55
-
Because of these settings, it's possible (and likely) that the number of runs visible to an individual in an annotation queue differs from the total number of runs in the queue as well as anyone else's queue size.
56
+
You can update these settings at any time by clicking on the pencil icon <Iconicon="pencil"/> in the **Annotation Queues** section.
56
57
57
-
You can update these settings at any time by clicking on the pencil icon in the **Annotation Queues** section.
58
+
## Assign runs to an annotation queue
58
59
59
-
60
+
To assign runs to an annotationqueue, do one of the following:
60
61
61
-
## Assign runsto an annotation queue
62
+
- Click on **Add to Annotation Queue** in top right corner of any [trace](/langsmith/observability-concepts#traces) view. You can add any intermediate [run](/langsmith/observability-concepts#runs) (span) of the trace to an annotation queue, but not the root span.
62
63
63
-
To assign runs to an annotation queue, either:
64
+
64
65
65
-
1. Click on **Add to Annotation Queue**in top right corner of any trace view. You can add ANY intermediate run (span) of the trace to an annotation queue, not just the root span. 
66
+
- Select multiple runs in the runs table then click **Add to Annotation Queue**at the bottom of the page.
66
67
67
-
2. Select multiple runs in the runs table then click **Add to Annotation Queue**at the bottom of the page.
68
+
68
69
69
-
3.[Set up an automation rule](/langsmith/rules) that automatically assigns runs which pass a certain filter and sampling condition to an annotation queue.
70
+
-[Set up an automation rule](/langsmith/rules) that automatically assigns runs that pass a certain filter and sampling condition to an annotation queue.
71
+
- Navigate to the **Datasets & Experiments** page and select a dataset. On the dataset's page select one or multiple [experiments](/langsmith/evaluation-concepts#experiment). At the bottom of the page, click **<Iconicon="pencil"/> Annotate**. From the resulting popup, you can either create a new queue or add the runs to an existing one.
70
72
71
-
4. Select one or multiple experiments from the dataset page and click **Annotate**. From the resulting popup, you may either create a new queue or add the runs to an existing one: 
73
+
72
74
73
75
<Check>
74
-
It is often a very good idea to assign runs that have a certain user feedback score (eg thumbs up, thumbs down) from the application to an annotation queue. This way, you can identify and address issues that are causing user dissatisfaction. To learn more about how to capture user feedback from your LLM application, follow [this guide](/langsmith/attach-user-feedback).
76
+
It is often a good idea to assign runs that have a particular type of user feedback score (e.g., thumbs up, thumbs down) from the application to an annotation queue. This way, you can identify and address issues that are causing user dissatisfaction. To learn more about how to capture user feedback from your LLM application, follow the guide on [attaching user feedback](/langsmith/attach-user-feedback).
75
77
</Check>
76
78
77
79
## Review runs in an annotation queue
78
80
79
-
To review runs in an annotation queue, navigate to the **Annotation Queues** section through the homepage or left-hand navigation bar. Then click on the queue you want to review. This will take you to a focused, cyclical view of the runs in the queue that require review.
81
+
To review runs in an annotation queue:
80
82
81
-
You can attach a comment, attach a score for a particular feedback criteria, add the run a dataset and/or mark the run as reviewed. You can also remove the run from the queue for all users, despite any current reservations or settings for the queue, by clicking the **Trash** icon next to "View run".
83
+
1. Navigate to the **Annotation Queues** section through the left-hand navigation bar.
84
+
1. Click on the queue you want to review. This will take you to a focused, cyclical view of the runs in the queue that require review.
85
+
1. You can attach a comment, attach a score for a particular [feedback](/langsmith/observability-concepts#feedback) criteria, add the run to a dataset or mark the run as reviewed. You can also remove the run from the queue for all users, despite any current reservations or settings for the queue, by clicking the **Trash** icon <Iconicon="trash"/> next to **View run**.
82
86
83
-
The keyboard shortcuts shown can help streamline the review process.
87
+
<Tip>
88
+
The keyboard shortcuts that are next to each option can help streamline the review process.
89
+
</Tip>
84
90
85
-
91
+
0 commit comments